Voice editing method, electronic device and storage medium

Abstract

The present invention discloses a speech editing method, an electronic device, and a storage medium, wherein a speech editing method includes: inputting the spectrum of original speech, the phoneme sequence of the original speech, and the alignment relationship between the phoneme sequence and the spectrum into an encoder for encoding to obtain an original latent space code; dividing the original latent space code into multiple segments, corresponding one of the multiple segments to the phoneme sequence that needs to be modified, and keeping the other segments of the multiple segments unchanged; obtaining an edited phoneme sequence, using a diffusion model to generate a new segment based on the edited phoneme sequence, wherein the diffusion process of the diffusion model is guided by the other segments; using a decoder to decode the new latent space code composed of the new segment and the other segments to obtain an edited spectrum. The method of this embodiment can achieve zero-sample speech editing.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of speech recognition technology, and in particular to a speech editing method, electronic device, and storage medium. Background Art

[0002] Voice cloning is a process that uses artificial intelligence to generate a voice that closely resembles the original speaker. The goal of voice cloning is to create a synthetic voice system that can mimic the voice characteristics of a specific person or speaker.

[0003] Zero-shot Voice Cloning refers to the process of using artificial intelligence technology to generate speech that is very similar to a target speaker without first obtaining an audio sample of the target speaker. The goal of this method is to train a general speech synthesis model that has strong generalization capabilities for different speakers, so that it can still imitate its voice characteristics given only a small amount of audio data of the target speaker. Text-based Speech Editing is a technology that uses text representation to modify and adjust human speech in audio files. With text-based speech editing, users can directly edit the text corresponding to the audio, thereby modifying the audio content without having to directly process the audio waveform or operate the timeline in a traditional audio editor.

[0004] Diffusion generative models (GMMs) are a class of deep generative models used to synthesize realistic data, such as images, text, and audio. Diffusion models are based on a process called diffusion, which involves gradually transforming data from a simple distribution (such as Gaussian noise) to a more complex, structured distribution. The key idea behind GMMs is to model the data distribution by learning the inverse diffusion process. This inverse process transforms a simple noise distribution into a target data distribution through a series of learned transformations. These transformations are typically implemented using neural networks.

[0005] Variational autoencoders (VAEs) are a type of generative model designed to learn latent representations of data and generate new samples similar to the training data. VAEs combine deep learning and probabilistic modeling methods and are based on a neural network architecture. They consist of two parts: an encoder and a decoder. The encoder maps input data (such as an image or text) into a latent space (i.e., a low-dimensional latent variable representation). The latent space typically follows a known probability distribution, such as a Gaussian distribution. The encoder's output is the mean and variance parameters in the latent space. These parameters define a distribution over the latent variables that captures the uncertainty of the input data. The decoder performs the opposite operation. Given a point in the latent space, the decoder transforms it back into the original data space. In other words, the decoder reconstructs the input data from the latent representation generated by the encoder.

[0006] Training a VAE involves maximizing a lower bound on the marginal likelihood of the data (also called the variational lower bound or ELBO), which can be viewed as a goal that balances the reconstruction error with the divergence between the latent variable distribution and the prior distribution (usually using the Kullback-Leibler divergence). By optimizing this goal, the VAE learns to capture the structure of the data in the latent space and can generate new samples with similar properties.

[0007] Generative Adversarial Networks (GAN) is a deep generative model proposed by Ian Goodfellow in 2014. GAN learns to generate realistic data such as images, text, and audio by simultaneously training two competing neural networks (a generator and a discriminator).

[0008] 1. Generator: The task of the generator is to generate realistic data from random noise. It tries to generate fake data that is as close as possible to the real data so that it can be disguised as real data in front of the discriminator.

[0009] 2. Discriminator: The task of the discriminator is to classify the input data and determine whether they are fake data generated by the generator or real data from the real dataset. The goal of the discriminator is to improve the ability to distinguish between real and fake data.

[0010] The GAN training process can be viewed as a two-player game. The generator attempts to generate realistic fake data to deceive the discriminator, while the discriminator strives to improve its ability to distinguish real data from fakes. This process is implemented through a min-max optimization problem. During training, the generator and discriminator's loss functions compete with each other. During training, the generator and discriminator continuously improve each other. As training progresses, the data produced by the generator becomes increasingly realistic, and the discriminator's classification ability continues to improve. When training reaches equilibrium, the generator can generate data so realistic that the discriminator has difficulty distinguishing real data from fake data.

[0011] Connectionist Temporal Classification (CTC) is a training method for sequence-to-sequence learning problems, particularly suitable for tasks where the sequence length is variable and alignment is not required. CTC was proposed by Alex Graves et al. in 2006 and is commonly used in tasks such as speech recognition and handwritten text recognition. In traditional sequence-to-sequence problems, it is usually necessary to strictly align the input sequence with the output sequence. However, this may not be feasible in some tasks, especially when there is no clear correspondence between the input and output. CTC solves this problem by introducing a special "blank" token in the output sequence, allowing the model to automatically learn alignment without explicit alignment information.

[0012] The key idea of ​​CTC is to marginalize over all possible alignments of the input sequence and then maximize the probability of the correct output sequence. During training, CTC uses dynamic programming algorithms (such as the forward-backward algorithm) to efficiently compute probabilities and gradients. CTC can be combined with recurrent neural networks (RNNs) or other sequence models (such as the Transformer).

[0013] Automatic speech recognition (ASR) is a technology that converts sound signals into text. The goal of an ASR system is to automatically generate a corresponding text representation given an acoustic input (such as speech or conversation). ASR has a wide range of applications, including voice assistants, transcription services, speech translation, and accessibility services.

[0014] Stochastic Differential Equations (SDEs) are a class of differential equations involving random processes. They describe the evolution of a system under the influence of random perturbations (usually noise). SDE has a wide range of applications in many fields, such as physics, chemistry, biology, finance, and engineering. SDE usually contains two main parts: the drift term and the diffusion term. The drift term represents the deterministic part of the system, that is, how the system will evolve in the absence of noise. The diffusion term introduces randomness and is usually multiplied by a random process (such as Brownian motion or Wiener process) to describe the impact of noise on the system state. A typical one-dimensional SDE is in the following form:

[0015] d X(t)= a(X(t),t)dt+ b(X(t),t)d W(t) (1)

[0016] Here, X(t) is a random variable representing the state of the system at time t; a(X(t), t) is a drift term describing the deterministic evolution; b(X(t), t) is a diffusion term describing the intensity of the random perturbation; and W(t) is a Wiener process representing the random perturbation. Because SDEs involve random processes, their solutions are often approximated using numerical methods such as the Euler-Maruyama method. These methods discretize the time interval and then compute an approximate solution to the system state by simulating the random process at each time step.

[0017] The Mean Opinion Score (MOS) is a subjective rating method used to evaluate voice quality in voice communication systems (such as telephones, VoIP, and video conferencing). The MOS score is obtained by asking a group of listeners to rate a given speech sample and calculating the average. MOS is widely used to evaluate and compare the impact of voice codecs, transmission protocols, and network conditions on voice quality. MOS scores typically range from 1 (worst) to 5 (best). Listeners are typically asked to rate a given sample based on factors such as speech clarity, distortion, and background noise. The resulting MOS score reflects overall satisfaction with voice quality. Summary of the Invention

[0018] The embodiments of the present invention provide a voice editing method, device, and electronic device, which are used to solve at least one of the above technical problems.

[0019] In a first aspect, an embodiment of the present invention provides a speech editing method, comprising: inputting a spectrum of an original speech, a phoneme sequence of the original speech, and an alignment relationship between the phoneme sequence and the spectrum into an encoder for encoding to obtain an original latent space code; dividing the original latent space code into multiple segments, making a segment of the multiple segments correspond to the phoneme sequence that needs to be modified, and keeping other segments of the multiple segments unchanged; obtaining an edited phoneme sequence, and generating a new segment based on the edited phoneme sequence using a diffusion model, wherein the diffusion process of the diffusion model is guided by the other segments; and using a decoder to decode the new latent space code composed of the new segment and the other segments to obtain an edited spectrum.

[0020] In a second aspect, an embodiment of the present invention provides an electronic device comprising: at least one processor, and a memory communicatively connected to the at least one processor, wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute any one of the above-mentioned voice editing methods of the present invention.

[0021] In a third aspect, an embodiment of the present invention provides a storage medium, in which one or more programs including execution instructions are stored. The execution instructions can be read and executed by an electronic device (including but not limited to a computer, a server, or a network device, etc.) to execute any of the above-mentioned voice editing methods of the present invention.

[0022] In a fourth aspect, an embodiment of the present invention further provides a computer program product, which includes a computer program stored on a storage medium, and the computer program includes program instructions. When the program instructions are executed by a computer, the computer executes any one of the above-mentioned speech editing methods.

[0023] An embodiment of the present application proposes a text-driven speech editing method based on a latent space diffusion model. The method compresses and encodes speech into a latent space, then divides the original latent space into multiple segments, and corresponds one of the segments to the phoneme sequence that needs to be modified, while the other segments remain unchanged. The diffusion model is used to generate a new segment based on the edited phoneme sequence, thereby achieving zero-sample speech editing. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0025] Figure 1 A flowchart of a voice editing method provided by one embodiment of the present invention;

[0026] Figure 2 A schematic diagram of a text-driven speech editing method based on a latent space diffusion model provided by one embodiment of the present invention;

[0027] Figure 3a This is a structural diagram of a spectrum encoder for a DiffVoice system according to an embodiment of the present invention:

[0028] Figure 3b A structural diagram of a spectrum decoder of a DiffVoice system provided in one embodiment of the present invention:

[0029] Figure 3c This is a structural diagram of the latent space diffusion model of the DiffVoice system provided by one embodiment of the present invention:

[0030] Figure 4 A diagram illustrating a decoder modified for adversarial training according to an embodiment of the present invention;

[0031] Figure 5 Hyperparameters of the Conformer module in DiffVoice provided in one embodiment of the present invention;

[0032] Figure 6 The result of speech synthesis on LJSpeech provided by an embodiment of the present invention;

[0033] Figure 7 The result of speech synthesis using a visible speaker on LibriTTS provided by an embodiment of the present invention;

[0034] Figure 8 The result of zero-sample voice cloning on LibriTTS provided by one embodiment of the present invention;

[0035] Figure 9 The result of text-based speech repair provided by an embodiment of the present invention;

[0036] Figure 10 A schematic structural diagram of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0037] To make the objectives, technical solutions, and advantages of the embodiments of the present invention more clear, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. All other embodiments obtained by ordinary technicians in this field based on the embodiments of the present invention without making any creative efforts shall fall within the scope of protection of the present invention.

[0038] Please refer to Figure 1 , which shows a flowchart of a voice editing method provided by an embodiment of the present invention.

[0039] like Figure 1 As shown, in step 101, the spectrum of the original speech, the phoneme sequence of the original speech, and the alignment relationship between the phoneme sequence and the spectrum are input into the encoder for encoding to obtain the original latent space code;

[0040] In step 102, the original latent space code is divided into multiple segments, and a segment among the multiple segments is made to correspond to the phoneme sequence that needs to be modified, while the other segments among the multiple segments remain unchanged;

[0041] In step 103, the edited phoneme sequence is obtained, and a new segment is generated based on the edited phoneme sequence using a diffusion model, wherein the diffusion process of the diffusion model is guided by the other segments;

[0042] In step 104, a decoder is used to decode the new latent space code composed of the new segment and the other segments to obtain an edited spectrum.

[0043] The method of this embodiment compresses and encodes speech into a latent space, then divides the original latent space into multiple segments, corresponds one segment to the phoneme sequence that needs to be modified, and keeps the other segments unchanged. A diffusion model is used to generate a new segment based on the edited phoneme sequence, thereby achieving zero-sample speech editing.

[0044] In some optional embodiments, the diffusion model performs a reverse diffusion process starting from Gaussian noise, and the diffusion model generates a new latent space code through repeated iterations. The diffusion process of the diffusion model is guided by the other segments to basically keep the other segments in the new latent space code consistent with the other segments in the original latent space code.

[0045] In some optional embodiments, the alignment relationship between the phoneme sequence and the spectrum is obtained by using a connection-time classification automatic speech recognition model trained based on the phoneme sequence to obtain the alignment relationship between the phoneme sequence and the spectrum.

[0046] In some optional embodiments, the inputting the signal to the encoder for encoding includes: using a variational autoencoder enhanced by adversarial training to encode the speech signal into a latent space to obtain a phoneme-level latent space representation.

[0047] In some optional embodiments, the variational autoencoder is a variational autoencoder-generative adversarial network with dynamic downsampling in time.

[0048] In some optional embodiments, the total training loss of the variational autoencoder-generative adversarial network is a weighted sum of the minimum mean square loss of the generator, the minimum mean square loss of the discriminator, the feature matching loss, and the variational autoencoder loss.

[0049] In some optional embodiments, the diffusion model jointly models phoneme duration and mel spectrum using a single diffusion model in a latent space based on dynamic rate downsampling.

[0050] Among related technologies, Descript is a commercial audio and video editing product that can recognize text from audio and video and allow users to directly edit text to complete audio and video editing. However, it does not implement zero-sample voice editing and requires learning the voice of the same person for dozens of minutes before voice editing can be performed. VALL-E is a preprint paper recently published by Microsoft. It uses a speech encoder to encode speech into discrete units and then performs speech synthesis through an autoregressive model. VALL-E can perform speech cloning by providing a short speech clip and generating the following content of the speech. However, because it is a sub-regressive generative model, VALL-E has difficulty in performing speech editing.

[0051] There are many approaches to voice editing that use neural network regression to complete speech fragments, with improvements primarily focused on the neural network structure. Modified text and speech are fed into the neural network, which then outputs the modified speech fragment. However, due to the difficulty of outputting the correct content all at once, the speech editing instructions from these approaches are often suboptimal.

[0052] The present embodiment proposes to compress speech and encode it into a latent space to achieve joint modeling of duration and pronunciation. The ingenious design of the present embodiment enables the image editing method to be applied to the field of speech editing. The method of the present embodiment of the application continuously iterates through a neural network, allowing the model to gradually optimize the output results.

[0053] Please refer to Figure 2 , which shows a schematic diagram of a solution principle of an embodiment of the present application.

[0054] like Figure 2As shown, first, the spectrum of the original speech + the original speech phoneme sequence + the alignment relationship between the phoneme and the spectrum (which can be obtained through speech recognition) are input to the encoder to obtain the latent space code. The latent space code is then divided into three segments A, B, and C, where segment B corresponds to the speech segment that needs to be modified. Among them, a diffusion model is used to generate a new segment B. Specifically, a reverse diffusion process is performed starting from Gaussian noise. The diffusion model generates a new latent space code through repeated iterations. During the diffusion process, the model is guided to try to keep the A and C segments consistent with the original code. Finally, the new latent space code is used to synthesize the edited spectrum. Those skilled in the art will understand that the number of segments can also be other numbers. The situation of more segments than 3 can be solved by repeatedly applying the situation of 3 segments, so only the situation of 3 segments is described in the embodiment of the present application.

[0055] In an embodiment of the present application, an embodiment of the present application proposes DiffVoice, a new text-to-speech model based on latent space diffusion. The embodiment of the present application suggests first using a variational autoencoder enhanced by adversarial training to encode the speech signal into a phoneme-level latent space representation, and then using a diffusion model to jointly model the duration and latent space representation, wherein the Mel spectrum reconstructed by the variational autoencoder alone does not have high-frequency details, which will cause metallic noise when synthesizing speech from the Mel spectrum using a neural network vocoder in the next step. Adding adversarial training enables the system to reconstruct high-frequency information, which can solve this problem. Furthermore, joint modeling simplifies the model structure; it makes the model convenient for speech editing, otherwise it is necessary to generate the duration and latent space representation of the edited part separately.

[0056] Subjective evaluations on the LJSpeech and LibriTTS datasets show that the method proposed in this application surpasses the best publicly available systems in terms of naturalness. By adopting the latest diffusion model generation inverse problem solving algorithm, DiffVoice achieves state-of-the-art performance in text-based speech editing and zero-shot speech cloning.

[0057] 1. Introduction

[0058] Diffusion models (DMs) have demonstrated promising performance on image and audio generation tasks. They have also been applied to non-autoregressive text-to-speech synthesis. Most previous work in this area has been based on diffusion acoustic models, generating log-mel spectrograms from given text input.

[0059] Use DMs to directly calculate x0∈R d The data density p(x0) is used to model the data density p(x0), which may cause several problems in application. First, the intermediate latent space variable x t is constrained to have the same shape as x0. Since DM sampling requires repeated evaluation of the score estimator s θ (xt ,t), which can be very inefficient. Secondly, since DMs try to capture all patterns in p(x0), they tend to spend a lot of modeling power on imperceptible details of the data. Latent Diffusion Models (LDMs) were proposed to alleviate these problems. First, an encoder f Φ (.) Encode the data into a latent space code z0=f Φ (x0), and then use DMs to calculate the latent space density p Φ (z0) modeling, and finally decoder g ψ (z0) Generate data.

[0060] The proposed DiffVoice model is a new acoustic model based on LDMs. The autoencoder in DiffVoice is a VAE-GAN with dynamic downsampling in time. Encoding is latent space encoding Where N is the number of frames and M is the number of phonemes. With the help of dynamic rate downsampling, DiffVoice can jointly model phoneme duration and mel-spectrograms using a single diffuse model in the latent space. In contrast, previous work on diffuse acoustic models relies on an additional duration predictor and works directly on mel-spectrograms.

[0061] DiffVoice demonstrates high performance in acoustic modeling, both on the single-speaker dataset LJSpeech and the more challenging multi-speaker dataset LibriTTS. Because duration is jointly modeled with other aspects of speech, general inverse problem-solving algorithms with DMs can be directly combined with DiffVoice to solve inverse problems in speech synthesis, including text-based speech editing and zero-shot voice cloning.

[0062] Text-based speech editing systems allow users to edit the content of a recorded speech waveform by providing the original text and the modified text. Modifications may include inserting, deleting, and replacing words. The goal of such systems is to synthesize the modified portion of the audio into audio that is highly coherent and natural. The examples of this application demonstrate that DiffVoice can achieve state-of-the-art performance on this task without the need for specialized model design and training procedures as in many previous works. The examples of this application further demonstrate that DiffVoice can solve the zero-sample speech cloning problem with state-of-the-art performance by treating it as a speech continuation or insertion problem.

[0063] Audio samples and further information are provided in the online supplementary material at https: / / zjlww.github.io / diffvoice / . The present application strongly recommends that readers listen to these audio samples.

[0064] 2. Filtering

[0065] Assumptions is a logarithmic Mel spectrum, where N is the number of frames, D mel is the size of the Mel filter bank. Assume w∈Σ M is the corresponding phoneme sequence, and Σ is the set of all phonemes.

[0066] 2.1. Dynamic Downsampling of Speech

[0067] DiffVoice uses a variational autoencoder to encode speech into a compact latent space. In this section, the embodiments of the present application will describe in detail the encoding and decoding of speech signals.

[0068] The embodiment of the present application can rely on the CTC ASR model trained based on phoneme sequences to obtain the alignment between w and y. Of course, other existing alignment methods are also possible and will not be described in detail here. The embodiment of the present application uses the minimum CTC to ensure that each phoneme generates one and only one spike. Assume that for each in w Its position in the CTC arrangement is Obviously, a is strictly increasing. Let a0:=0, d i :=(a i -a i-1 ). Positive sequence Contains the approximate duration of the phonemes.

[0069] Approximate posterior q Φ (z0|y,a) is defined as follows (see Figure 3a ). y is first processed by the encoder Conformer. Then, the output frame rate latent space representation By collecting (a i ) M i＝1 The frame values ​​are downsampled to Then perform linear projection and splitting on e~ to generate the mean μ∈ and log variance Finally, for

[0070] The prior p(z0) is defined as the standard normal density.

[0071]

[0072] Conditional density p ψ (y|z0,a) is defined as follows (see Figure 3b ). First, upsample according to the arrangement method a in and Then z~0 is sent to the decoder Conformer to get Then linearly project to Now the present application embodiment will p ψ It is defined as follows, where b∈(0,∞) is a hyperparameter that is adjusted during training.

[0073]

[0074] By optimizing L VAE =E (y,a) [L Φ,ψ (y,a)] to train the variational autoencoder, where

[0075]

[0076] Figure 3a 、 Figure 3b and Figure 3c Figure 2 shows the structure of the proposed DiffVoice system. M is the length of the phoneme sequence w and N is the number of frames.

[0077] 2.2. Adversarial Training

[0078] Training only with the ELBO described in Section 2.1 will result in the spectrograms generated by the autoencoder lacking high-frequency details. This embodiment of the application adds an adversarial loss to ensure high-fidelity reconstruction.

[0079] First, the VAE is trained until convergence. Then, the embodiment of the present application continues to train with an additional adversarial loss. During the adversarial training phase, the spectrum decoder ( Figure 3b ) is expanded, such as Figure 4 As shown in Figure 2. A stack of randomly initialized 2D convolutions, and an additional linear projection is added to generate the spectral residual. The 2D convolution is regularized by spectral norm and interleaved with Leaky ReLU activations. The discriminator is also a 2D convolution with spectral norm, interleaved with Leaky ReLU activations.

[0080] Figure 4 Schematic diagram of the decoder modifications for adversarial training. Blue blocks are added to the decoder and randomly initialized.

[0081] The random mapping (y, a)|→y^ is called the generator G(y, a), and D(.) is called the discriminator. The embodiment of the present application uses the minimum mean square loss L G ,L D , plus the feature matching loss L feat G To train G and D. The total loss L in adversarial training adv It's L G ,L D ,L feat G ,L VAE The weighted sum of each weight needs to be adjusted during the training process. The method of adjusting the weight is relatively existing and will not be described here. The total loss L in adversarial training adv It's L G ,L D ,L feat G ,L VAE The weighted sum of .

[0082]

[0083]

[0084]

[0085] The output of the discriminator is a two-dimensional matrix, D j,k is the j,kth value. In the feature matching loss L feat G where L is the number of layers in D. Presentation Layer The hidden feature map of element.

[0086] 2.3. Latent Space Diffusion Model

[0087] In this section, the present application embodiment will describe the latent space diffusion model ( Figure 3c After the speech autoencoder described in Sections 2.1 and 2.2 is fully trained, the embodiment of the present application freezes its weights and uses it to encode speech into a latent space representation.

[0088] In order to model the integer time sequence d using the diffusion model, the embodiment of the present application first samples u~Uniform[0,1) M Perform uniform dequantization on d

[29] , and then define d ～ =du. The present embodiment further takes l j :=log(d ～ j+c0)+c1, where c0, c1 are artificially chosen constants to normalize the distribution. The connection with z0 is defined as The goal of the latent space diffusion model is to sample from the density p0(x0|w).

[0089] The embodiment of this application uses stochastic differential equations (SDE) for generative modeling.

[0090]

[0091] X t It is R M×(D+1) A random process in , t∈[0,1], B t is an R M×(D+1) The standard Brownian motion of the value. The transition density of Equation 1 is given by

[0092]

[0093] Text θ (x t ,t,w) is trained using denoised score matching, so that In the embodiment of the present application, the time λ t The weighting of is the same as in the related art.

[0094]

[0095] During inference, a latent space diffusion model is first used to extract x0 from p0(x0|w). Then, x0 is split into l and z0. The present embodiment then reconstructs the permutation a from l and decodes the log-Mel spectrum y^ from (z0, a) using a spectrum decoder.

[0096] 2.4. Solving the Inverse Problem with DiffVoice

[0097] Assume o = A(x0)∈R O , where A is fine-tunable. The embodiments of this application have To sample from p0(x0|o,w). This embodiment of the application also requires a The estimated value of this application embodiment is found to be approximately defined as E[x0|x t ,w] of π θ (x t ,t,w).

[0098]

[0099] Using some weighting functions ξ(t): [0, 1] → [0, ∞), the embodiment of the present application adopts

[0100]

[0101] 2.4.1. Text-based voice editing and zero-shot voice cloning

[0102] The embodiment of the present application only describes the voice replacement algorithm of text-based continuous phonemes in this section, because other forms of editing work are similar.

[0103] Given a log-Mel spectrogram y and the corresponding phoneme sequence w, as described in Section 2.1, encode them as mean μ and variance σ. Split w into three segments [w (A) ;w (B) ;w (C) ], lengths are M A ,M B ,M C . Use w- (B) ∈Σ M-B Replace segment w (B) , we get w-=[w (A) ;w- (B) ;w (C) ]. The task of the model is to generate a new x-0 and a corresponding new spectrogram y-, in addition to modifying w- (B) Outside the corresponding section, it is equal to y.

[0104] With DiffVoice, text-based voice replacement can be solved in the same way as image smearing. For ease of description, the embodiment of the present application defines the mask selection function M as [u (A) ;u (B) ;u (C) ]|→[u (A) ;u (C) ] mapping, for any matrix u, where For any K∈N+.

[0105] definition Let o=M(μ ～ ) and take Approximate value of

[0106]

[0107] Among them, subtraction and division are performed in element units. Finally, the embodiment of the present application can use the modified fraction To solve the inverse SDE or probability flow ODE, so that from the conditional density p(x - t|o,w - ) is approximate sampling.

[0108] Zero-shot speech cloning can be viewed as a special form of speech insertion, i.e., synthesizing an entire new sentence. For example, let w (A) is the phoneme sequence of the reference speech, w (C) is an empty sequence, w -(B) The embodiment of the present application refers to this method as prompt-based zero-sample speech cloning.

[0109] 3. Experiments and Results

[0110] The Conformer architecture is widely used in DiffVoice. The hyperparameters of all modules in DiffVoice can be found in Figure 5 Found in, where Figure 5 Shows the hyperparameters of the Conformer module in DiffVoice.

[0111] In the experiments in this application example, all dropout rates were set to 0.1. All CTC alignment models were trained on the same dataset as the acoustic model. Training with a minimal CTC topology and forced alignment were implemented using the k2 toolkit. Further details on model training can be found in the online supplement.

[0112] The embodiments of the present application rely on mean opinion scores (MOS) for evaluation. Listeners were asked to rate the synthesized speech on a scale of 1.0 to 5.0 with a step size of 0.5. In all evaluations in this section, 15 listeners scored 20 randomly sampled corpora. In all evaluations, multiple stimuli with the same text content were presented in a single trial and presented to the listeners. All MOS scores are reported with a 95% confidence interval.

[0113] 3.1. Text-to-speech for a Single Speaker

[0114] This embodiment of the application uses LJSpeech to evaluate the performance of text-to-speech on a single-speaker speech dataset. This embodiment of the application sets aside the same 500 sentences as VITS and GradTTS for testing.

[0115] This example compares the proposed model with the best publicly available model. For VITS and GradTTS, this example uses their official public implementations and pre-trained weights. For GradTTS, the temperature is set to 1.5, and the sampler is an Euler ODE sampler with 100 steps.

[0116] Figure 6The results of speech synthesis on LJSpeech are shown.

[0117] For DiffVoice, this application example uses a 100-step Euler-Maruyama sampler. For FastSpeech 2, this application example uses the implementation in ESPN 2, where HiFi-GAN is fine-tuned together with the FastSpeech 2 model to improve performance. For GradTTS and DiffVoice, this application example uses the pre-trained HiFi-GAN (v1) to generate waveforms. 90 sentences were combined for evaluation. The results can be found in Figure 6 Found in.

[0118] 3.2. Multi-speaker text-to-speech

[0119] This example uses a combination of "train-clean-100" and "train-clean-360" from LibriTTS to evaluate text-to-speech performance on a multi-speaker dataset. This dataset contains approximately 245 hours of speech from 1,151 speakers. This example randomly selects 500 instances for evaluation. All audio is down-sampled to 16 kHz.

[0120] The embodiments of the present application extend DiffVoice with a speaker encoder trained together with a score estimator, hereinafter referred to as DiffVoice (encoder). The speaker encoder is a Conformer with input x0. The speaker embedding is obtained by mean-pooling the output of the Conformer. The embedding is repeated in time and connected to the score estimators of other inputs. For VITS and FastSpeech 2, the embodiments of the present application use their multi-speaker extensions in ESPN2 to adjust the X-vector. For YourTTS and Meta-StyleSpeech, the embodiments of the present application use their official public implementations and pre-trained weights. For FastSpeech 2, Meta-StyleSpeech and DiffVoice, the embodiments of the present application use a general HiFi-GAN trained on "train-clean-460" to synthesize waveforms.

[0121] Used to evaluate text-to-speech performance in a multilingual environment. This example reports MOS, Similarity MOS (Sim-MOS), and Speaker Encoder Cosine Similarity (SECS). SECS scores are calculated using the speaker encoder in Resemblyzer.

[0122] For the speakers we see, we randomly selected 40 sentences from 40 different speakers in "train-clean-460". We used speaker-level Xvectors in FastSpeech 2 and VITS, and random reference audio in the DiffVoice (encoder) model. The evaluation results can be found in Figure 7 Please note that this embodiment of the application uses the ground truth audio as a reference for similarity testing.

[0123] To evaluate zero-shot voice cloning, this embodiment of the present application uses the same 21 reference audios as YourTTS. Five sentences are synthesized for each reference audio. In the zero-shot voice cloning evaluation, this embodiment of the present application uses corpus-level X-vectors for the X-vector model. DiffVoice (Prompt) is a system that uses prompt-based zero-shot voice cloning, which is described in Section 3.3. To sample from this model, this embodiment of the present application uses a 300-step Euler-Maruyama sampler.

[0124] Figure 7 Shows the results of speech synthesis with known speakers on LibriTTS.

[0125] Figure 8 Shows the results of zero-shot voice cloning on LibriTTS.

[0126] 3.3. Text-based voice editing

[0127] The performance of text-based speech patching is evaluated by comparing it with samples from RetrieverTTS, which is a special case of replacement. This embodiment uses the same SDE sampler as in DiffVoice(Prompt). The MOS values ​​of the three different mask durations can be found in Figure 9 Among them, Figure 9 Results of text-based speech inpainting are shown.

[0128] 4. Conclusion and Future Work

[0129] DiffVoice demonstrates strong performance on text-to-speech synthesis, speech editing, and voice cloning. However, sampling remains relatively slow, requiring hundreds of neural function evaluations. Better SDE samplers and other acceleration methods for diffusion models could potentially reduce sampling time while maintaining the same sampling quality. Using better intermediate representations other than log-mel spectrograms and applying improved waveform generation techniques could also improve performance.

[0130] In other embodiments, embodiments of the present invention further provide a non-volatile computer storage medium storing computer-executable instructions, wherein the computer-executable instructions can execute the voice editing method in any of the above method embodiments;

[0131] As an embodiment, the non-volatile computer storage medium of the present invention stores computer-executable instructions, and the computer-executable instructions are configured as follows:

[0132] Inputting the spectrum of the original speech, the phoneme sequence of the original speech, and the alignment relationship between the phoneme sequence and the spectrum into the encoder for encoding to obtain the original latent space code;

[0133] Dividing the original latent space code into multiple segments, making one of the multiple segments correspond to the phoneme sequence that needs to be modified, and keeping the other segments of the multiple segments unchanged;

[0134] Obtaining an edited phoneme sequence, and generating a new segment based on the edited phoneme sequence using a diffusion model, wherein a diffusion process of the diffusion model is guided by the other segments;

[0135] A decoder is used to decode the new latent space code composed of the new segment and the other segments to obtain an edited spectrum.

[0136] The non-volatile computer-readable storage medium may include a program storage area and a data storage area, wherein the program storage area may store an operating system and application programs required for at least one function; the data storage area may store data created based on the use of the voice editing device, etc. In addition, the non-volatile computer-readable storage medium may include high-speed random access memory and may also include non-volatile memory, such as at least one disk storage device, flash memory device, or other non-volatile solid-state memory device. In some embodiments, the non-volatile computer-readable storage medium may optionally include a memory remotely located relative to the processor, and these remote memories may be connected to the voice editing device via a network. Examples of the aforementioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0137] An embodiment of the present invention further provides a computer program product, which includes a computer program stored on a non-volatile computer-readable storage medium. The computer program includes program instructions. When the program instructions are executed by a computer, the computer executes any of the above-mentioned voice editing methods.

[0138] Figure 7 Schematic diagram of the structure of an electronic device provided by an embodiment of the present invention. Figure 7 As shown, the device includes: one or more processors 710 and a memory 720, Figure 7 A processor 710 is used as an example. The device of the speech editing method may further include: an input device 730 and an output device 740. The processor 710, the memory 720, the input device 730 and the output device 740 may be connected via a bus or other means. Figure 7 The example of a bus connection is shown. Memory 720 is the aforementioned non-volatile computer-readable storage medium. Processor 710 executes various functional applications and data processing of the server by running the non-volatile software programs, instructions, and modules stored in memory 720, thereby implementing the voice editing method of the aforementioned method embodiment. Input device 730 can receive input digital or character information and generate key signal input related to user settings and function control of the communication compensation device. Output device 740 may include a display device such as a display screen.

[0139] The above-mentioned product can execute the method provided by the embodiment of the present invention, and has the functional modules and beneficial effects corresponding to the execution method. For technical details not fully described in this embodiment, please refer to the method provided by the embodiment of the present invention.

[0140] As an embodiment, the electronic device is applied to a voice editing device, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to:

[0141] Inputting the spectrum of the original speech, the phoneme sequence of the original speech, and the alignment relationship between the phoneme sequence and the spectrum into the encoder for encoding to obtain the original latent space code;

[0142] Dividing the original latent space code into multiple segments, making one of the multiple segments correspond to the phoneme sequence that needs to be modified, and keeping the other segments of the multiple segments unchanged;

[0143] Obtaining an edited phoneme sequence, and generating a new segment based on the edited phoneme sequence using a diffusion model, wherein a diffusion process of the diffusion model is guided by the other segments;

[0144] A decoder is used to decode the new latent space code composed of the new segment and the other segments to obtain an edited spectrum.

[0145] The electronic devices of the embodiments of the present application exist in various forms, including but not limited to:

[0146] (1) Mobile communication devices: These devices are characterized by their mobile communication capabilities and primarily provide voice and data communications. These terminals include smartphones (e.g., iPhones), multimedia phones, feature phones, and low-end phones.

[0147] (2) Ultra-mobile personal computer devices: These devices fall under the category of personal computers, have computing and processing capabilities, and generally also have mobile Internet access. These terminals include PDAs, MIDs, and UMPCs, such as the iPad.

[0148] (3) Portable entertainment devices: These devices can display and play multimedia content. These devices include audio and video players (such as iPods), handheld game consoles, e-books, smart toys, and portable car navigation devices.

[0149] (4) Server: A device that provides computing services. The server consists of a processor, hard disk, memory, system bus, etc. The server is similar to a general computer architecture, but because it needs to provide highly reliable services, it has higher requirements in terms of processing power, stability, reliability, security, scalability, and manageability.

[0150] (5) Other electronic devices with data interaction functions.

[0151] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units. That is, they may be located in one place or distributed across multiple network units. Some or all of the modules may be selected based on actual needs to achieve the objectives of the present embodiment. Persons of ordinary skill in the art will be able to understand and implement the present invention without inventive effort.

[0152] Through the description of the above embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus the necessary general hardware platform, or of course, by hardware. Based on this understanding, the essence of the above technical solution or the part that contributes to the existing technology can be embodied in the form of a software product. The computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a magnetic disk, an optical disk, etc., and includes a number of instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods of each embodiment or certain parts of the embodiment.

[0153] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the various embodiments of the present invention.

Claims

1. A voice editing method, comprising: Inputting the spectrum of the original speech, the phoneme sequence of the original speech, and the alignment relationship between the phoneme sequence and the spectrum into the encoder for encoding to obtain the original latent space code; Dividing the original latent space code into multiple segments, making one of the multiple segments correspond to the phoneme sequence that needs to be modified, and keeping the other segments of the multiple segments unchanged; Obtaining an edited phoneme sequence, and generating a new segment based on the edited phoneme sequence using a diffusion model, wherein a diffusion process of the diffusion model is guided by the other segments; A decoder is used to decode the new latent space code composed of the new segment and the other segments to obtain an edited spectrum.

2. The method according to claim 1, wherein The diffusion model performs a reverse diffusion process starting from Gaussian noise, and generates a new latent space code through repeated iterations. The diffusion process of the diffusion model is guided by the other segments to basically keep the other segments in the new latent space code consistent with the other segments in the original latent space code.

3. The method according to claim 1, wherein The alignment relationship between the phoneme sequence and the spectrum is obtained in the following manner: An alignment relationship between the phoneme sequence and the spectrum is obtained by using a connection-time classification automatic speech recognition model trained based on a phoneme sequence.

4. The method according to claim 1, wherein The input to the encoder for encoding includes: The speech signal is encoded into a latent space using a variational autoencoder enhanced by adversarial training to obtain a phoneme-level latent space representation.

5. The method according to claim 4, wherein The variational autoencoder is a variational autoencoder-generative adversarial network with dynamic downsampling in time.

6. The method according to claim 4, wherein: The total training loss of the variational autoencoder-generative adversarial network is the weighted sum of the minimum mean square loss of the generator, the minimum mean square loss of the discriminator, the feature matching loss and the variational autoencoder loss.

7. The method according to claim 5, wherein: The diffusion model jointly models phoneme duration and mel spectrum using a single diffusion model in latent space based on dynamic rate downsampling.

8. An electronic device comprising: At least one processor, and a memory communicatively connected to the at least one processor, wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the steps of the method according to any one of claims 1 to 7.

9. A storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

Images