A zero-shot voice cloning method and apparatus
By combining a text encoder, speaker encoder, and detail encoder model with low-level and high-level prosodic information, the problems of resource consumption and inaccurate generation in zero-sample speech cloning are solved, and high-quality speech synthesis is achieved.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- INST OF ACOUSTICS CHINESE ACAD OF SCI
- Filing Date
- 2025-02-24
- Publication Date
- 2026-05-19
AI Technical Summary
Existing zero-sample speech cloning technology suffers from problems such as high resource consumption, information compression, and inaccurate sample generation. In particular, it is difficult to generate high-quality target speaker speech when there is a lack of text and speech pairing data.
A combined model of text encoder, speaker encoder and detail encoder is adopted. By adding low-level and high-level prosodic information, and combining noise terms and real speech acoustic features, the detail encoder is trained to generate high-quality target audio. The model parameters are optimized by using weight factors and objective functions.
It improves the accuracy of generated samples, reduces the dependence on resources, and takes into account text content, speaker features and prosodic information, achieving high-quality zero-sample speech synthesis.
Smart Images
Figure CN120071891B_ABST
Abstract
Description
Technical Field
[0001] This specification relates to the field of speech synthesis technology, and more particularly to a zero-sample speech cloning method and apparatus. Background Technology
[0002] Zero-shot speech cloning, as an important branch of multi-speaker speech generation, allows models to clone or mimic a speaker's voice using reference speech when no specific speaker samples are available. Existing techniques either maintain the model using a limited list of speakers and then fine-tune it through thousands of training steps, a method that is labor-intensive and resource-intensive; or, to make the generated samples more realistic in style and timbre, use high-dimensional global speaker embedding (GSE) that stores a large amount of speech feature data. This method compresses speaker information, increasing the difficulty for the model to generate the target speaker's speech based on it; or, train an audio compression model using a large amount of text and speech pairing data, which also requires a large amount of text and speech pairing data; or, when using an ordinary differential equation model as the generation model, use random noise as training samples, resulting in the neural network's predicted variance having a high curvature, which severely affects the accuracy of the generated samples, leading to inaccurate samples generated by the trained model. Summary of the Invention
[0003] This application describes a zero-sample speech cloning method and apparatus that can solve the above-mentioned technical problems.
[0004] According to a first aspect, a zero-sample speech cloning method is provided, the method comprising:
[0005] The reference audio text and the target audio text are input into a text encoder to obtain the first acoustic feature. The text encoder is used to predict low-level prosodic information related to the pronunciation content of the target audio text from the reference audio text. The low-level prosodic information is added to the phoneme string of the target audio text to obtain the first acoustic feature.
[0006] The acoustic feature context of the reference audio and the first acoustic feature are input into the speaker encoder to obtain the second acoustic feature. The speaker encoder is used to add high-level prosodic information related to the speaker to the first acoustic feature according to the acoustic feature context of the reference audio to obtain the second acoustic feature.
[0007] The acoustic feature context of the second acoustic feature with added noise, the first acoustic feature, the second acoustic feature, and the reference audio are concatenated and then input into the detail encoder to obtain the acoustic feature difference quantity.
[0008] Based on the acoustic feature difference and the second acoustic feature, a predicted Mel spectrum is obtained, and the predicted Mel spectrum is converted into audio to obtain the target audio corresponding to the target audio text.
[0009] Based on the above further embodiments, it also includes:
[0010] The reference audio text sample set and the target audio text sample set are input into the text encoder to obtain the first acoustic feature sample set;
[0011] The first acoustic feature sample set and the reference audio sample set are input into the speaker encoder to obtain the second acoustic feature sample set;
[0012] A noise term is added to the second acoustic feature sample set as initial data. Then, the first acoustic feature sample set, the acoustic feature context of the reference audio, and the second acoustic feature sample set are concatenated as input data for the detail encoder to train the detail encoder. The second acoustic feature sample set has a correspondence with the acoustic features of real speech.
[0013] Based on the above further embodiments, it also includes:
[0014] The real speech acoustic features are used as target data, and the real speech acoustic features are acoustic features extracted from real speech signals of the target audio text sample set.
[0015] The initial data and the target data are weighted and mixed by a weighting factor to obtain intermediate data. During the training process, the weighting factor is adjusted so that the intermediate data gradually reaches the target data from the initial data.
[0016] The intermediate data is concatenated with the acoustic feature context of the first acoustic feature sample set, the reference audio sample set, and the second acoustic feature sample set as input data for the detail encoder. The detail encoder is then trained so that the acoustic feature difference output by the detail encoder, combined with the second acoustic feature sample set and the noise term, has the smallest difference from the real speech acoustic features.
[0017] Based on the above further embodiments, the step of concatenating the intermediate data with the acoustic feature context of the first acoustic feature sample set, the reference audio sample set, and the second acoustic feature sample set as input data for the detail encoder, and training the detail encoder so that the acoustic feature difference output by the detail encoder, combined with the second acoustic feature sample set and noise terms, has the smallest difference from the real speech acoustic features, specifically includes:
[0018] An objective function is constructed using the real speech acoustic features, the second acoustic feature sample set, and the acoustic feature difference quantity. The objective function reflects the difference between the acoustic feature difference quantity output by the detail encoder, combined with the second acoustic feature sample set and the noise term, and the real speech acoustic features.
[0019] The parameters of the detail encoder are determined through iterative updates, with the goal of minimizing the function value of the objective function.
[0020] Based on the above further embodiments, the objective function is expressed as:
[0021]
[0022] Mel represents the acoustic features of real speech. It is the second acoustic feature sample set. It is a detail encoder. These are the input values of the detail encoder. It is the weighting factor of the iterative process, where T is the number of iterations. It is the expected value;
[0023] in,
[0024] Noise is the noise term. It is the first acoustic feature sample set. It is the acoustic feature context of the reference audio sample set.
[0025] Based on the above further embodiments, the low-level prosodic information is low-level prosodic information, which includes information on pitch, duration, and intensity in the text.
[0026] The advanced prosodic information refers to advanced prosodic features, which include shifts in intonation, rhythm, syllable duration, and speech intensity influenced by the speaker and emotional information.
[0027] According to a second aspect, a zero-sample speech cloning device is provided, the device comprising:
[0028] The first processing module is used to input the reference audio text and the target audio text into a text encoder to obtain a first acoustic feature. The text encoder is used to predict low-level prosodic information related to the pronunciation content of the target audio text from the reference audio text, and add the low-level prosodic information to the phoneme string of the target audio text to obtain the first acoustic feature.
[0029] The second processing module is used to input the reference audio and the first acoustic feature into the speaker encoder to obtain the second acoustic feature. The speaker encoder is used to add high-level prosodic information related to the speaker to the first acoustic feature according to the acoustic feature context of the reference audio to obtain the second acoustic feature.
[0030] The third processing module is used to concatenate the acoustic feature context of the second acoustic feature with added noise, the first acoustic feature, the second acoustic feature and the reference audio, and input it into the detail encoder to obtain the acoustic feature difference quantity.
[0031] The fourth processing module is used to obtain the predicted Mel spectrum based on the acoustic feature difference and the second acoustic feature, convert the predicted Mel spectrum into audio, and obtain the target audio corresponding to the target audio text.
[0032] Based on the above further embodiments, a fifth processing module is also included, specifically used to input the reference audio text sample set and the target audio text sample set into the text encoder to obtain the first acoustic feature sample set;
[0033] The first acoustic feature sample set and the reference audio sample set are input into the speaker encoder to obtain the second acoustic feature sample set;
[0034] A noise term is added to the second acoustic feature sample set as initial data. Then, the first acoustic feature sample set, the acoustic feature context of the reference audio, and the second acoustic feature sample set are concatenated as input data for the detail encoder to train the detail encoder. The second acoustic feature sample set has a correspondence with the acoustic features of real speech.
[0035] Based on the above further embodiments, the fifth processing module is specifically used to use the real speech acoustic features as target data, wherein the real speech acoustic features are acoustic features extracted from the real speech signals of the target audio text sample set;
[0036] The initial data and the target data are weighted and mixed by a weighting factor to obtain intermediate data. During the training process, the weighting factor is adjusted so that the intermediate data gradually reaches the target data from the initial data.
[0037] The intermediate data is concatenated with the acoustic feature context of the first acoustic feature sample set, the reference audio sample set, and the second acoustic feature sample set as input data for the detail encoder. The detail encoder is then trained so that the acoustic feature difference output by the detail encoder, combined with the second acoustic feature sample set and the noise term, has the smallest difference from the real speech acoustic features.
[0038] Based on the above further embodiments, the third processing module is specifically used to construct an objective function using the real speech acoustic features, the second acoustic feature sample set, and the acoustic feature difference quantity. The objective function reflects the difference between the acoustic feature difference quantity output by the detail encoder, combined with the second acoustic feature sample set and the noise term, and the real speech acoustic features.
[0039] The parameters of the detail encoder are determined through iterative updates, with the goal of minimizing the function value of the objective function.
[0040] Based on the above further embodiments, the objective function is expressed as:
[0041]
[0042] Mel represents the acoustic features of real speech. It is the second acoustic feature sample set. It is a detail encoder. These are the input values of the detail encoder. It is the weighting factor of the iterative process, where T is the number of iterations. It is the expected value;
[0043] in,
[0044] Noise is the noise term. It is the first acoustic feature sample set. It is the acoustic feature context of the reference audio sample set.
[0045] Based on the above further embodiments, the low-level prosodic information is low-level prosodic information, which includes information on pitch, duration, and intensity in the text.
[0046] The advanced prosodic information refers to advanced prosodic features, which include shifts in intonation, rhythm, syllable duration, and speech intensity influenced by the speaker and emotional information.
[0047] According to a third aspect, a computer storage medium is provided, on which a computer program is stored, which, when executed by one or more processors, implements the zero-sample speech cloning method as described in any of the above technical solutions.
[0048] According to a fourth aspect, an electronic device is provided, including a memory and one or more processors, wherein a computer program is stored on the memory, and the computer program, when executed by the one or more processors, implements the zero-sample speech cloning method as described in any of the above technical solutions.
[0049] In the systems and methods provided in the embodiments of this specification, a large amount of text and speech pairing data is not required. Features with a clear correspondence to the acoustic features of real speech are used as training sets to train the model, which improves the accuracy of generated samples. Furthermore, multiple key factors in speech synthesis are considered, including text content, speaker features, and prosodic information of speech. Through advanced neural network structures and training strategies, high-quality zero-sample speech synthesis is achieved. Attached Figure Description
[0050] To more clearly illustrate the technical solutions of the embodiments of this application, the drawings used in the following description of the embodiments will be briefly introduced. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0051] Figure 1 A schematic diagram illustrating the transmission path of the neural network estimation provided in the embodiments of this specification is shown.
[0052] Figure 2 This diagram illustrates the framework of the zero-sample speech cloning model provided in the embodiments of this specification.
[0053] Figure 3 This diagram illustrates the performance of the model based on the neural ordinary differential equation under different number of solution steps provided in the embodiments of this specification.
[0054] Figure 4 A flowchart illustrating the zero-sample speech cloning method provided in the embodiments of this specification is shown.
[0055] Figure 5 A schematic diagram of the zero-sample speech cloning device provided in the embodiments of this specification is shown. Detailed Implementation
[0056] The solution provided in this specification will now be described with reference to the accompanying drawings.
[0057] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions in the embodiments of this application will be described below with reference to the accompanying drawings.
[0058] In the description of the embodiments of this application, the words "exemplary," "for example," or "for instance" are used to indicate examples, illustrations, or explanations. Any embodiment or design described as "exemplary," "for example," or "for instance" in the embodiments of this application should not be construed as being more preferred or advantageous than other embodiments or designs. Specifically, the use of the words "exemplary," "for example," or "for instance" is intended to present the relevant concepts in a specific manner.
[0059] In the description of the embodiments of this application, the term "and / or" is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, B existing alone, and A and B existing simultaneously. Furthermore, unless otherwise stated, the term "multiple" means two or more.
[0060] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the indicated technical features. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature. The terms "comprising," "including," "having," and their variations all mean "including but not limited to," unless otherwise specifically emphasized.
[0061] Multi-speaker speech generation technology has been able to synthesize speech with quality comparable to human pronunciation, and this technology has been widely used in voice assistants, audiobooks, video dubbing, and other fields. Zero-sample speech cloning, as an important branch of multi-speaker speech generation, allows a model to clone or mimic a speaker's voice using only reference speech, even without specific speaker samples.
[0062] To achieve zero-shot speech cloning, existing methods include: Method 1: Maintaining a model using a limited list of speakers and fine-tuning the model through thousands of training steps to achieve a zero-shot speech clone. However, each zero-shot speech clone requires model fine-tuning, which is very labor-intensive and resource-intensive. Method 2: Extracting Global Speaker Embeddings (GSEs) from reference speech. GSEs capture unique speaker attributes such as timbre, emotion, and accent. Using the extracted GSEs to train a speaker recognition model achieves zero-shot speech cloning. However, even with high-dimensional GSEs, the entire reference audio segment is compressed into a vector, resulting in information compression and insufficient speaker information for the model, increasing the difficulty of generating speech that matches the target speaker's style and timbre. Method 3: Using fine-grained unseen speaker information extracted from reference speech as samples. These features can be used as training samples to improve model performance. This method can capture unique speaker attributes such as timbre, emotion, and accent, or use context learning to complete or fill in reference speech as training samples. This approach can help the model better understand and reproduce the speaker's style and timbre, but it uses an audio compression model to process speech data and relies on a large amount of paired data to learn the mapping from text to discrete speech representations.
[0063] In situations where sufficient text-speech pairing data is unavailable, existing method four learns to repair masked audio by utilizing context and the text to be filled in within the acoustic feature space of the speech. This method employs flow matching (FM) techniques to train an Ordinary Differential Equation (ODE) model to reconstruct the masked Mel-spectrum from random noise. It effectively handles situations where sufficient text-speech pairing data is unavailable, without relying on large amounts of pairing data to learn the mapping from text features to speech representations. Existing method four includes the following steps:
[0064] For a generation task, the ordinary differential equation model can be represented as follows:
[0065]
[0066] The above equation simulates the state from the initial state. To the target state A continuous dynamic process, This represents the intermediate state at time t. It represents the direction of transmission from the initial state to the target state, that is, the difference between the initial state and the target state.
[0067]
[0068] The above formula indicates that the direction of state change is determined by the function. Decision, function It can be considered as a direction estimator.
[0069] To fit the orientation estimator, a neural network is used. To approximate The goal of training is to minimize the following loss function:
[0070]
[0071] in, The expected value is represented by the loss function, which is applied over the time interval [0,T]. and The integral of the square of the difference between them. When the neural network After training is complete, you can start from the initial state. Begin by iterating Obtain the target state .
[0072] because It is causal, only with Because it is related to causality, its specific expression is difficult to give. In some methods, if we start from a non-causal perspective, for non-causal intermediate states... of It can be explicitly defined as:
[0073]
[0074]
[0075] here It is the intermediate state at time t. From arrive The direction of transmission. Since ODE is a causal model, The neural network for fitting the target will mimic... The marginal probability gives an estimated transmission direction distribution. This... The method of training to fit a target is called FM (Factor-Driven Model). However, the fact that the marginal probabilities of the transmission direction estimated by the neural network are the same as those of the actual transmission path direction does not mean that the two states are identical. This is because, in deep learning, neural networks are typically used to approximate the true distribution, but the predicted distribution of the neural network may differ from the true distribution. This difference is not only reflected in the marginal probabilities but also in the overall structure of the distribution. Sometimes, even if the marginal probabilities are the same, the overall structure of the two distributions can still be very different. Moreover, the predicted distribution of the neural network may be affected by various factors such as model structure, training data, and optimization algorithms, all of which can lead to differences between the predicted and true distributions. Therefore, the fact that the marginal probabilities of the transmission direction estimated by the neural network are the same as those of the actual transmission path direction only indicates that the probability distributions in each dimension are the same, but it does not guarantee that the two distributions are completely identical.
[0076] To reduce the gap between the transmission path estimated by the neural network and the actual transmission path, this study investigates the differences between the neural network and the actual path when handling different types of coupling, such as... Figure 1 This illustrates the differences in how neural networks handle different types of coupling changes. The four parts in the figure correspond to... and The coupling relationships between them include independent coupling and redundant coupling. The top left diagram illustrates this. and Independently coupled and real transmission paths Brown lines indicate from arrive Independent coupling changes As can be seen, the transmission paths in each dimension are relatively independent with minimal overlap. The upper right figure shows... and Independent coupling between them, predicted by the neural network from arrive Changes The transmission path predicted by the neural network is relatively straight, indicating that the neural network can learn the actual transmission path quite accurately. (The lower left figure shows...) and The repeated coupling and real transmission paths between them show that there are many intersections in the transmission paths between different dimensions, indicating that from arrive The coupling relationships between them are more complex. The diagram on the lower right illustrates this. and Repeated coupling between them, predicted by the neural network from arrive Changes As can be seen, the transmission paths predicted by the neural network appear curved, indicating that when dealing with complex coupling relationships, the neural network may produce errors, leading to a distributional discrepancy between the predicted and actual paths. In summary, the degree of intersection between actual transmission paths depends on... and The coupling relationship between them. When and When coupling is repeated, the degree of overlap is high, and the transmission path learned by the corresponding neural network is less accurate. and The more independent the coupling relationship, the smaller the degree of overlap, and the more accurate the transmission path learned by the corresponding neural network. Existing technologies use random noise as the initial state. Using the target Mel spectrum as However, the paths of random noise and the target Mel spectrum intersect significantly, resulting in repetitive coupling between them. Therefore, random noise is used as the initial state. When training neural networks, the predicted transmission paths typically have high curvature, which severely impacts the accuracy of generated samples. To address this issue, it's necessary to design neural network architectures capable of handling high-curvature paths, or employ other techniques to reduce path curvature, thereby improving the accuracy of generated samples.
[0077] To solve the above problems, such as Figure 2 As shown, this invention proposes a training method for a zero-shot speech cloning model, specifically including: First, inputting a reference audio text sample set and a target audio text sample set into a text encoder (TextEncoder) to obtain a first acoustic feature sample set. The first acoustic feature sample set includes the phoneme sequence and low-level prosodic information of the target audio text. The low-level prosodic information consists of variations in pitch, duration, and intensity related to the text content, excluding phoneme pronunciation. For example, for a pronunciation sequence... This can correspond to a set carrying multiple lower-level rhythms. , Indicates time, Y represents the j-th lower-level prosodic pattern, and Y is the first acoustic feature.
[0078] Subsequently, the unmasked Mel feature context Prompt is extracted from the reference audio sample set. The Prompt and the first acoustic feature sample set are then input into the Speaker Module of the speaker encoder to obtain the second acoustic feature sample set. Second acoustic feature sample set This includes primary acoustic features and higher prosodic information. Higher prosodic information refers to prosodic features influenced by paralinguistic information, such as spectral energy distribution patterns affected by the speaker's timbre, intonation patterns influenced by factors like emotion and accent, rhythm and syllable duration influenced by emotion and accent, intensity and energy of speech affected by emotional state, and speech style formed by different accents and emotional states. For example, the set of combinations of pronunciation sequence Y with different paralinguistic information is represented as... ,in, Indicates the first Z is the secondary language information, which is the second acoustic feature sample set. Z does not encompass all speech components, but it is sufficient to uniquely identify a speech.
[0079] Finally, during the training of the Detial ODE model, the second acoustic feature sample set is fed into the model. Add random noise as the actual initial data. The real speech acoustic features Mel are used as the target data, and intermediate data are constructed using a weighting factor t. intermediate data and first acoustic feature sample set Second acoustic features The values are concatenated with the Prompt value along the hidden dimension and used as input to the Detial ODE model to obtain the Mel spectrum change. Noise, Mel spectrum variation and second acoustic features After combining, the predicted Mel spectrum is obtained. The ODE model is trained with the minimum difference between the predicted Mel spectrum and the target Mel spectrum as the fitting target. Finally, the trained ODE model is obtained, thus obtaining the zero-shot speech cloning model.
[0080] In the subsequent inference process using the zero-shot speech cloning model, the reference audio text, reference audio, and target audio text are input into the zero-shot speech cloning model to obtain the target audio.
[0081] In the methods described in the embodiments of this specification, during the training of the zero-shot speech cloning model, a second acoustic feature sample set that has a clear correspondence with the acoustic features of real speech is used as the initial state. , and The coupling relationships are independent with minimal overlap, thereby reducing the curvature of the neural network generation path and improving the accuracy of generated samples. Furthermore, this invention considers several key factors in speech synthesis, including text content, speaker features, and prosodic information, achieving high-quality speech synthesis through advanced neural network structures and training strategies.
[0082] The following is combined Figure 4 This invention provides a detailed description of a zero-shot speech cloning method. The zero-shot speech cloning model includes a text encoder, a speaker encoder, and a detail encoder, and specifically includes the following steps:
[0083] 110. Input the reference audio text and the target audio text into the text encoder to obtain the first acoustic feature. The text encoder is used to predict low-level prosodic information related to the pronunciation content of the target audio text from the phoneme string of the reference audio text. Add the low-level prosodic information to the phoneme string of the target audio text to obtain the first acoustic feature.
[0084] 120. Input the acoustic feature context of the reference audio and the first acoustic feature into the speaker encoder to obtain the second acoustic feature. The speaker encoder is used to add high-level prosodic information related to the speaker to the first acoustic feature based on the acoustic feature context of the reference audio to obtain the second acoustic feature.
[0085] Low-level prosodic information includes low-level prosodic features, which in turn include information about pitch, duration, and intensity in the text.
[0086] Advanced prosodic information includes advanced prosodic features, which include shifts in intonation, rhythm, syllable duration, and speech intensity influenced by speaker and emotional information.
[0087] Acoustic features are a set of parameters that describe the physical properties and characteristics of sound. They may include frequency-related parameters, amplitude-related parameters, time-domain related parameters, frequency domain feature-related parameters, and energy-related features. In this embodiment, it may be the Mel spectrum. There are no specific limitations, and it can be determined according to the actual implementation.
[0088] Furthermore, in a practical implementation, a three-layer one-dimensional convolutional neural network with layer normalization (LayerNorm) and a one-layer bidirectional recurrent memory neural network can be used as the network structure for the text encoder. In this case, the kernel size of the convolutional neural network is set to 5, and the hidden layer dimension of all intermediate layers is set to 512.
[0089] The speaker encoder module's network structure can combine convolutional neural networks (CNNs) and bidirectional recurrent memory (BRN) neural networks. This network structure captures contextual information from both global and local perspectives, adding speaker-related paralinguistic and prosodic information to the primary acoustic features. For example, the speaker encoding module consists of a single-layer BRN neural network and seven layers of convolutional neural networks with layer normalization. The kernel size of the CNN is set to 3, the BRN neural network and the following four convolutional layers have a dimension of 512, the middle two convolutional layers have a dimension of 256, and the last layer can be set to 80.
[0090] 130. After concatenating the second acoustic feature with added noise, the first acoustic feature, the second acoustic feature and the acoustic feature context of the reference audio, input the concatenation into the detail encoder to obtain the acoustic feature difference.
[0091] 140. Based on the acoustic feature difference and the second acoustic feature, obtain the predicted Mel spectrum, convert the predicted Mel spectrum into audio, and obtain the target audio corresponding to the target audio text.
[0092] Specifically, it also includes step 200, where step 200 is used to train the zero-shot speech cloning model:
[0093] 210. Input the reference audio text sample set and the target audio text sample set into the text encoder to obtain the first acoustic feature sample set;
[0094] 220. Input the first acoustic feature sample set and the reference audio sample set into the speaker encoder to obtain the second acoustic feature sample set;
[0095] 230. Add a noise term to the second acoustic feature sample set as initial data, then concatenate the first acoustic feature sample set, the acoustic feature context of the reference audio, and the second acoustic feature sample set as input data for the detail encoder, train the detail encoder, and the second acoustic feature sample set has a correspondence with the acoustic features of real speech.
[0096] Specifically, it also includes step 240.
[0097] 240. Use real speech acoustic features as target data. Real speech acoustic features are acoustic features extracted from real speech signals in the target audio text sample set.
[0098] 250. The initial data and target data are weighted and mixed by weighting factors to obtain intermediate data. During the training process, the weighting factors are adjusted so that the intermediate data gradually reaches the target data from the initial data.
[0099] 260. The intermediate data is concatenated with the acoustic feature context of the first acoustic feature sample set and the reference audio sample set, and the second acoustic feature sample set as input data for the detail encoder. The detail encoder is trained so that the acoustic feature difference output by the detail encoder, combined with the second acoustic feature sample set and the noise term, has the smallest difference from the acoustic features of real speech.
[0100] Specifically, step 260 includes:
[0101] An objective function is constructed using real speech acoustic features, a second acoustic feature sample set, and acoustic feature difference. The objective function reflects the difference between the acoustic feature difference output by the detail encoder, combined with the second acoustic feature sample set and the noise term, and the real speech acoustic features.
[0102] The parameters of the detail encoder are determined through iterative updates, with the goal of minimizing the function value of the objective function.
[0103] The objective function is expressed as:
[0104]
[0105] Mel represents the acoustic features of real speech. It is the second acoustic feature sample set. It is a detail encoder. These are the input values of the detail encoder. It is the weighting factor of the iterative process, where T is 1. It is the expected value;
[0106] in,
[0107] Noise is the noise term. It is the first acoustic feature sample set. It is the acoustic feature context of the reference audio sample set.
[0108] Specifically, in the implementation, a Transformer with convolutional positional embeddings, ALiBi attention bias, root mean square normalization, and U-Net-style skip connections can be used to construct the detail encoder in the zero-shot speech cloning model. The Transformer can be configured with 8 layers, each with 16 attention heads, 1024 hidden dimensions, and 4096 fully connected dimensions. The entire zero-shot speech cloning model contains 117M trainable parameters.
[0109] Once the detail encoder is trained, a zero-sample speech clone model is obtained, which includes the text encoder, speaker encoder, and detail encoder.
[0110] In the methods described in the embodiments of this specification, a large amount of text and speech pairing data is used as a training set to train the model using features that have a clear correspondence with the acoustic features of real speech, which improves the accuracy of the generated samples. Furthermore, multiple key factors in speech synthesis are considered, including text content, speaker features, and prosodic information of speech. Through advanced neural network structures and training strategies, high-quality zero-sample speech synthesis is achieved.
[0111] The following is combined Figure 5 This invention introduces a zero-shot speech cloning device. The zero-shot speech cloning model includes a text encoder, a speaker encoder, and a detail encoder. The device includes:
[0112] The first processing module is used to input the reference audio text and the target audio text into a text encoder to obtain a first acoustic feature. The text encoder is used to predict low-level prosodic information related to the pronunciation content of the target audio text from the reference audio text, and add the low-level prosodic information to the phoneme string of the target audio text to obtain the first acoustic feature.
[0113] The second processing module is used to input the reference audio and the first acoustic feature into the speaker encoder to obtain the second acoustic feature. The speaker encoder is used to add high-level prosodic information related to the speaker to the first acoustic feature according to the acoustic feature context of the reference audio to obtain the second acoustic feature.
[0114] The third processing module is used to concatenate the acoustic feature context of the second acoustic feature with added noise, the first acoustic feature, the second acoustic feature and the reference audio, and input it into the detail encoder to obtain the acoustic feature difference quantity.
[0115] The fourth processing module is used to obtain the predicted Mel spectrum based on the acoustic feature difference and the second acoustic feature, convert the predicted Mel spectrum into audio, and obtain the target audio corresponding to the target audio text.
[0116] Based on the above further embodiments, a fifth processing module is also included, specifically used to input the reference audio text sample set and the target audio text sample set into the text encoder to obtain the first acoustic feature sample set;
[0117] The first acoustic feature sample set and the reference audio sample set are input into the speaker encoder to obtain the second acoustic feature sample set;
[0118] A noise term is added to the second acoustic feature sample set as initial data. Then, the first acoustic feature sample set, the acoustic feature context of the reference audio, and the second acoustic feature sample set are concatenated as input data for the detail encoder to train the detail encoder. The second acoustic feature sample set has a correspondence with the acoustic features of real speech.
[0119] Based on the above further embodiments, the fifth processing module is specifically used to use the real speech acoustic features as target data, wherein the real speech acoustic features are acoustic features extracted from the real speech signals of the target audio text sample set;
[0120] The initial data and the target data are weighted and mixed by a weighting factor to obtain intermediate data. During the training process, the weighting factor is adjusted so that the intermediate data gradually reaches the target data from the initial data.
[0121] The intermediate data is concatenated with the acoustic feature context of the first acoustic feature sample set, the reference audio sample set, and the second acoustic feature sample set as input data for the detail encoder. The detail encoder is then trained so that the acoustic feature difference output by the detail encoder, combined with the second acoustic feature sample set and the noise term, has the smallest difference from the real speech acoustic features.
[0122] Based on the above further embodiments, the third processing module is specifically used to construct an objective function using the real speech acoustic features, the second acoustic feature sample set, and the acoustic feature difference quantity. The objective function reflects the difference between the acoustic feature difference quantity output by the detail encoder, combined with the second acoustic feature sample set and the noise term, and the real speech acoustic features.
[0123] The parameters of the detail encoder are determined through iterative updates, with the goal of minimizing the function value of the objective function.
[0124] Based on the above further embodiments, the objective function is expressed as:
[0125]
[0126] Mel represents the acoustic features of real speech. It is the second acoustic feature sample set. It is a detail encoder. These are the input values of the detail encoder. It is the weighting factor of the iterative process, where T is 1. It is the expected value;
[0127] in,
[0128] Noise is the noise term. It is the first acoustic feature sample set. It is the acoustic feature context of the reference audio sample set.
[0129] Based on the above further embodiments, the low-level prosodic information is low-level prosodic information, which includes information on pitch, duration, and intensity in the text.
[0130] The advanced prosodic information refers to advanced prosodic features, which include shifts in intonation, rhythm, syllable duration, and speech intensity influenced by the speaker and emotional information.
[0131] The present invention also provides a computer storage medium on which a computer program is stored, wherein when the computer program is executed by one or more processors, it implements the zero-sample speech cloning method as described in any of the above technical solutions.
[0132] In addition, an electronic device is provided, including a memory and one or more processors, wherein a computer program is stored on the memory, and the computer program, when executed by the one or more processors, implements the zero-sample speech cloning method as described in any of the above technical solutions.
[0133] To further analyze the advantages of the proposed model compared to the state-of-the-art ODE model, the objective performance metrics of the SF-Speech model and two VoiceBoxes of different sizes were measured on ZS-TTS under different numbers of function evaluation (NFE) steps. Figure 3 The diagram illustrates the performance of ODE-based models under different NFEs, showing the fluctuation trend of objective metrics as the NFE of the ODE model increases. This demonstrates that adding inference steps does not improve the performance of ODE-based models. This is because as the NFE increases, ODE-based models introduce more harmful noise (reverberation, ambient sound, artificial noise, etc.) into the generated Mel feature spectrum, and this noise is widely present in the training dataset. Furthermore, different objective metrics exhibit different trends. Figure 3 As shown in (a), the SIM of these three models exhibits the same trend, peaking when NFE equals 8. The SF-Speech model outperforms VoiceBox-S in all NFE cases, and outperforms VoiceBox when NFE is greater than 8. This result suggests that the SF-Speech model can mitigate the performance degradation caused by increasing NFE. On the other hand, as... Figure 3As shown in (b), the SF-Speech model achieves the lowest WER under any NFE condition. Furthermore, when the NFE exceeds 8, the WER of the SF-Speech model increases less than that of VoiceBox-S and VoiceBox, further demonstrating the superiority of the SF-Speech model. Finally, since WER and SIM cannot fully represent the quality of generated speech, DNSMOS of generated speech under different NFE conditions was also tested. DNSMOS is a model that uses a neural network to simulate the average opinion score of human speech quality. Figure 3 As shown in (c), the SF-Speech model outperforms VoiceBox and VoiceBox-S in DNSMOS scores at any NFE. Furthermore, and most importantly, VoiceBox and VoiceBox-S achieve the best DNSMOS at an NFE of 8, while the SF-Speech model reaches its optimal generation quality at an NFE of 4. This suggests that uniformly selecting a solution step count of 8 for comparison with all ODE-based models is advantageous for VoiceBox and VoiceBox-S. Even so, the SF-Speech model still approaches or even surpasses existing best models on multiple metrics, further demonstrating the superiority of the proposed model in zero-resource speech cloning.
[0134] The specific embodiments described above further illustrate the purpose, technical solution, and beneficial effects of this application. It should be understood that the above description is only a specific embodiment of this application and is not intended to limit the scope of protection of this application. Any modifications, equivalent substitutions, improvements, etc., made on the basis of the technical solution of this application should be included within the scope of protection of this application.
Claims
1. A zero-sample speech cloning method, characterized in that, The method includes: The reference audio text and the target audio text are input into a text encoder to obtain the first acoustic feature. The text encoder is used to predict low-level prosodic information related to the pronunciation content of the target audio text from the reference audio text. The low-level prosodic information is added to the phoneme string of the target audio text to obtain the first acoustic feature. The acoustic feature context of the reference audio and the first acoustic feature are input into the speaker encoder to obtain the second acoustic feature. The speaker encoder is used to add high-level prosodic information related to the speaker to the first acoustic feature according to the acoustic feature context of the reference audio to obtain the second acoustic feature. The acoustic feature context of the second acoustic feature with added noise, the first acoustic feature, the second acoustic feature, and the reference audio are concatenated and then input into the detail encoder to obtain the acoustic feature difference quantity. Based on the acoustic feature difference and the second acoustic feature, a predicted Mel spectrum is obtained, and the predicted Mel spectrum is converted into audio to obtain the target audio corresponding to the target audio text.
2. The method according to claim 1, characterized in that, The method further includes: The reference audio text sample set and the target audio text sample set are input into the text encoder to obtain the first acoustic feature sample set; The first acoustic feature sample set and the reference audio sample set are input into the speaker encoder to obtain the second acoustic feature sample set; A noise term is added to the second acoustic feature sample set as initial data. Then, the first acoustic feature sample set, the acoustic feature context of the reference audio, and the second acoustic feature sample set are concatenated as input data for the detail encoder to train the detail encoder. The second acoustic feature sample set has a correspondence with the acoustic features of real speech.
3. The method according to claim 2, characterized in that, The method further includes: The real speech acoustic features are used as target data, and the real speech acoustic features are acoustic features extracted from real speech signals of the target audio text sample set. The initial data and the target data are weighted and mixed by a weighting factor to obtain intermediate data. During the training process, the weighting factor is adjusted so that the intermediate data gradually reaches the target data from the initial data. The intermediate data is concatenated with the acoustic feature context of the first acoustic feature sample set, the reference audio sample set, and the second acoustic feature sample set as input data for the detail encoder. The detail encoder is then trained so that the acoustic feature difference output by the detail encoder, combined with the second acoustic feature sample set and the noise term, has the smallest difference from the real speech acoustic features.
4. The method according to claim 3, characterized in that, The step of concatenating the intermediate data with the acoustic feature context of the first acoustic feature sample set, the reference audio sample set, and the second acoustic feature sample set as input data for the detail encoder, and training the detail encoder so that the acoustic feature difference output by the detail encoder, combined with the second acoustic feature sample set and noise terms, has the smallest difference from the real speech acoustic features, specifically includes: An objective function is constructed using the real speech acoustic features, the second acoustic feature sample set, and the acoustic feature difference quantity. The objective function reflects the difference between the acoustic feature difference quantity output by the detail encoder, combined with the second acoustic feature sample set and the noise term, and the real speech acoustic features. The parameters of the detail encoder are determined through iterative updates, with the goal of minimizing the function value of the objective function.
5. The method according to claim 4, characterized in that, The objective function is expressed as: Mel represents the acoustic features of real speech. It is the second acoustic feature sample set. It is a detail encoder. These are the input values of the detail encoder. It is the weighting factor of the iterative process, where T is 1. It is the expected value; in, Noise is the noise term. It is the first acoustic feature sample set. It is the acoustic feature context of the reference audio sample set.
6. The method according to claim 1, characterized in that, The low-level prosodic information includes information about pitch, duration, and intensity in the text.
7. The method according to claim 1, characterized in that, The advanced prosodic information refers to advanced prosodic features, which include shifts in intonation, rhythm, syllable duration, and speech intensity influenced by the speaker and emotional information.
8. A zero-sample speech cloning device, characterized in that, The device includes: The first processing module is used to input the reference audio text and the target audio text into a text encoder to obtain a first acoustic feature. The text encoder is used to predict low-level prosodic information related to the pronunciation content of the target audio text from the reference audio text, and add the low-level prosodic information to the phoneme string of the target audio text to obtain the first acoustic feature. The second processing module is used to input the reference audio and the first acoustic feature into the speaker encoder to obtain the second acoustic feature. The speaker encoder is used to add high-level prosodic information related to the speaker to the first acoustic feature according to the acoustic feature context of the reference audio to obtain the second acoustic feature. The third processing module is used to concatenate the acoustic feature context of the second acoustic feature with added noise, the first acoustic feature, the second acoustic feature and the reference audio, and input it into the detail encoder to obtain the acoustic feature difference quantity. The fourth processing module is used to obtain the predicted Mel spectrum based on the acoustic feature difference and the second acoustic feature, convert the predicted Mel spectrum into audio, and obtain the target audio corresponding to the target audio text.
9. A computer storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by one or more processors, implements the zero-sample speech cloning method as described in any one of claims 1 to 7.
10. An electronic device, characterized in that, It includes a memory and one or more processors, wherein the memory stores a computer program that, when executed by the one or more processors, implements the zero-sample speech cloning method as described in any one of claims 1 to 7.