A training method for a sound cloning model, a sound cloning method, and a product

Abstract

This application provides a training method, a sound cloning method, and a product for a sound cloning model, belonging to the technical field of speech synthesis. The method includes preprocessing multiple acquired audio data to construct a training sample set; training a preset sound cloning model based on the training sample set and a preset block size; a text encoder in the sound cloning model extracts text vectors from text sequence samples of any training sample; a decoder outputs a predicted Mel spectrum based on the text vector, Mel spectrum samples, and speaker vector of any training sample; the block size is used to adjust the computation mode of the sound cloning model, including non-streaming and streaming modes; and iteratively training the preset sound cloning model based on the target Mel spectrum label, predicted Mel spectrum, and loss function of the optimal transport stream for any training sample. This application aims to provide a method for achieving a high-quality streaming sound cloning model.

Description

Technical Field

[0001] This application relates to the technical field of speech synthesis, and more specifically, to a training method for a voice cloning model, a voice cloning method, and a product. Background Technology

[0002] With the development of technology, text-to-speech (TTS) technology plays an increasingly important role in daily life, significantly improving efficiency and convenience. For example, TTS technology is widely used in customer service systems to provide 24 / 7 service, and it also plays an important role in human-computer interaction in smartphones and smart home systems. Users' needs for voice systems are not limited to providing clear and natural voice output, but also include more diverse functions, one of the most important of which is high-quality zero-sample voice cloning capability.

[0003] Zero-shot voice cloning, also known as zero-shot speaker adaptation or zero-shot speech synthesis (zero-shotTTS), aims to synthesize the voice of a speaker that did not appear during training, based on any reference speech.

[0004] To achieve better zero-shot cloning results, research has begun on the application of large language models in the TTS field and the application of diffusion models based on flow matching technology in the TTS field. However, the speech models obtained by the current methods have not solved the problem of streaming generation and require the synthesis of complete sentence audio before output.

[0005] Therefore, there is an urgent need for a streaming sound cloning model with better performance to achieve high quality. Summary of the Invention

[0006] This application provides a training method, a sound cloning method, and a product for a sound cloning model, aiming to provide a high-quality streaming sound cloning model.

[0007] In a first aspect, embodiments of this application provide a method for training a sound cloning model, the method comprising:

[0008] Multiple audio data sets are preprocessed to construct a training sample set. Each training sample in the training sample set includes a text sequence sample, a Mel spectrum sample, and a target Mel spectrum label corresponding to any audio data set.

[0009] Based on the training sample set and the preset block size, a preset sound clone model is trained. The text encoder in the preset sound clone model is used to extract text vectors based on the text sequence samples of any training sample. The decoder in the preset sound clone model is used to output a predicted Mel spectrum based on the text vector, Mel spectrum sample, and speaker vector of any training sample. The block size is used to adjust the calculation mode of the preset sound clone model to non-streaming or streaming.

[0010] Based on the target Mel spectrum label, predicted Mel spectrum, and loss function of the optimal transport stream for any training sample, the preset sound cloning model is iteratively trained to obtain the trained sound cloning model.

[0011] Optionally, the acquired audio data is preprocessed to construct a training sample set, including:

[0012] For any acquired audio data, generate the corresponding text sequence and Mel spectrum;

[0013] Perform a masking operation on the Mel spectrum corresponding to any audio data to obtain the masked Mel spectrum and the unmasked Mel spectrum. Use the unmasked Mel spectrum as the Mel spectrum sample and the masked Mel spectrum as the target Mel spectrum label.

[0014] Insert characters into the text sequence corresponding to the target Mel spectrum label in the audio data so that the length of the text sequence corresponding to the target Mel spectrum label is consistent with the frame number of the target Mel spectrum label, and use the text sequence corresponding to the Mel spectrum sample as the text sequence sample;

[0015] Use the text sequence sample, Mel spectrum sample, and target Mel spectrum label corresponding to any audio data as a training sample.

[0016] Optionally, masking the Mel spectrum corresponding to any audio data includes:

[0017] For any audio data, perform a masking operation on the preset mask range of the Mel spectrum corresponding to the audio data, or perform a masking operation on the entire Mel spectrum corresponding to the audio data.

[0018] Optionally, the method further includes:

[0019] Construct a pre-defined sound cloning model;

[0020] The preset sound cloning model includes a text encoder and a decoder. The text encoder includes a Conformer layer for extracting text vectors from any text sequence sample. The decoder is based on a Transformer structure in the form of U-net and includes a downsampling module, an intermediate module, and an upsampling module. Each of the downsampling module, the intermediate module, and the upsampling module consists of a residual module based on convolutional layers and a Transformer module based on an attention mechanism. Furthermore, any convolutional layer in the preset sound cloning model is a block convolution, and the attention mechanism of the Transformer module is a block mask-based attention mechanism.

[0021] Optionally, the method further includes:

[0022] In response to a preset operation, the range of block size variation of the preset sound clone model during the training process is determined, wherein the block size is used to adjust the calculation mode of the preset sound clone model to non-streaming or streaming.

[0023] In response to the training strategy selection operation, during the process of training a preset sound clone model based on the training sample set, a first training strategy or a second training strategy is executed.

[0024] Specifically, when executing the first training strategy, during each iteration of training of the preset sound clone model, a block size is randomly selected from the range of block size variations as the block size for this iteration; when executing the second training strategy, the loss function in the non-streaming computation mode and the loss function in the streaming computation mode are determined respectively, and during the backpropagation of the preset sound clone model, the model parameters of the preset sound clone model are updated simultaneously based on the loss functions in the non-streaming computation mode and the streaming computation mode.

[0025] Secondly, embodiments of this application provide a sound cloning method, the method comprising:

[0026] The target prompt speech and the text sequence to be synthesized are obtained, and the prompt Mel spectrum of the target prompt speech and the prompt text sequence are generated.

[0027] Based on the text sequence to be synthesized and the prompt text sequence, the text sequence to be input is obtained through preprocessing.

[0028] The Mel spectrum of the target prompt speech and the text sequence to be input are input into the sound cloning model, and the Mel spectrum result corresponding to the text sequence to be synthesized is output by the sound cloning model. The sound cloning model is obtained by the training method of the sound cloning model described in the first aspect of the embodiment.

[0029] Based on the Mel spectrum results, the audio data corresponding to the text sequence to be input is obtained.

[0030] Optionally, based on the text sequence to be synthesized and the prompt text sequence, preprocessing is performed to obtain the text sequence to be input, including:

[0031] Based on the text sequence to be synthesized, estimate the number of frames of the Mel spectrum result corresponding to the text sequence to be synthesized;

[0032] Characters are inserted into the text sequence composed of the text sequence to be synthesized and the prompt text sequence to obtain the text sequence to be input. The length of the text sequence to be input is equal to the sum of the number of frames of the prompt Mel spectrum and the number of frames of the Mel spectrum result corresponding to the text sequence to be synthesized.

[0033] Thirdly, embodiments of this application provide an electronic device, including a memory, a processor, and a computer program stored in the memory. The processor executes the computer program to implement the training method for the sound cloning model as described in the first aspect of the embodiment, or to implement the sound cloning method as described in the second aspect of the embodiment.

[0034] Fourthly, embodiments of this application provide a computer-readable storage medium storing a computer program / instructions thereon, which, when executed by a processor, implements the training method for the sound cloning model as described in the first aspect of the embodiments, or implements the sound cloning method as described in the second aspect of the embodiments.

[0035] Fifthly, embodiments of this application provide a computer program product, including a computer program / instruction, which, when executed by a processor, implements the training method for the sound cloning model as described in the first aspect, or implements the sound cloning method as described in the second aspect.

[0036] Beneficial effects:

[0037] In training the sound cloning model, multiple acquired audio data are preprocessed to construct a training sample set. Each training sample in the training sample set includes a text sequence sample, a Mel spectrum sample, and a target Mel spectrum label corresponding to any audio data. Based on the training sample set and a preset block size, a preset sound cloning model is trained. The text encoder in the preset sound cloning model is used to extract text vectors from the text sequence samples of any training sample. The decoder in the preset sound cloning model is used to output a predicted Mel spectrum based on the text vector, Mel spectrum sample, and speaker vector of any training sample. The block size is used to adjust the computation mode of the preset sound cloning model to non-streaming or streaming. Based on the target Mel spectrum label, the predicted Mel spectrum, and the loss function of the optimal transport stream for any training sample, the preset sound cloning model is iteratively trained to obtain the trained sound cloning model.

[0038] The sound cloning model trained by this method can output the predicted Mel spectrum corresponding to the input portion as soon as it receives an input that meets the block size requirement, without having to wait until all inputs have been received before outputting the overall predicted Mel spectrum. This provides a way to achieve a high-quality streaming sound cloning model. Attached Figure Description

[0039] To more clearly illustrate the technical solutions of the embodiments of this application, the drawings used in the description of the embodiments of this application will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0040] Figure 1 This is a flowchart of the steps of a training method for a sound cloning model proposed in an embodiment of this application;

[0041] Figure 2 This is a schematic diagram illustrating the training of a sound cloning model proposed in one embodiment of this application;

[0042] Figure 3 This is a schematic diagram of the structure of a decoder provided in an embodiment of this application;

[0043] Figure 4 This is a schematic diagram of block convolution proposed in an embodiment of this application;

[0044] Figure 5 This is a schematic diagram of a chunk mask proposed in an embodiment of this application;

[0045] Figure 6 This is a flowchart of the steps of a sound cloning method provided in an embodiment of this application;

[0046] Figure 7 This is a schematic diagram illustrating the reasoning of a sound cloning model proposed in one embodiment of this application;

[0047] Figure 8 This is a functional block diagram of a training device for a sound cloning model provided in an embodiment of this application;

[0048] Figure 9 This is a functional block diagram of a sound cloning device proposed in an embodiment of this application. Detailed Implementation

[0049] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this application. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0050] TTS: Text to Speech, speech synthesis;

[0051] Flow matching technique: a training method for training continuous normalized flow models, which trains the model by learning a vector field related to the conditional probability path; it can be used to train diffusion models and can be combined with optimal transport techniques to further accelerate the training speed; during inference, ordinary differential equations can be used to solve for the generation of new samples.

[0052] Mel spectrum: Mel spectrogram; Mel spectrum refers to the spectrum converted from frequency to Mel scale. Studies have shown that the human ear's perception of frequency is not linear. Mel scale is a non-linear scale unit based on frequency definition, representing the human ear's equidistant changes in pitch.

[0053] Encodec is a neural network-based audio encoding scheme designed to achieve efficient and high-quality audio compression. By using autoencoder and residual vector quantization (RVQ) techniques, Encodec can significantly reduce the size of audio files while maintaining high sound quality.

[0054] Matcha-TTS is an efficient text-to-speech (TTS) architecture based on conditional flow matching. Its main goal is to provide an efficient non-autoregressive neural network TTS model that uses conditional flow matching technology to accelerate speech synthesis based on ordinary differential equations (ODEs).

[0055] With the development of technology, text-to-speech (TTS) technology is playing an increasingly important role in daily life, significantly improving efficiency and convenience. For example, TTS technology is widely used in customer service systems to provide 24 / 7 service, and it also plays an important role in human-computer interaction in smartphones and smart home systems.

[0056] With the rapid development of information technology, users' needs for voice systems are no longer limited to providing clear and natural voice output, but also include more diverse functions. One important one is the ability to create high-quality zero-shot voice cloning. Zero-shot voice cloning, also known as zero-shot speaker adaptation or zero-shot speech synthesis (zero-shot TTS), aims to synthesize the voice of a speaker that did not appear during the training process based on any reference speech.

[0057] Traditional research approaches primarily explore how to fine-tune the overall or partial parameters of existing multi-person TTS models using data from the target speaker. While such fine-tuning methods can achieve certain synthesis effects, they have some obvious drawbacks: additional training time is required for each speaker outside the training set, and the synthesis effect is limited by the length of the speaker's speech corpus; a corpus of less than 10 sentences often severely reduces the quality of the synthesized audio.

[0058] To achieve better zero-shot cloning results, research has begun on the application of large language models in the TTS field. For example, one approach involves discretizing speech using a speech encoder and then training a large language model on large-scale audio data for "next token prediction." This enables the model to extract information such as the timbre and prosody of the target speaker from a 3-second audio prompt, thus achieving zero-shot sound cloning. However, the synthesis quality of such models is highly susceptible to the quality of speech discretization, and large language models employ autoregressive inference, generating sample points one by one, resulting in slow generation speeds that often fail to meet practical business needs. In contrast, non-autoregressive models can model in a continuous vector space and utilize parallel processing to improve inference speed, making them more favored in practical applications.

[0059] Diffusion models based on flow matching are one of the mainstream approaches in non-autoregressive generative models. By controlling the noise addition and denoising processes, diffusion models can capture and fit complex data distributions, thereby generating high-quality data samples. For example, Matcha-TTS, by combining a flow matching-based diffusion model as a decoder with a text encoder, outperforms traditional speech synthesis models in both word error rate and subjective average score. Current proposals also include using diffusion models to model the quantized hidden layer vectors of discrete encoders and injecting reference speech information into the model through an attention mechanism to achieve zero-shot generation capability. Voicebox, through its masking mechanism, enables the model to learn context, resulting in a simpler and more elegant overall design.

[0060] However, these research directions all rely on third-party duration alignment tools to obtain phoneme-level duration information. Further, E2 TTS proposes a simpler approach: inserting specified characters—filler tokens—into the character sequence input to the model, ensuring consistency in the dimensions of the input character sequence and the target audio features. The model then learns to align the text content with the audio. However, none of the aforementioned models solve the problem of streaming generation; they require synthesizing complete sentence audio for output.

[0061] In summary, current zero-sample sound cloning models still have various problems, and there is an urgent need for a better-performing sound cloning model that can achieve high quality.

[0062] Reference Figure 1 The diagram illustrates a flowchart of a training method for a sound cloning model according to an embodiment of this application. The method may specifically include the following steps:

[0063] S101: Preprocess the acquired audio data to construct a training sample set. Each training sample in the training sample set includes a text sequence sample, a Mel spectrum sample, and a target Mel spectrum label corresponding to any audio data.

[0064] Specifically, the process of preprocessing the acquired audio data to construct a training sample set includes:

[0065] A1: For any acquired audio data, generate the corresponding text sequence and Mel spectrum.

[0066] First, for each audio data item, generate the text sequence in that audio data and the Mel spectrum of that audio data.

[0067] A2: Perform a masking operation on the Mel spectrum corresponding to any audio data to obtain the masked Mel spectrum and the unmasked Mel spectrum. Use the unmasked Mel spectrum as the Mel spectrum sample and the masked Mel spectrum as the target Mel spectrum label.

[0068] Specifically, for any audio data, a preset masking operation is performed on the Mel spectrum corresponding to the audio data, or the entire Mel spectrum corresponding to the audio data is masked. The preset masking range can be customized according to the actual application requirements. For example, the range of X1% to X2% of the Mel spectrum of an audio data can be masked.

[0069] For example, for each audio data, there is a 50% probability that 70% to 90% of its Mel spectrum can be masked, and there is a 50% probability that the entire Mel spectrum of the audio data can be masked to conceal it.

[0070] In practice, during the masking operation, a binary mask matrix is ​​generated and multiplied by the Mel spectrum of the audio data, as shown in the following formula:

[0071]

[0072] in, Represents the product of Hadamah, The Mel spectrum of the audio data. Here are the Mel spectrum samples of the audio data, where M is the binarization mask matrix. D and T are the dimensions of the mask matrix, which are the same as the dimensions of the Mel spectrum of the audio data.

[0073] During the training process, the loss function of the sound cloning model only considers the loss of the masked part. That is, the model will predict the masked Mel spectrum based on the Mel spectrum sample, i.e. the unmasked Mel spectrum. The sound cloning model is iteratively trained based on the loss of the predicted masked Mel spectrum and the target Mel spectrum label, so that the masked Mel spectrum predicted by the sound cloning model is closer to the target Mel spectrum label.

[0074] A3: Insert characters into the text sequence corresponding to the target Mel spectrum label in the audio data so that the length of the text sequence corresponding to the target Mel spectrum label is consistent with the frame number of the target Mel spectrum label, and use the text sequence corresponding to the Mel spectrum sample as a text sequence sample.

[0075] The Mel spectrum of the target Mel spectrum label, i.e. the Mel spectrum part of the mask, needs to be predicted by the model. However, since the length of the text content is much shorter than the length of the audio for speech, it is often necessary to align the lengths of the two in TTS systems.

[0076] In this method, in order to reduce the reliance on third-party duration alignment tools, the length of the text sequence corresponding to the target Mel spectrum label can be made consistent with the number of frames of the target Mel spectrum label by inserting characters, i.e., inserting filler token <F>.

[0077] For example, suppose the text sequence corresponding to the target Mel spectrum label in the audio data is: Then, by inserting characters, the text sequence corresponding to the expanded target Mel spectrum label is obtained as follows:

[0078]

[0079] in, This represents the filler token. The number of TMs is equal to the number of characters in the text sequence corresponding to the target Mel spectrum label.

[0080] Therefore, the text sequence corresponding to the expanded target Mel spectrum label The length is consistent with the frame number T of the target Mel spectrum tag.

[0081] In practice, characters can be inserted at the end of a text sequence, and the inserted characters cannot conflict with the original characters in the text sequence. For example, if the text sequence includes text, punctuation marks can be inserted; if the text sequence includes both text and punctuation marks, English words can be inserted.

[0082] A4: Use the text sequence sample, Mel spectrum sample, and target Mel spectrum label corresponding to any audio data as a training sample.

[0083] In practice, text sequence samples can include two formats:

[0084] In the first format, the text sequence sample consists only of characters and punctuation marks. The text part contains two languages: Chinese and English. For example, English can be represented by words and Chinese can be represented by pinyin.

[0085] In the second format, the text sequence samples are represented by characters, phonemes, and punctuation marks. That is, phoneme representations are added on top of characters. During the training process, there is a 15% probability that the characters will be converted into phoneme representations. Parentheses are added before and after the phoneme representations as separators between phonemes and characters.

[0086] By setting two text sequence sample formats, the aim is to address situations where certain characters in actual business scenarios have special pronunciations. If only the first format, which consists only of characters and punctuation marks, is used, it is difficult to intervene in the pronunciation of certain special characters in actual business scenarios, such as specific personal names, place names, and abbreviations. Furthermore, training data for special characters is often scarce or even non-existent, so the model's pronunciation predictions for these parts will be strange or inconsistent with the context. Therefore, by introducing phoneme representation, a means of intervening in the pronunciation information in the model's synthesized audio is provided, improving the model's adaptability to complex real-world scenarios.

[0087] S102: Based on the training sample set and the preset block size, train a preset sound clone model. The text encoder in the preset sound clone model is used to extract text vectors based on the text sequence samples of any training sample. The decoder in the preset sound clone model is used to output a predicted Mel spectrum based on the text vector, Mel spectrum sample, and speaker vector of any training sample. The block size is used to adjust the calculation mode of the preset sound clone model to non-streaming or streaming.

[0088] Reference Figure 2 The diagram illustrates the training schematic of the sound cloning model provided in this application embodiment. A preset sound cloning model structure is pre-constructed. The preset sound cloning model includes a text encoder and a decoder. After inputting the text sequence samples, Mel spectrum samples, and target Mel spectrum labels of the training samples into the model, the text encoder extracts text vectors based on the text sequence samples of any training sample. The decoder in the sound cloning model outputs the predicted Mel spectrum based on the text vector, Mel spectrum samples, and speaker vector of any training sample.

[0089] The text encoder includes a Conformer layer for extracting the text vector of any text sequence sample.

[0090] Reference Figure 3 The diagram shows a schematic of the decoder provided in an embodiment of this application. The decoder is based on a Transformer structure in the form of U-net and includes down-sampling blocks, mid-blocks, and up-sampling blocks. Each of the down-sampling blocks, mid-blocks, and up-sampling blocks consists of a residual module based on convolutional layers and a Transformer module based on an attention mechanism.

[0091] The decoder's input is a Mel spectrum sample that has been partially masked after a masking operation. The speaker vector V extracted by the voice cloning model from the training samples, the text vector Y output by the text encoder, the time step t, and the intermediate features corresponding to time t. .

[0092] In one feasible implementation, the loss function of the sound cloning model can be constructed using an optimal-transport conditional flow matching model (OT-CFM).

[0093] OT-CFM is an extension of flow matching techniques. The purpose of OT-CFM is based on a prior distribution. To predict the data distribution for generating target Mel spectrum labels .

[0094] First, using time-dependent vector fields To define the probability density path, where The flow can be generated using the following ordinary differential equation. :

[0095] (1)

[0096] in, , Prior distribution Follows a normal distribution By solving the initial value problem of equation (1), the target distribution is obtained. It can be used To approximate and sample from it.

[0097] To fit the vector field We construct a loss function based on the optimal transport stream to train the sound cloning model:

[0098]

[0099] in:

[0100]

[0101] .

[0102] Furthermore, in order for the trained voice clone model to support both non-streaming and streaming inference, in this embodiment, the voice clone model adopts chunk-based streaming computation. This computation method requires the model to calculate the corresponding output when it receives an input that meets a chunk size, without having to wait for all inputs to be received before it can perform computation.

[0103] Therefore, in the sound cloning model structure of this embodiment, any convolutional layer in the preset sound cloning model is a chunk convolution, and the attention mechanism of the Transformer module is a chunk mask attention mechanism.

[0104] Reference Figure 4 The diagram illustrates the block convolution provided in this embodiment. Convolution calculation typically involves three methods: regular convolution, causal convolution, and block convolution.

[0105] Assuming the chunk size is 7, then the sequence corresponding to the first chunk is: If the kernel size is 5 and the stride is 1, then a regular convolution will pad the beginning of the sequence with two zeros, and at the end of the sequence, it needs to use... This information is clearly not sufficient for streaming computing, because in streaming computing... Information belonging to the next chunk.

[0106] Causal convolution first pads the beginning of the sequence with four zeros to ensure that the output at each time point only sees the information of the current time and past time, without using the information of future time. Although this satisfies the requirements of streaming computing, it obviously reduces the performance of the model.

[0107] The difference between block convolution and regular convolution is that it pads the end of the sequence with two zeros, while keeping everything else unchanged. This ensures that the model does not use future information when calculating each chunk, while still being able to see information about future time points at each time point in a chunk.

[0108] Therefore, the sound cloning model structure in this embodiment adopts chunk-based streaming computation, which can simultaneously support non-streaming and streaming inference.

[0109] Furthermore, in the attention-based Transformer layer of the sound cloning model, a chunkmask is used to control the input of the sound cloning model.

[0110] Reference Figure 5 This diagram illustrates the chunk mask provided in this embodiment. A binary mask matrix is ​​generated based on the chunk size as the chunk mask, with values ​​of 1 only within each chunk and 0 elsewhere. When the chunk size equals the total length of the input sequence, the chunk mask is equivalent to a full mask, i.e., a matrix of all 1s. If the chunk size is 2, then... Figure 5 As shown in the middle chunk mask.

[0111] Since block-based streaming computation allows the sound cloning model to adjust the computation mode to non-streaming or streaming by setting the chunk size, two training strategies were also adopted during the training phase.

[0112] For example, in response to a preset operation, the range of block size variation of a preset sound clone model during training is determined, the block size being used to adjust the computation mode of the preset sound clone model to non-streaming or streaming.

[0113] Then, in response to the training strategy selection operation, during the process of training the preset sound clone model based on the training sample set, the first training strategy or the second training strategy is executed.

[0114] When executing the first training strategy, during each iteration of training of the preset sound clone model, a block size is randomly selected from the range of block size variations as the block size for this iteration. This allows the chunk size of the sound clone model to change dynamically within a certain range during training, enabling the model to adapt to different chunk sizes during inference and improving the wide adaptability of the sound clone model in streaming computing.

[0115] When implementing the second training strategy, a distillation loss function is used, employing non-streaming computation results to guide the streaming computation results. During training, when the model computes in streaming mode, it calculates not only the loss function between the model and the accurate result, but also the loss function between the model and the result computed in non-streaming mode.

[0116] Specifically, the loss functions in the non-streaming computation mode and the streaming computation mode are determined respectively. During the backpropagation of the preset sound cloning model, the model parameters of the preset sound cloning model are updated simultaneously based on the loss functions in the non-streaming computation mode and the streaming computation mode.

[0117] S103: Based on the target Mel spectrum label, predicted Mel spectrum, and loss function of the optimal transport stream of any training sample, iteratively train the preset sound clone model to obtain the trained sound clone model.

[0118] The voice cloning model trained by this method has the ability to perform zero-sample voice cloning. During training, the model learns to predict the masked parts based on the unmasked Mel spectrum through mask design. Combined with large-scale data training, the model has a strong context learning ability and can achieve zero-sample voice replication for speakers outside the set.

[0119] Furthermore, flow matching technology is used to train the diffusion process, enabling fine-grained control over data distribution and thus improving the quality of synthesized speech. Based on the model's strong context learning ability, speaker vectors are introduced to further enhance the timbre similarity of voice clones.

[0120] The model structure and inference method are constructed based on the block size, which enables the model to achieve non-streaming or streaming generation by adjusting the block size. The inference method is more flexible and can effectively reduce the latency of the first packet and the response time.

[0121] Reference Figure 6 The diagram illustrates a flowchart of a sound cloning method provided in an embodiment of this application. The method includes the following steps:

[0122] S201: Obtain the target prompt speech and the text sequence to be synthesized, and generate the prompt Mel spectrum of the target prompt speech and the prompt text sequence.

[0123] S202: Based on the text sequence to be synthesized and the prompt text sequence, preprocess to obtain the text sequence to be input.

[0124] Suppose that the Mel spectrum of the target prompt speech is denoted as... The text sequence corresponding to the target voice prompt is: The text sequence to be synthesized is .

[0125] Specifically, the process of preprocessing the text sequence to be input based on the text sequence to be synthesized and the prompt text sequence includes the following steps:

[0126] B1: Based on the text sequence to be synthesized, estimate the number of frames of the Mel spectrum result corresponding to the text sequence to be synthesized.

[0127] Since the number of frames in the Mel spectrum result corresponding to the text sequence to be synthesized is unknown, it can be estimated by the average duration of each text in the training set and the prompt speech. The estimation method can be selected according to the needs of the actual application.

[0128] For example, if a text sequence of length L1 corresponds to a Mel spectrum frame number Z1, then the Mel spectrum frame number Z per unit length of the text sequence can be calculated. Thus, based on the length of the text sequence to be synthesized and the Mel spectrum frame number Z per unit length of the text sequence, the number of frames of the Mel spectrum result corresponding to the text sequence to be synthesized can be calculated.

[0129] It is also possible to estimate the number of Mel spectrum frames for each text in the text sequence. For example, if the text sequence to be synthesized includes the text "weather", and the number of Mel spectrum frames for the text "weather" in the training set or prompt speech is Z2, then the number of Mel spectrum frames for the text sequence to be synthesized including the text "weather" can be determined as Z2, thereby determining the number of Mel spectrum frames for each text in the text sequence to be synthesized, and using the number of Mel spectrum frames for each text as the number of frames in the Mel spectrum result corresponding to the text sequence to be synthesized.

[0130] It is worth noting that when estimating the number of frames in the Mel spectrum result corresponding to the text sequence to be synthesized, the following conditions must be met: ,in, The number of frames in the Mel spectrum representing the target speech prompt. This represents the frame number of the Mel spectrum corresponding to the text sequence to be synthesized. This represents the length of the prompt text sequence. This represents the length of the text sequence to be synthesized.

[0131] B2: Insert characters into the text sequence composed of the text sequence to be synthesized and the prompt text sequence to obtain the text sequence to be input. The length of the text sequence to be input is equal to the sum of the number of frames of the prompt Mel spectrum and the number of frames of the Mel spectrum result corresponding to the text sequence to be synthesized.

[0132] Specifically, a filler token can be inserted at the end of the text sequence. Furthermore, the inserted character cannot conflict with the existing characters in the text sequence. For example, if the text sequence includes text, punctuation characters can be inserted; if the text sequence includes both text and punctuation, English words can be inserted.

[0133] By inserting characters, the resulting text sequence to be input is:

[0134]

[0135] in, The number of equals .

[0136] This approximates the length of the text sequence to be synthesized to the number of frames in the corresponding Mel spectrum result, thus aligning the text length with the audio length.

[0137] S203: Input the Mel spectrum of the target prompt speech and the text sequence to be input into the sound cloning model, and output the Mel spectrum result corresponding to the text sequence to be synthesized through the sound cloning model.

[0138] The sound cloning model is obtained through the training method of the sound cloning model as described in this embodiment.

[0139] Reference Figure 7 The diagram illustrates the reasoning of the sound cloning model provided in this application embodiment. After inputting the text sequence to be input and the prompt Mel spectrum into the sound cloning model, the Mel spectrum corresponding to the text sequence to be synthesized can be regarded as the part that is masked after the masking operation. Thus, the sound cloning model generates the Mel spectrum corresponding to the masked text sequence to be synthesized.

[0140] The text encoder of the voice cloning model generates the text vector of the text sequence to be input. The voice cloning model extracts the speaker vector based on the cue mel spectrum. Finally, the decoder of the voice cloning model generates and outputs the mel spectrum corresponding to the text sequence to be synthesized based on the text vector of the text sequence to be input, the speaker vector, and the cue mel spectrum.

[0141] S204: Based on the Mel spectrum results, convert the audio data corresponding to the text sequence to be input.

[0142] For example, the audio data corresponding to the text sequence to be input can be obtained by converting the Mel spectrum result using a vocoder tool, such as HiFiGAN.

[0143] The sound cloning model trained in this embodiment is a high-quality streaming zero-sample sound cloning model based on stream matching technology. By designing the overall model architecture based on a stream matching diffusion model and optimizing the text sequence, the reliance on third-party duration alignment tools is reduced. It also supports setting the pronunciation of specific text sequences, introduces masking design, and constructs a streaming inference architecture based on block computation, enabling the model to achieve high-quality sound cloning.

[0144] Specifically, the sound cloning model of this embodiment has at least the following beneficial effects:

[0145] 1. Enables zero-sample voice cloning; utilizes masking to design the model's loss function, trains the model based on large-scale corpus training data, improves the model's context learning ability, and enables the model to extract the timbre and prosody information of speakers outside the training set based on given prompt speech, and then synthesizes the audio of the target speaker to achieve voice cloning.

[0146] 2. Improve the quality and timbre similarity of synthesized speech; design the model architecture based on the diffusion model of flow matching technology, which can capture the details of complex data distribution and achieve high-quality speech synthesis; introduce speaker vectors to further improve the timbre similarity of voice clones.

[0147] 3. Supports streaming voice interaction; The voice cloning model adopts a block-sample-based computation method, which can support non-streaming inference or streaming inference by adjusting the chunk size, realizing flexible and diverse generation methods. Streaming inference can also effectively reduce the latency of the first packet and the response time.

[0148] In one feasible implementation, the hifitts dataset, aishell3 dataset, and manually annotated dataset are used, which include both Chinese and English languages, with a total audio data duration of approximately 10,000 hours. One hour of audio data is randomly selected as the validation set, and the remaining audio data is used to construct the training sample set. In the actual implementation, all audio data is resampled to 24k, the Mel spectrum has a frame length of 1024, a frame shift of 256, and a dimension of 80.

[0149] In the constructed sound clone model, the text decoder has 6 Conformer layers, the decoder's downsampling module and upsampling module are both 2 layers, each layer contains 1 ResNet block and 4 Transformer layers, and the intermediate module consists of 12 Transformer layers.

[0150] In streaming computation, the chunk size is set to 52 frames, and the history cache length for the attention mechanism is set to 10 frames.

[0151] Verification showed that the voice cloning model can reproduce the speaker's timbre, style, and rhythm with high quality, provided that the prompt speech is longer than 3 seconds, for speakers outside the training sample set.

[0152] Reference Figure 8 The diagram illustrates a functional block diagram of a training device for a sound cloning model according to an embodiment of this application. The training device includes:

[0153] The training sample set construction module 101 is used to preprocess the acquired multiple audio data to construct a training sample set. Any training sample in the training sample set includes a text sequence sample, a Mel spectrum sample, and a target Mel spectrum label corresponding to any audio data.

[0154] Training module 102 is used to train a preset sound clone model based on the training sample set and a preset block size. The text encoder in the preset sound clone model is used to extract text vectors based on the text sequence samples of any training sample. The decoder in the preset sound clone model is used to output a predicted Mel spectrum based on the text vector, Mel spectrum sample and speaker vector of any training sample. The block size is used to adjust the calculation mode of the preset sound clone model to non-streaming or streaming.

[0155] The iteration module 103 is used to iteratively train the preset sound clone model based on the target Mel spectrum label, predicted Mel spectrum and loss function of the optimal transport stream of any training sample, so as to obtain the trained sound clone model.

[0156] Optionally, the module for constructing the training sample set includes a unit for constructing the training sample set, used for:

[0157] For any acquired audio data, generate the corresponding text sequence and Mel spectrum;

[0158] Perform a masking operation on the Mel spectrum corresponding to any audio data to obtain the masked Mel spectrum and the unmasked Mel spectrum. Use the unmasked Mel spectrum as the Mel spectrum sample and the masked Mel spectrum as the target Mel spectrum label.

[0159] Insert characters into the text sequence corresponding to the target Mel spectrum label in the audio data so that the length of the text sequence corresponding to the target Mel spectrum label is consistent with the frame number of the target Mel spectrum label, and use the text sequence corresponding to the Mel spectrum sample as the text sequence sample;

[0160] Use the text sequence sample, Mel spectrum sample, and target Mel spectrum label corresponding to any audio data as a training sample.

[0161] Optionally, the unit for constructing the training sample set includes a mask unit, used for:

[0162] For any audio data, perform a masking operation on the preset mask range of the Mel spectrum corresponding to the audio data, or perform a masking operation on the entire Mel spectrum corresponding to the audio data.

[0163] Optionally, the training device further includes:

[0164] The model building module is used to build preset sound clone models;

[0165] The preset sound cloning model includes a text encoder and a decoder. The text encoder includes a Conformer layer for extracting text vectors from any text sequence sample. The decoder is based on a Transformer structure in the form of U-net and includes a downsampling module, an intermediate module, and an upsampling module. Each of the downsampling module, the intermediate module, and the upsampling module consists of a residual module based on convolutional layers and a Transformer module based on an attention mechanism. Furthermore, any convolutional layer in the preset sound cloning model is a block convolution, and the attention mechanism of the Transformer module is a block mask-based attention mechanism.

[0166] Optionally, the training device further includes:

[0167] The block size setting module is used to determine the range of block size variation of the preset sound clone model during the training process in response to a preset operation. The block size is used to adjust the calculation mode of the preset sound clone model to non-streaming or streaming.

[0168] The training strategy setting module is used to execute a first training strategy or a second training strategy in response to the training strategy selection operation during the process of training a preset sound clone model based on the training sample set.

[0169] Specifically, when executing the first training strategy, during each iteration of training of the preset sound clone model, a block size is randomly selected from the range of block size variations as the block size for this iteration; when executing the second training strategy, the loss function in the non-streaming computation mode and the loss function in the streaming computation mode are determined respectively, and during the backpropagation of the preset sound clone model, the model parameters of the preset sound clone model are updated simultaneously based on the loss functions in the non-streaming computation mode and the streaming computation mode.

[0170] Reference Figure 9 The diagram illustrates a functional block diagram of a sound cloning device according to an embodiment of this application. The sound cloning device includes:

[0171] The acquisition module 201 is used to acquire the target prompt speech and the text sequence to be synthesized, and to generate the prompt Mel spectrum of the target prompt speech and the prompt text sequence.

[0172] The preprocessing module 202 is used to preprocess the text sequence to be synthesized and the prompt text sequence to obtain the text sequence to be input.

[0173] The Mel spectrum generation module 203 is used to input the Mel spectrum of the target prompt speech and the text sequence to be input into the sound cloning model, and output the Mel spectrum result corresponding to the text sequence to be synthesized through the sound cloning model. The sound cloning model is obtained by the training method of the sound cloning model as described in any one of claims 1-5.

[0174] The audio data conversion module 204 is used to convert the audio data corresponding to the text sequence to be input based on the Mel spectrum result.

[0175] Optionally, the preprocessing module includes a preprocessing unit for:

[0176] Based on the text sequence to be synthesized, estimate the number of frames of the Mel spectrum result corresponding to the text sequence to be synthesized;

[0177] Characters are inserted into the text sequence composed of the text sequence to be synthesized and the prompt text sequence to obtain the text sequence to be input. The length of the text sequence to be input is equal to the sum of the number of frames of the prompt Mel spectrum and the number of frames of the Mel spectrum result corresponding to the text sequence to be synthesized.

[0178] This application also provides an electronic device, including a memory, a processor, and a computer program stored in the memory. The processor executes the computer program to implement the training method for the sound cloning model as described in the embodiments, or to implement the sound cloning method as described in the embodiments.

[0179] This application also provides a computer-readable storage medium storing a computer program / instructions thereon, which, when executed by a processor, implements the training method for the sound cloning model as described in the embodiments, or implements the sound cloning method as described in the embodiments.

[0180] This application also provides a computer program product, including a computer program / instruction, which, when executed by a processor, implements the training method for the sound cloning model as described in the embodiments, or implements the sound cloning method as described in the embodiments.

[0181] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. The same or similar parts between the various embodiments can be referred to each other.

[0182] Those skilled in the art will understand that embodiments of this application can be provided as methods, apparatus, or computer program products. Therefore, embodiments of this application can take the form of entirely hardware embodiments, entirely software embodiments, or embodiments combining software and hardware aspects. Furthermore, embodiments of this application can take the form of computer program products implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0183] This application describes embodiments with reference to flowchart illustrations and / or block diagrams of methods, terminal devices (systems), and computer program products according to embodiments of this application. It should be understood that each block of the flowchart illustrations and / or block diagrams, as well as combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing terminal device to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing terminal device, generate instructions for implementing the flowchart illustrations. Figure 1 One or more processes and / or boxes Figure 1A device that provides the functions specified in one or more boxes.

[0184] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing terminal device to operate in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0185] These computer program instructions can also be loaded onto a computer or other programmable data processing terminal equipment, causing a series of operational steps to be performed on the computer or other programmable terminal equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable terminal equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0186] Although preferred embodiments of the present application have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments as well as all changes and modifications falling within the scope of the embodiments of the present application.

[0187] Finally, it should be noted that in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or terminal device that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or terminal device. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or terminal device that includes said element.

[0188] This document uses specific examples to illustrate the principles and implementation methods of this application. The descriptions of the above embodiments are only for the purpose of helping to understand the methods and core ideas of this application. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of this application. Therefore, the content of this specification should not be construed as a limitation of this application.

Claims

1. A training method for a sound cloning model, characterized in that, The method includes: Multiple audio data sets are preprocessed to construct a training sample set. Each training sample in the training sample set includes a text sequence sample, a Mel spectrum sample, and a target Mel spectrum label corresponding to any audio data set. Based on the training sample set and the preset block size, a preset sound clone model is trained. The text encoder in the preset sound clone model is used to extract text vectors based on the text sequence samples of any training sample. The decoder in the preset sound clone model is used to output a predicted Mel spectrum based on the text vector, Mel spectrum sample, and speaker vector of any training sample. The block size is used to adjust the calculation mode of the preset sound clone model to non-streaming or streaming. Based on the target Mel spectrum label, predicted Mel spectrum, and loss function of the optimal transport stream for any training sample, the preset sound cloning model is iteratively trained to obtain the trained sound cloning model.

2. The method according to claim 1, characterized in that, Multiple audio data sets were preprocessed to construct a training sample set, including: For any acquired audio data, generate the corresponding text sequence and Mel spectrum; Perform a masking operation on the Mel spectrum corresponding to any audio data to obtain the masked Mel spectrum and the unmasked Mel spectrum. Use the unmasked Mel spectrum as the Mel spectrum sample and the masked Mel spectrum as the target Mel spectrum label. Insert characters into the text sequence corresponding to the target Mel spectrum label in the audio data so that the length of the text sequence corresponding to the target Mel spectrum label is consistent with the frame number of the target Mel spectrum label, and use the text sequence corresponding to the Mel spectrum sample as the text sequence sample; Use the text sequence sample, Mel spectrum sample, and target Mel spectrum label corresponding to any audio data as a training sample.

3. The method according to claim 2, characterized in that, Masking the Mel spectrum corresponding to any audio data includes: For any audio data, perform a masking operation on the preset mask range of the Mel spectrum corresponding to the audio data, or perform a masking operation on the entire Mel spectrum corresponding to the audio data.

4. The method according to claim 1, characterized in that, The method further includes: Construct a pre-defined sound cloning model; The preset sound cloning model includes a text encoder and a decoder. The text encoder includes a Conformer layer for extracting text vectors from any text sequence sample. The decoder is based on a Transformer structure in the form of U-net and includes a downsampling module, an intermediate module, and an upsampling module. Each of the downsampling module, the intermediate module, and the upsampling module consists of a residual module based on convolutional layers and a Transformer module based on an attention mechanism. Furthermore, any convolutional layer in the preset sound cloning model is a block convolution, and the attention mechanism of the Transformer module is a block mask-based attention mechanism.

5. The method according to claim 4, characterized in that, The method further includes: In response to a preset operation, the range of block size variation of the preset sound clone model during the training process is determined, wherein the block size is used to adjust the calculation mode of the preset sound clone model to non-streaming or streaming. In response to the training strategy selection operation, during the process of training a preset sound clone model based on the training sample set, a first training strategy or a second training strategy is executed. Specifically, when executing the first training strategy, during each iteration of training of the preset sound clone model, a block size is randomly selected from the range of block size variations as the block size for this iteration; when executing the second training strategy, the loss function in the non-streaming computation mode and the loss function in the streaming computation mode are determined respectively, and during the backpropagation of the preset sound clone model, the model parameters of the preset sound clone model are updated simultaneously based on the loss functions in the non-streaming computation mode and the streaming computation mode.

6. A sound cloning method, characterized in that, The method includes: The target prompt speech and the text sequence to be synthesized are obtained, and the prompt Mel spectrum of the target prompt speech and the prompt text sequence are generated. Based on the text sequence to be synthesized and the prompt text sequence, the text sequence to be input is obtained through preprocessing. The Mel spectrum of the target prompt speech and the text sequence to be input are input into the sound cloning model, and the Mel spectrum result corresponding to the text sequence to be synthesized is output by the sound cloning model. The sound cloning model is obtained by the training method of the sound cloning model as described in any one of claims 1-5. Based on the Mel spectrum results, the audio data corresponding to the text sequence to be input is obtained.

7. The method according to claim 6, characterized in that, Based on the text sequence to be synthesized and the prompt text sequence, preprocessing is performed to obtain the text sequence to be input, including: Based on the text sequence to be synthesized, estimate the number of frames of the Mel spectrum result corresponding to the text sequence to be synthesized; Characters are inserted into the text sequence composed of the text sequence to be synthesized and the prompt text sequence to obtain the text sequence to be input. The length of the text sequence to be input is equal to the sum of the number of frames of the prompt Mel spectrum and the number of frames of the Mel spectrum result corresponding to the text sequence to be synthesized.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory, characterized in that, The processor executes the computer program to implement the training method for the sound cloning model as described in any one of claims 1-5, or to implement the sound cloning method as described in any one of claims 6-7.

9. A computer-readable storage medium having a computer program / instructions stored thereon, characterized in that, When the computer program / instruction is executed by the processor, it implements the training method of the sound cloning model as described in any one of claims 1-5, or the sound cloning method as described in any one of claims 6-7.

10. A computer program product comprising a computer program / instructions, characterized in that, When the computer program / instruction is executed by the processor, it implements the training method of the sound cloning model as described in any one of claims 1-5, or the sound cloning method as described in any one of claims 6-7.

Images