Speech cloning system and method fusing rhythm characteristics

Abstract

The invention discloses a voice cloning system and method fusing rhythm characteristics, belongs to the technical field of voice synthesis and natural language, and is applied to the aspect of fine-grained rhythm control in zero-sample voice synthesis. The implementation method comprises the following steps of: 1, extracting rhythm features and audio features of an audio file, and further respectively acquiring pause features, speed features and tone features in the rhythm features by sequentially adopting transcriptional text inverse coding, syllable-level speed registration quantization and pitch sequence feature splicing modes; 2, fusing the features of the audio files in a manner of discarding feature screening without guidance of a classifier; 3, generating a target Mel spectrogram based on conditional flow matching; generating a target audio file from a to-be-cloned audio file through the trained voice cloning model controlled by the fusion rhythm; compared with the prior art, fine-grained rhythm control of tone and rhythm feature decoupling is realized in zero-sample speech synthesis, so that intonation accuracy based on a context scene is improved.

Description

Technical Field

[0001] The present invention relates to a voice cloning system and method integrating prosodic features, belonging to the technical fields of speech synthesis and natural language, and is applied to fine-grained prosody control in zero-shot speech synthesis. Background Art

[0002] As a core branch of multi-modal large models, text-to-speech (TTS) technology has made remarkable breakthroughs driven by deep learning in recent years. It focuses on five directions: fast synthesis, low-resource adaptation, robustness improvement, expressiveness enhancement, and adaptive generation, and is widely used in fields such as navigation announcements, intelligent conversations, audiobooks, and assistive tools. Since Kratzenstein constructed the first human vocal tract model in 1779, its technological evolution has gone through three stages: The early synthesis method based on waveform splicing relied on the combination of speech library segments; in the statistical parameter synthesis stage, HMM / DNN models were used to generate acoustic parameters, and the long-term context modeling problem was solved through LSTM; the current mainstream is the end-to-end neural network synthesis stage, which directly inputs characters or phonemes, replaces traditional acoustic features with mel spectrograms, and implicitly aligns the text and speech sequences through the attention mechanism, significantly simplifying the explicit alignment steps of text analysis, acoustic modeling, and vocoder synthesis in the traditional process. Representative models such as Tacotron, FastSpeech, etc. jointly optimize the mapping of language features and acoustic features through an encoder-decoder architecture, improving the synthesis naturalness while reducing manual feature engineering. With the development of large model technology, a new framework based on diffusion models further enhances the expressiveness and controllability of the generated speech, promoting the evolution of speech synthesis technology towards the zero-shot direction and providing a better solution for personalized and low-resource voice cloning.

[0003] In terms of personalized feature restoration, existing end-to-end voice cloning methods are difficult to achieve precise decoupling control of the target speaker's timbre characteristics and prosodic features (such as intonation, speech rate, pause). The core problem lies in the strong coupling of the latent space encoding of timbre and prosody parameters - when the system attempts to adjust the prosodic features, the timbre representation will be irreversibly distorted due to parameter interference, and vice versa. This coupling effect mainly stems from the joint encoding mechanism of the reference audio in the existing framework: the timbre fingerprint and prosodic features are usually compressed into the same latent space, resulting in mutual penetration between feature dimensions.

[0004] Therefore, achieving fine-grained prosody control by decoupling timbre and prosodic features in zero-shot speech synthesis and thus improving the intonation accuracy based on context scenarios has become an urgent problem to be solved. Summary of the Invention

[0005] The object of the present invention is to address the technical problem of achieving fine-grained prosody control by decoupling timbre and prosody features in zero-shot speech synthesis and thereby improving the intonation accuracy based on context scenarios, and to propose a voice cloning system and method integrating prosody features.

[0006] The object of the present invention is achieved through the following technical solutions:

[0007] A voice cloning method integrating prosody features disclosed by the present invention includes the following steps:

[0008] Step 1: Extract prosody features and audio features from the audio file. Further, successively adopt the methods of transcription text anti-encoding, syllable-level speech rate registration quantization, and pitch sequence feature splicing to obtain the pause feature, speech rate feature, and pitch feature within the prosody features respectively;

[0009] Step 1.1: Extract the audio file address, transcription text, and audio duration; convert the audio file into a Mel spectrogram using the Fourier transform, and form a masked audio vector by masking and synthesizing the Mel spectrogram through random masking;

[0010] Step 1.2: Extract the prosody features of the audio file;

[0011] Step 1.2.1: Set that the prosody features consist of a pause feature, a speech rate feature, and a pitch feature;

[0012] Step 1.2.2: Use a multi-language pre-trained text-to-text converter to perform anti-encoding on the transcription text to obtain the pause feature, and mark the pause positions in the transcription text to form a pause-marked text;

[0013] Step 1.2.3: Obtain the speech rate feature using the syllable-level speech rate method and perform normalization processing;

[0014] Step 1.2.3.1: Use the voice activity detection method to remove the silence in the audio file to obtain the duration of the speech file;

[0015] Step 1.2.3.2: Obtain the number of speech syllables in the transcription text;

[0016] Step 1.2.3.3: Divide the number of speech syllables by the duration of the speech file to obtain the syllable-level speech rate;

[0017] Step 1.2.3.4: Normalize the syllable-level speech rate in the manner shown in Equation (1);

[0018]

[0019] where SR is the speech rate, SR min 、SR maxrespectively represent the minimum and maximum values of the speech rate; and SR normalized satisfies the numerical distribution of [0, 1];

[0020] Step 1.2.3.5: Quantize the normalized syllable-level speech rate into levels to form speech rate features;

[0021] Step 1.2.4: Interpolate and enhance the original pitch sequence, and then splice the pitch sequence features captured by the encoder and the statistical features to form pitch features;

[0022] Step 1.2.4.1: Extract the fundamental frequency of the audio file to generate the original pitch sequence, and use the interpolation enhancement method to compensate the pitch sequence for the original pitch sequence;

[0023] Step 1.2.4.1.1: Use linear interpolation to fill in the missing values of the pitch sequence for the original pitch sequence;

[0024] Step 1.2.4.1.2: Use a low-pass filter to denoise the original pitch sequence;

[0025] Step 1.2.4.1.3: Perturb and enhance the denoised pitch sequence with a random offset to obtain an enhanced pitch sequence;

[0026] Step 1.2.4.2: Encode the pitch sequence by splicing the pitch sequence features and the statistical features to form pitch features;

[0027] Step 1.2.4.2.1: Input the pitch sequence into a convolutional network to obtain local pitch sequence features;

[0028] Step 1.2.4.2.2: Use the self-attention mechanism to obtain global pitch sequence features from the local pitch sequence features of the pitch sequence input to the Transformer encoder and the pitch sequence;

[0029] Step 1.2.4.2.3: Obtain the pitch mean and variance of the pitch sequence to form statistical features;

[0030] Step 1.2.4.2.4: Concatenate the global pitch sequence features and the statistical features at the head and tail to obtain pitch features;

[0031] Step 2: Fuse the features of the audio file by the feature screening method of discarding under the guidance of no classifier;

[0032] Step 2.1: Obtain the feature embedding of the audio file and adjust the dimension;

[0033] Step 2.1.1: Randomly generate time steps, perform time position encoding on the time steps to obtain time embeddings, and perform non-linear mapping transformation on the time embeddings to generate time step embedding vectors;

[0034] Step 2.1.2: Use sine positional encoding to mark the positional information, add the positional encoding to the pause marker text to form a text embedding, and capture the text vector with pause features through a convolutional layer;

[0035] Step 2.1.3: Vectorize the speech rate feature and perform tensor expansion on the pitch feature to adjust them to the same dimension;

[0036] Step 2.2: Use classifier-free guidance to filter out the discarded features and form a fused feature;

[0037] Step 2.2.1: Set a random dropout probability parameter threshold, take any combination of the text vector of the pause feature, speech rate feature, pitch feature, and masked audio vector with the same dimension as the dropout condition, and use the dropout condition to filter the dimension of the discarded feature combination;

[0038] Step 2.2.2: Concatenate the head and tail of the undropped features in a predetermined dimension and linearly map them to the output dimension to form a concatenated feature, and further use convolutional positional encoding to enhance the positional information of the concatenated feature;

[0039] Step 2.2.3: Introduce the random dropout probability parameter into the classifier-free guidance, use the dropout condition to filter out those with the same dimension of the discarded feature as the unconditional target Mel spectrogram, and those with different dimensions of the discarded feature as the conditional target Mel spectrogram;

[0040] Step 2.2.4: Concatenate the head and tail of the text vector of the pause feature, speech rate feature, pitch feature, and masked audio vector with the same dimension in a predetermined dimension and linearly map them to the output dimension to form a concatenated feature, and further use convolutional positional encoding to enhance the positional information of the concatenated feature;

[0041] Step 2.2.5: Perform tensor expansion on the time step embedding vector, concatenate it with the concatenated feature with enhanced positional information along the time dimension, and use rotational positional encoding for secondary positional information enhancement to form a fused feature;

[0042] Step 3: Generate the target Mel spectrogram based on conditional flow matching;

[0043] Step 3.1: Perform noise perturbation and flow field construction on the Mel spectrogram;

[0044] Step 3.1.1: Use tensor initialization technology to randomly generate Gaussian noise with the same shape as the Mel spectrogram;

[0045] Step 3.1.2: Based on flow matching, obtain the flow matching intermediate state as shown in Equation (2) through linear interpolation;

[0046]

[0047] Among them, x0 is Gaussian noise, x1 is the Mel spectrogram, and t is the time step. represents the intermediate state at time t;

[0048] Step 3.1.3: Calculate the displacement field as shown in Equation (3) to obtain the direction vector field flow of the inverse diffusion process.

[0049] flow = x1 - x0 (3)

[0050] Among them, x0 is Gaussian noise, x1 is the Mel spectrogram, and flow represents the single-point displacement in the direction vector field from x0 to x1.

[0051] Step 3.2: Generate the Mel spectrogram based on conditional flow matching.

[0052] Step 3.2.1: Enhance the quadratic position information to form the fused features and the intermediate state of flow matching, and input them into the U-Net Transformer hierarchical architecture for multi-scale feature refinement. Use the first N / 2 layers as the encoder to extract multi-level acoustic features, and use the last N / 2 layers as the decoder to achieve feature reuse through the dynamic skip connection mechanism. Then output the high-order acoustic features through the attention layer and the feed-forward layer in sequence.

[0053] Step 3.2.2: Normalize the high-order acoustic features using RMSNorm as shown in Equation (4) to obtain the normalized high-order acoustic features.

[0054]

[0055] Among them, x i represents each element in the input tensor, and N is the dimension size of the input.

[0056] Step 3.2.3: Generate the Mel spectrogram from the normalized high-order acoustic features through a linear layer.

[0057] Step 3.3: Constrain the generator trained by conditional flow matching using a composite loss function.

[0058] Step 3.3.1: Obtain the flow matching loss through Equation (5) using the generated Mel spectrogram, direction vector field, and random mask.

[0059]

[0060] Among them, is used to calculate the point-wise mean squared error between the predicted flow field and the true flow field. t ∼ U(0,1) is the time step sampling, is the data sampling; represents the data point Position at time t, true velocity Indicates the true motion direction of the data point at time t, and y is the fused feature;

[0061] Step 3.3.2: Calculate the point-by-point mean square error between the pitch sequence extracted from the generated Mel spectrogram and the pitch sequence extracted from the original Mel spectrogram to obtain the pitch loss;

[0062] Step 3.3.3: Extract the audio duration from the generated Mel spectrogram, and calculate the point-by-point mean square error with the audio duration extracted from the audio file to obtain the duration loss;

[0063] Step 3.3.4: Compound the flow matching loss, pitch loss, and duration loss through Equation (6)

[0064] to form a compound loss function for constraining prosodic features;

[0065] L total = L CFM + λ1L pitch + λ2L duration (6)

[0066] where λ1 is the pitch weight and λ2 is the duration weight;

[0067] Step 3.4: Input the transcribed text and the target pitch features into the BERT-based cross-modal pitch prediction model for training to obtain the pitch feature vector under the voice cloning model with fused prosody control, realizing the reference-free mapping from text to pitch;

[0068] Step 3.4.1: Input the transcribed text and the target pitch features into the pre-trained BERT model, generate 768-dimensional context-aware word embedding vectors through 12 layers of Transformer encoders, and perform average pooling on the word vectors corresponding to each syllable to form syllable-level text feature representations;

[0069] Step 3.4.2: Use a three-layer MLP network to map the text features to preliminary pitch features;

[0070] Step 3.4.3: Establish the association between the text features and the pitch features through a 4-head attention mechanism, and output the pitch feature vector;

[0071] Step 3.4.4: Obtain the L2 distance loss between the predicted pitch and the true pitch in the manner shown in Equation (7) and train the pitch prediction model;

[0072] where V pred is the predicted pitch vector, V true is the true pitch vector, and N is the number of samples;

[0073] Step 4: Generate a target audio file for the audio file to be cloned through a trained voice cloning model with fused prosody control;

[0074]

[0075] Step 4.1: Input the reference audio, reference text, and target text into the trained voice cloning model with fused prosody control;

[0076] Step 4.2: Generate prosodic features for the audio file to be cloned;

[0077] Step 4.2.1: Set the speech rate level for the audio file to be cloned in a hierarchical quantization manner;

[0078] Step 4.2.2: Based on the syllable division of the target text, generate pitch feature vectors using the trained voice cloning model with fused prosody control;

[0079] Step 4.2.3: Perform text pause annotation on the target text;

[0080] Step 4.2.4: Use the trained voice cloning model with fused prosody control to generate the target Mel spectrogram;

[0081] Step 4.2.5: Convert the generated target Mel spectrogram into a speech waveform through a Vocos vocoder to obtain the target audio;

[0082] A voice cloning system with fused prosodic features disclosed by the present invention is used to implement the above method; the system includes a feature extraction module, a feature fusion module, and a model construction and training module;

[0083] The feature extraction module is used to perform Mel spectrogram conversion and prosodic feature extraction on the audio file, including pause feature marking, syllable-level speech rate calculation, and pitch sequence enhancement processing; the audio waveform and transcription text are used as the inputs of the module;

[0084] The feature fusion module is used to perform dimensionality adjustment and classifier-free guided feature screening and fusion on the audio features and prosodic features; the Mel spectrogram vector, pause marker text, speech rate feature vector, and pitch feature vector are used as the inputs of the module; through time step embedding, random feature dropout, and multiple position encoding enhancement, it outputs a fused feature vector with temporal information;

[0085] The model construction and training module is used to generate and optimize the target Mel spectrogram based on conditional flow matching; take the fused feature vector and Mel spectrogram as the input of the module; output high-quality synthetic Mel spectrogram through noise perturbation and flow field construction, UNet-Transformer multi-scale feature refinement, and composite loss function constraint; and take the generated Mel spectrogram as the input of the vocoder module.

[0086] Compared with the prior art, the present invention has the following beneficial effects:

[0087] 1. By introducing a text-based independent regulation mechanism for speech rate, pause, and pitch, the present invention realizes fine-grained control of speech prosody. Compared with the prior art that relies on reference audio or single prosody parameter adjustment, the present invention combines the mT5 model to predict the pause position and intensity and a pitch predictor based on text input to dynamically generate pause positions and pitch contours that conform to the context semantics under the condition of no reference audio, significantly improving the expressiveness and naturalness of synthetic speech.

[0088] 2. Compared with the prior speech cloning technology, which often has insufficient personalized restoration in terms of pauses and speech rate due to the coupling of timbre and prosody features, based on an end-to-end flow matching model, the prosody encoder is separated as a generation condition and combined with a classifier-free guidance strategy to effectively decouple the timbre features and prosody attributes of the target speaker. Furthermore, while maintaining the timbre consistency, the speech rate and pause pattern can be freely adjusted, expanding the personalized application scenarios of speech cloning.

[0089] 3. The system of the present invention effectively solves the problem of fuzzy prosody features in traditional speech cloning systems through an innovative prosody decoupling design and dynamic fusion mechanism, and can generate synthetic speech with high fidelity and natural prosody, which can be widely applied to scenarios such as virtual characters and audiobook production that require high-quality speech cloning. Description of the Drawings

[0090] Figure 1 It is a schematic diagram of the training process of the present invention;

[0091] Figure 2 It is a schematic diagram of the inference process of the present invention. Detailed Embodiments

[0092] To better illustrate the purpose and advantages of the present invention, the content of the invention will be further described below with reference to the drawings and examples. It should be noted that the implementation of the present invention is not limited to the following examples, and any form of modification or change to the present invention will fall within the protection scope of the present invention.

[0093] The present invention extracts multi-level prosodic features from the original audio signal, then realizes the cross-modal fusion of acoustic features and prosodic features through a feature fusion module, and finally uses conditional flow matching generation technology to enable the model to learn both the acoustic characteristics and prosodic characteristics of speech simultaneously, thereby achieving high-quality voice cloning. This system can accurately capture and reproduce the prosodic features of the source speech, including key elements such as pauses, speech rate, and pitch, and generate natural and fluent target speech.

[0094] Embodiment

[0095] As Figure 1 shown, a voice cloning method for fusing prosodic features in this embodiment is specifically implemented as follows:

[0096] Step 1: Extract prosodic features and audio features from the audio file. Further, in sequence, adopt the methods of transcription text decoding, syllable-level speech rate registration quantization, and pitch sequence feature splicing to obtain the pause feature, speech rate feature, and pitch feature within the prosodic features respectively;

[0097] Step 1.1: Extract the audio file address, transcription text, and audio duration; convert the audio file into a Mel spectrogram using the Fourier transform, and form a masked audio vector by masking and synthesizing the Mel spectrogram through random masking;

[0098] In the embodiment, the publicly available EMILIA dataset is adopted to extract the audio file address, transcription text, and audio duration; convert the audio file into a Mel spectrogram using the Fourier transform; form a masked audio vector by masking and synthesizing the Mel spectrogram through random masking;

[0099] Step 1.2: Extract the prosodic features of the audio file;

[0100] Step 1.2.1: Set that the prosodic features consist of a pause feature, a speech rate feature, and a pitch feature;

[0101] Step 1.2.2: Use a multi-language pre-trained text-to-text converter to decode the transcription text to obtain the pause feature, and mark the pause positions of the transcription text to form a pause-marked text;

[0102] In the embodiment, the publicly available Chinese standard female voice library and DeepSeek-based English and English-mixed pause annotation data are adopted, and the integrated pause dataset is obtained through manual inspection and screening. The integrated pause dataset is input into the mT5-large pre-trained model, and the pause annotation model is obtained through fine-tuning training.

[0103] Step 1.2.3: Use the syllable-level speech rate method to obtain the speech rate feature and perform normalization processing;

[0104] Step 1.2.3.1: Remove the silence from the audio file using the voice activity detection method to obtain the duration of the speech file;

[0105] Step 1.2.3.2: Obtain the number of speech syllables in the transcribed text;

[0106] Step 1.2.3.3: Divide the number of speech syllables by the duration of the speech file to obtain the syllable-level speech rate;

[0107] Step 1.2.3.4: Normalize the syllable-level speech rate in the manner shown in Equation (1);

[0108]

[0109] where SR is the speech rate, SR min and SR max represent the minimum and maximum values of the speech rate respectively; and SR normalized satisfies the numerical distribution of [0,1];

[0110] Step 1.2.3.5: Quantize the normalized syllable-level speech rate into levels to form the speech rate feature;

[0111] In the embodiment, the extraction process of the speech rate information is based on the accurate calculation of the audio duration and the text syllables, and the method of separate processing for Chinese and English is adopted: for Chinese, each Chinese character is regarded as a syllable; for English, the pyphen library is used for syllable division. First, the voice activity detection (VAD) technology is used to remove the silent part in the audio to obtain the effective speaking duration; second, the number of syllables is calculated according to the above-mentioned Chinese-English divide-and-conquer statistical method; finally, according to the ratio of the total number of syllables in the text to the effective duration, the syllable-level speech rate is calculated. Based on the data distribution, the normalized speech rate is divided into five levels: extremely slow, slow, medium, fast, and extremely fast, corresponding to the 20th, 40th, 60th, 80th percentiles respectively, to more reasonably reflect the actual distribution characteristics of the speech rate, and finally the speech rate feature is output.

[0112] Step 1.2.4: Interpolate and enhance the original pitch sequence, and splice the pitch sequence features captured by the encoder and the statistical features to form the pitch feature;

[0113] Step 1.2.4.1: Extract the fundamental frequency of the audio file to generate the original pitch sequence, and use the interpolation enhancement method to compensate the pitch sequence for the original pitch sequence;

[0114] Step 1.2.4.1.1: Use linear interpolation to fill the missing values of the pitch sequence for the original pitch sequence;

[0115] Step 1.2.4.1.2: Use a low-pass filter to denoise the original pitch sequence;

[0116] Step 1.2.4.1.3: Perform random offset perturbation enhancement on the denoised pitch sequence to obtain an enhanced pitch sequence;

[0117] Step 1.2.4.2 encodes the pitch sequence by splicing pitch sequence features and statistical features to form pitch features;

[0118] Step 1.2.4.2.1: Input the pitch sequence into a convolutional network to obtain local features of the pitch sequence;

[0119] Step 1.2.4.2.2: Use the self-attention mechanism to obtain global features of the pitch sequence from the local features of the pitch sequence input to the Transformer encoder and the pitch sequence;

[0120] Step 1.2.4.2.3: Obtain the pitch mean and variance of the pitch sequence to form statistical features;

[0121] Step 1.2.4.2.4: Concatenate the global features of the pitch sequence and the statistical features at the head and tail to obtain pitch features;

[0122] In the embodiment, the fundamental frequency F0 is extracted from the original audio by using the PyWorld tool to generate the original pitch sequence. Since there may be noise or silent segments in the audio signal, the original pitch sequence usually contains missing values and outliers such as zero values or mutation points. For the interpolation and smoothing processing of the pitch sequence, the missing points are filled by linear interpolation, and a low-pass filter such as moving average or Gaussian filter is used to eliminate high-frequency noise to generate a continuous and smooth pitch contour; to further enhance the generalization ability of the model to the pitch shape pattern, a perturbation operation is introduced: subtract a random offset from each pitch point, that is, uniformly sample from the preset fundamental frequency range [fmin, fmax], retain the overall shape characteristics of the pitch curve, and at the same time weaken the dependence on the specific fundamental frequency value. The pitch sequence is encoded by using the method of splicing pitch shape features and statistical features to generate a pitch representation vector with a fixed dimension. First, a one-dimensional convolutional network CNN is used to extract local patterns of the pitch sequence, such as short-term rising, falling, and tremor changes, so as to enhance the modeling ability of local dynamic features. Subsequently, a Transformer encoder is adopted to capture the global pitch change trend by using the self-attention mechanism, and the overall shape encoding is extracted by mean pooling to output pitch features.

[0123] Step 2: Fuse the features of the audio file by the feature screening method of dropout without a classifier;

[0124] Step 2.1: Obtain the feature embedding of the audio file and perform dimension adjustment;

[0125] Step 2.1.1: Randomly generate time steps, perform time position encoding on the time steps to obtain time embeddings, and perform a non-linear mapping transformation on the time embeddings to generate time step embedding vectors;

[0126] Step 2.1.2: Use sine position encoding to mark position information, add the position encoding to the pause marker text to form text embeddings, and capture text vectors with pause features through a convolutional layer;

[0127] Step 2.1.3: Vectorize the speech rate feature and perform tensor expansion adjustment on the pitch feature to the same dimension;

[0128] Step 2.2: Use classifier-free guidance to perform discarded feature screening to form fused features;

[0129] Step 2.2.1: Set a random dropout probability parameter threshold, use any combination of features in the text vector of pause features, speech rate feature, pitch feature, and masked audio vector with the same dimension as the dropout condition, and use the dropout condition to screen the dimensions of the discarded feature combinations;

[0130] Step 2.2.2: Concatenate the features that are not discarded at the beginning and end in a predetermined dimension and linearly map them to the output dimension to form concatenated features, and further use convolutional position encoding to enhance the position information of the concatenated features;

[0131] Step 2.2.3: Introduce the random dropout probability parameter into the classifier-free guidance, use the dropout condition to screen out those with the same discarded feature dimensions as the unconditional target Mel spectrogram, and those with different discarded feature dimensions as the conditional target Mel spectrogram;

[0132] Step 2.2.4: Concatenate the text vector of pause features, speech rate feature, pitch feature, and masked audio vector with the same dimension at the beginning and end in a predetermined dimension and linearly map them to the output dimension to form concatenated features, and further use convolutional position encoding to enhance the position information of the concatenated features;

[0133] Step 2.2.5: Perform tensor expansion on the time step embedding vector, concatenate it with the concatenated features with enhanced position information along the time dimension, and use rotational position encoding for secondary position information enhancement to form fused features;

[0134] In an embodiment, first, dimensional unification processing is performed on the features of the audio file, including generating time step embedding vectors through time position encoding, enhancing pause marker text using sine position encoding, and adjusting the speech rate feature and pitch feature to the same dimension. Subsequently, a classifier-free guidance feature screening mechanism is introduced, and various feature combinations are dynamically screened by setting a random dropout probability. After the retained features are concatenated head-to-tail and linearly mapped, convolutional position encoding and rotational position encoding are successively used for dual position information enhancement. Among them, the time step embedding vector is concatenated with the processed features in the time dimension, and finally, a fused feature vector with rich temporal information is generated. This embodiment specifically designs a generation strategy for conditional / unconditional target mel spectrograms, and different guided conditions of fused features are generated through the difference in the feature dropout dimension.

[0135] Step 3: Generate a target mel spectrogram based on conditional flow matching;

[0136] Step 3.1: Perform noise perturbation and flow field construction on the mel spectrogram;

[0137] Step 3.1.1: Randomly generate Gaussian noise with the same shape as the mel spectrogram using tensor initialization technology;

[0138] Step 3.1.2: Obtain the intermediate state of flow matching as shown in Equation (2) through linear interpolation based on flow matching;

[0139]

[0140] where x0 is Gaussian noise, x1 is the mel spectrogram, t is the time step, represents the intermediate state at time t;

[0141] Step 3.1.3: Calculate the displacement field as shown in Equation (3) to obtain the direction vector field flow of the inverse diffusion process;

[0142] flow = x1 - x0 (3)

[0143] where x0 is Gaussian noise, x1 is the mel spectrogram, and flow represents the single-point displacement in the direction vector field from x0 to x1;

[0144] Step 3.2: Generate a mel spectrogram based on conditional flow matching;

[0145] Step 3.2.1: Input the fused feature formed by enhancing the quadratic position information and the intermediate state of flow matching into the UNet-Transformer hierarchical architecture for multi-scale feature refinement. Use the first N / 2 layers as the encoder to extract multi-level acoustic features, and use the last N / 2 layers as the decoder to achieve feature reuse through a dynamic skip connection mechanism. Successively output high-order acoustic features through the attention layer and the feed-forward layer;

[0146] Step 3.2.2: Normalize the high-order acoustic features using RMSNorm as shown in Equation (4) to obtain the normalized high-order acoustic features;

[0147]

[0148] where x i represents each element in the input tensor, and N is the dimensionality size of the input;

[0149] Step 3.2.3: Generate the mel spectrogram from the normalized high-order acoustic features through a linear layer;

[0150] Step 3.3: Constrain the generator trained with conditional flow matching using a composite loss function;

[0151] Step 3.3.1: Obtain the flow matching loss through Equation (5) using the generated mel spectrogram, the direction vector field, and a random mask;

[0152]

[0153] where is used to calculate the point-wise mean squared error between the predicted flow field and the true flow field, t ∼ U(0,1) is the time step sampling, is the data sampling; represents the data point at time t, and the true velocity represents the true motion direction of the data point at time t, and y is the fused feature;

[0154] Step 3.3.2: Calculate the point-wise mean squared error between the pitch sequence extracted from the generated mel spectrogram and the pitch sequence extracted from the original mel spectrogram to obtain the pitch loss;

[0155] Step 3.3.3: Extract the audio duration from the generated mel spectrogram and calculate the point-wise mean squared error with the audio duration extracted from the audio file to obtain the duration loss;

[0156] Step 3.3.4: Combine the flow matching loss, the pitch loss, and the duration loss through Equation (6),

[0157] to form a composite loss function for constraining the prosodic features;

[0158] L total = L CFM + λ1L pitch + λ2L duration (6)

[0159] where λ1 is the pitch weight and λ2 is the duration weight;

[0160] In the embodiment, first, noise perturbation and flow field construction are performed: Gaussian noise x0 with the same shape as the target Mel spectrum is generated through tensor initialization technology, and the intermediate state φ t is calculated based on the linear interpolation formula φ t =(1 - t)·x0 + t·x1, where x1 is the target Mel spectrum and t is the time step. Subsequently, the direction vector field is calculated according to the displacement field formula flow = x1 - x0; then, the fused feature and the flow-matched intermediate state are input into the UNet-Transformer architecture for multi-scale feature refinement: the encoder part extracts multi-level acoustic features, and the decoder realizes feature reuse through dynamic skip connections, and outputs high-order features through the attention layer and the feed-forward layer. RMSNorm normalization is adopted, and the final Mel spectrum is generated through a linear layer; finally, a composite loss function is constructed for optimization, realizing the generation of high-quality Mel spectra, and saving the trained voice cloning model with fused prosody control;

[0161] Step 3.4: Input the transcribed text and the target pitch feature into the cross-modal pitch prediction model based on BERT for training to obtain the pitch feature vector under the voice cloning model with fused prosody control, realizing the text-to-pitch non-reference mapping;

[0162] Step 3.4.1: Input the transcribed text and the target pitch feature into the pre-trained BERT model, generate 768-dimensional context-aware word embedding vectors through 12-layer Transformer encoders, and perform average pooling on the word vectors corresponding to each syllable to form syllable-level text feature representations;

[0163] Step 3.4.2: Adopt a three-layer MLP network to map the text features into preliminary pitch features;

[0164] Step 3.4.3: Establish the association between the text features and the pitch features through a 4-head attention mechanism, and output the pitch feature vector;

[0165] Step 3.4.4: Obtain the L2 distance loss between the predicted pitch and the true pitch in the manner shown in Equation (7) and train the pitch prediction model;

[0166]

[0167] where V pred is the predicted pitch vector, V true is the true pitch vector, and N is the number of samples;

[0168] Step 4: Generate the target audio file for the audio file to be cloned through the trained voice cloning model with fused prosody control;

[0169] Step 4.1: Input the reference audio, reference text, and the trained voice cloning model with integrated prosody control;

[0170] Step 4.2: Generate prosody features for the audio file to be cloned;

[0171] Step 4.2.1: Set the speech rate level for the audio file to be cloned using hierarchical quantization;

[0172] Step 4.2.2: Based on the syllable segmentation of the target text, use the trained voice cloning model with integrated prosody control to generate pitch feature vectors;

[0173] Step 4.2.3: Perform text pause annotation on the target text;

[0174] Step 4.2.4: Use the trained voice cloning model with integrated prosody control to generate the target Mel spectrogram;

[0175] Step 4.2.5: Convert the generated target Mel spectrogram into a speech waveform through the Vocos vocoder to obtain the target audio;

[0176] In the embodiment, as Figure 2 shown, input the target text such as "I am who I am, the great Taiyi Zhenren admired by all!", and input the reference audio and reference text of the target speaker. Set the target speech rate levels from 1 to 5 according to the cloning requirements, and set the speech rate level to 4 in this embodiment; use the multi - language pre - trained text - to - text converter to perform text pause annotation on the target text. For pause information annotation, manual annotation of pause information or text pause annotation through the pre - trained model can be selected for the target text; use the trained pitch predictor to predict the pitch of the target text, take all prosody features, reference audio, and reference text as the input of the model, use the trained voice cloning model with integrated prosody control to generate the target Mel spectrogram, and convert the Mel spectrogram into an audio file through the Vocos vocoder;

[0177] A voice cloning system with integrated prosody features in this embodiment is used to implement the above - mentioned method; the system includes a feature extraction module, a feature fusion module, and a model construction and training module;

[0178] The feature extraction module is used to perform Mel spectrogram conversion and prosody feature extraction on the audio file, including pause feature marking, syllable - level speech rate calculation, and pitch sequence enhancement processing; take the audio waveform and the transcribed text as the input of the module;

[0179] The feature fusion module is used to perform dimensionality adjustment and classifier-free guided feature screening and fusion on audio features and prosodic features; takes the Mel spectrogram vector, pause marker text, speech rate feature vector, and pitch feature vector as the input of the module; and outputs a fused feature vector with temporal information through time step embedding, random feature dropout, and multiple position encoding enhancements.

[0180] The model construction and training module is used to generate and optimize the target Mel spectrogram based on conditional flow matching; takes the fused feature vector and Mel spectrogram as the input of the module; outputs a high-quality synthesized Mel spectrogram through noise perturbation and flow field construction, UNet-Transformer multi-scale feature refinement, and composite loss function constraints; and takes the generated Mel spectrogram as the input of the vocoder module.

[0181] The above specific description further details the purpose, technical solution, and beneficial effects of the invention. It should be understood that the above is only a specific embodiment of the present invention and is not used to limit the protection scope of the present invention. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A voice cloning method integrating prosodic features, characterized in that: It includes the following steps: Step 1: Extract prosodic features and audio features from the audio file. Further, successively adopt the methods of transcription text anti-encoding, syllable-level speech rate registration quantization, and pitch sequence feature splicing to obtain the pause feature, speech rate feature, and pitch feature within the prosodic features respectively; Step 2: Fuse the features of the audio file through a feature screening method guided by a classifier-free; Step 2.1: Obtain the feature embedding of the audio file and perform dimensionality adjustment; Step 2.1.1: Randomly generate time steps, perform time position encoding on the time steps to obtain time embeddings, and perform a non-linear mapping transformation on the time embeddings to generate time step embedding vectors; Step 2.1.2: Use sine position encoding to mark position information, add the position encoding to the pause-marked text to form text embeddings, and capture text vectors with pause features through a convolutional layer; Step 2.1.3: Perform tensor expansion adjustment on the speech rate feature vectorized with the pitch feature to the same dimension; Step 2.2: Use a classifier-free guidance to perform discarded feature screening to form fused features; Step 2.2.1: Set a random dropout probability parameter threshold, take any combination of features in the text vector of the pause feature, speech rate feature, pitch feature, and masked audio vector with the same dimension as the dropout condition, and use the dropout condition to screen the dimensions of the discarded feature combinations; Step 2.2.2: Concatenate the head and tail of the non-discarded features in a predetermined dimension and linearly map them to the output dimension to form concatenated features. Further, use convolutional position encoding to enhance the position information of the concatenated features; Step 2.2.3: Introduce the random dropout probability parameter into the classifier-free guidance, use the dropout condition to screen out those with the same dimension of discarded feature dimensions as the unconditional target Mel spectrogram, and those with different dimensions of discarded feature dimensions as the conditional target Mel spectrogram; Step 2.2.4: Concatenate the head and tail of the text vector of the pause feature, speech rate feature, pitch feature, and masked audio vector with the same dimension in a predetermined dimension and linearly map them to the output dimension to form concatenated features. Further, use convolutional position encoding to enhance the position information of the concatenated features; Step 2.2.5: Perform tensor expansion on the time step embedding vectors and concatenate them with the position information-enhanced concatenated features along the time dimension, and use rotational position encoding for secondary position information enhancement to form fused features; Step 3: Generate the target Mel spectrogram based on conditional flow matching; Step 3.1: Perform noise perturbation and flow field construction on the Mel spectrogram; Step 3.1.1: Use tensor initialization technology to randomly generate Gaussian noise with the same shape as the Mel spectrogram; Step 3.1.2: Based on flow matching, obtain the intermediate state of flow matching as shown in Equation (2) through linear interpolation; where, x0 is Gaussian noise, x1 is Mel spectrogram, and t is the time step, representing the intermediate state at time t; Step 3.1.3: Calculate the displacement field as shown in Equation (3) to obtain the direction vector field flow of the inverse diffusion process; flow = x1 - x0 (3) where x0 is the Gaussian noise, x1 is the Mel spectrogram, and flow represents the single-point displacement amount in the direction vector field from x0 to x1; Step 3.2: Generate Mel spectrogram based on conditional flow matching; Step 3.2.1: Enhance the secondary position information to form fused features and the intermediate state of flow matching, and input them into the UNet-Transformer hierarchical architecture for multi-scale feature refinement. Use the first N / 2 layers as the encoder to extract multi-level acoustic features, and the last N / 2 layers as the decoder to achieve feature reuse through the dynamic skip connection mechanism. Then, output high-order acoustic features through the attention layer and the feed-forward layer in sequence; Step 3.2.2: Normalize the high-order acoustic features using RMSNorm as shown in Equation (4) to obtain the normalized high-order acoustic features; where x i represents each element in the input tensor, and N is the dimensionality size of the input; Step 3.2.3: Generate Mel spectrogram from the normalized high-order acoustic features through a linear layer; Step 3.3: Use a composite loss function to constrain the generator trained by conditional flow matching; Step 3.4: Input the transcribed text and the target pitch features into the cross-modal pitch prediction model based on BERT for training to obtain the pitch feature vector under the voice cloning model with fused prosody control, and achieve text-to-pitch non-reference mapping; Step 4: Generate the target audio file for the audio file to be cloned through the trained voice cloning model with fused prosody control; Step 4.1: Input the reference audio, reference text, and target text into the trained voice cloning model with fused prosody control; Step 4.2: Generate prosody features for the audio file to be cloned; Step 4.2.1: Set the speech rate level for the audio file to be cloned in a hierarchical quantization manner; Step 4.2.2: Based on the syllable segmentation of the target text, use the trained voice cloning model with fused prosody control to generate pitch feature vectors; Step 4.2.3: Perform text pause annotation on the target text; Step 4.2.4: Use the trained voice cloning model with fused prosody control to generate the target Mel spectrogram; Step 4.2.5: Convert the generated target Mel spectrogram into a speech waveform through the Vocos vocoder to obtain the target audio.

2. The voice cloning method integrating prosodic features according to claim 1, characterized in that: The implementation method of Step 1 is as follows: Step 1.1: Extract the audio file address, transcribed text, and audio duration; convert the audio file into a Mel spectrogram using Fourier transform, and synthesize the masked audio vector by randomly masking the Mel spectrogram; Step 1.2: Extract the prosody features of the audio file.

3. The voice cloning method integrating prosodic features according to claim 2, wherein: The implementation method of Step 1.2 is as follows: Step 1.2.1: Set that the prosody features consist of pause features, speech rate features, and pitch features; Step 1.2.2: Use the multi-lingual pre-trained text-to-text converter to inverse-encode the transcribed text to obtain pause features, and mark the pause positions of the transcribed text to form a pause-marked text; Step 1.2.3: Obtain the speech rate features using the syllable-level speech rate method and perform normalization; Step 1.2.4: After interpolating and enhancing the original pitch sequence, splice the pitch sequence features captured by the encoder and the statistical features to form pitch features.

4. The voice cloning method integrating prosodic features according to claim 3, characterized in that: The implementation method of Step 1.2.3 is as follows: Step 1.2.3.1: Use the voice activity detection method to remove the silence in the audio file to obtain the duration of the speech file; Step 1.2.3.2: Obtain the number of speech syllables in the transcribed text; Step 1.2.3.3: Divide the number of speech syllables by the duration of the speech file to obtain the syllable-level speech rate; Step 1.2.3.4: Normalize the syllable-level speech rate in the manner shown in Equation (1); Among them, SR is the speech rate, SR min , SR max respectively represent the minimum and maximum values of the speech rate; and SR normalized satisfies the numerical distribution of [0,1]; Step 1.2.3.5: Quantize the normalized syllable-level speech rate into levels to form the speech rate feature.

5. The voice cloning method integrating prosodic features according to claim 3, wherein: The implementation method of Step 1.2.4 is as follows. Step 1.2.4.1: Extract the fundamental frequency of the audio file to generate the original pitch sequence, and compensate the pitch sequence using the interpolation enhancement method for the original pitch sequence; Step 1.2.4.2: Encode the pitch sequence by concatenating the pitch sequence features and statistical features to form the pitch feature.

6. The voice cloning method integrating prosodic features according to claim 5, wherein: The implementation method of Step 1.2.4.1 is as follows. Step 1.2.4.1.1: Use linear interpolation to fill in the missing values of the original pitch sequence for the original pitch sequence; Step 1.2.4.1.2: Denoise the original pitch sequence using a low-pass filter; Step 1.2.4.1.3: Perform random offset perturbation enhancement on the denoised pitch sequence to obtain the enhanced pitch sequence.

7. The voice cloning method integrating prosodic features according to claim 5, characterized in that: The implementation method of Step 1.2.4.2 is as follows. Step 1.2.4.2.1: Input the pitch sequence into the convolutional network to obtain the local features of the pitch sequence; Step 1.2.4.2.2: Use the self-attention mechanism to obtain the global features of the pitch sequence from the local features of the pitch sequence input to the Transformer encoder and the pitch sequence; Step 1.2.4.2.3: Obtain the pitch mean and variance of the pitch sequence to form the statistical features; Step 1.2.4.2.4: Concatenate the global features of the pitch sequence and the statistical features at the head and tail to obtain the pitch feature.

8. The voice cloning method integrating prosody features according to claim 1, characterized in that: The implementation method of Step 3.3 is as follows. Step 3.3.1: Use the generated Mel spectrogram, direction vector field, and random mask to obtain the flow matching loss through Equation (5); Among them, is used to calculate the point-by-point mean square error between the predicted flow field and the true flow field. t~U(0,1) is the time step sampling, is the data sampling; represents the data point at the position at time t, and the true velocity represents the true motion direction of the data point at time t, and y is the fusion feature; Step 3.3.2: Calculate the pointwise mean squared error between the pitch sequence extracted from the generated Mel spectrogram and the pitch sequence extracted from the original Mel spectrogram to obtain the pitch loss; Step 3.3.3: Extract the audio duration in the generated Mel spectrogram and calculate the pointwise mean squared error with the audio duration extracted from the audio file to obtain the duration loss; Step 3.3.4: Compound the flow matching loss, pitch loss, and duration loss through Equation (6) to form a compound loss function for constraining the prosody features; L total = L CFM + λ1L pitch + λ2L duration (6) where λ1 is the pitch weight and λ2 is the duration weight.

9. The voice cloning method integrating prosodic features according to claim 1, wherein: The implementation method of Step 3.4 is as follows. Step 3.4.1: Input the transcribed text and the target pitch feature into the pre-trained BERT model, generate 768-dimensional context-aware word embedding vectors through the 12-layer Transformer encoder, and perform average pooling on the word vectors corresponding to each syllable to form the syllable-level text feature representation; Step 3.4.2: Use a three-layer MLP network to map the text features to the preliminary pitch features; Step 3.4.3: Establish the association between the text features and the pitch features through the 4-head attention mechanism and output the pitch feature vector. Step 3.4.4: Obtain the L2 distance loss between the predicted pitch and the true pitch in the manner shown in Equation (7) and train the pitch prediction model; Among them, V pred is the predicted pitch vector, and V true is the true pitch vector, and N is the number of samples.

10. A voice cloning system integrating prosodic features for implementing the method according to claim 1, characterized in that: It includes a feature extraction module, a feature fusion module, and a model construction and training module; The feature extraction module is used to perform Mel spectrogram conversion and prosody feature extraction on the audio file, including pause feature marking, syllable-level speech rate calculation, and pitch sequence enhancement processing; the audio waveform and the transcribed text are used as the inputs of the module; The feature fusion module is used to perform dimensionality adjustment and classifier-free guided feature screening and fusion on the audio features and prosody features; the Mel spectrogram vector, the pause marker text, the speech rate feature vector, and the pitch feature vector are used as the inputs of the module; through time step embedding, random feature dropout, and multiple position encoding enhancement, it outputs a fused feature vector with temporal information; The model construction and training module is used to generate and optimize the target Mel spectrogram based on conditional flow matching; the fused feature vector and the Mel spectrogram are used as the inputs of the module; Through noise perturbation and flow field construction, UNet-Transformer multi-scale feature refinement, and composite loss function constraint, it outputs a high-quality synthesized Mel spectrogram; the generated Mel spectrogram is used as the input of the vocoder module.

Images