A personalized voice cloning method and system based on deep learning

Abstract

The invention discloses a personalized voice cloning method and system based on deep learning, which relates to the technical field of voice cloning, including: collecting voice samples of target users; extracting high-dimensional features of voice samples based on convolutional neural networks; encoding high-dimensional features into low-dimensional vectors based on variational autoencoders; using a preset synthesis model to convert the low-dimensional vector into a voice feature sequence; using the text and voice feature sequence corresponding to the voice sample as training samples, training the preset synthesis model to obtain a trained synthesis model; extracting the voice feature sequence of the text to be cloned based on the trained synthesis model; inputting the voice feature sequence of the text to be cloned into a speech synthesis model based on an improved WaveNet structure to generate a target voice waveform. The invention can achieve high-quality voice cloning based on a small amount of voice data.

Description

Technical Field

[0001] The present invention relates to the technical field of voice cloning, and in particular to a personalized voice cloning method and system based on deep learning. Background Art

[0002] Voice cloning technology has been widely used in smart assistants, dubbing, education and other fields. However, the existing voice cloning technology still has room for improvement in naturalness and personalization.

[0003] Specifically, current voice cloning technologies usually rely on a large amount of training data to achieve high-quality speech synthesis. However, collecting and annotating such a large amount of data is time-consuming, expensive, and impractical for many application scenarios. For personalized voice cloning, it is even more difficult to obtain a large number of target user voice samples.

[0004] Existing voice cloning technologies still face challenges in generating natural and fluent speech. In particular, the generated speech often lacks the naturalness of human speakers in terms of emotional expression and subtle changes in timbre. In addition, existing technologies are limited in retaining the target user's unique speaking style and timbre, resulting in insufficient personalization.

[0005] Many speech synthesis and cloning technologies perform poorly when encountering diverse speech environments (such as background noise) and different accents. This lack of adaptability restricts the application of speech cloning technology in a global, multilingual environment.

[0006] Existing technologies usually require high computing resources when processing complex speech generation tasks and are difficult to operate effectively in low-latency, real-time application scenarios. This computing performance bottleneck limits the popularity and application of voice cloning technology. Summary of the invention

[0007] The purpose of the present invention is to provide a personalized voice cloning method and system based on deep learning in order to solve at least one of the above technical problems.

[0008] In a first aspect, an embodiment of the present invention provides a personalized voice cloning method based on deep learning, comprising: collecting a voice sample of a target user; extracting high-dimensional features of the voice sample based on a convolutional neural network; encoding the high-dimensional features into a low-dimensional vector based on a variational autoencoder; converting the low-dimensional vector into a voice feature sequence using a preset synthesis model; training the preset synthesis model using the text corresponding to the voice sample and the voice feature sequence as training samples to obtain a trained synthesis model; extracting the voice feature sequence of the text to be cloned based on the trained synthesis model; inputting the voice feature sequence of the text to be cloned into a speech synthesis model based on an improved WaveNet structure to generate a target speech waveform.

[0009] Furthermore, the preset synthesis model is trained, including: training the preset synthesis model based on a few-sample learning and a self-supervised learning method.

[0010] Furthermore, based on a convolutional neural network, high-dimensional features of the speech sample are extracted, including: performing a short-time Fourier transform on the speech sample to generate a spectrogram; extracting Mel-frequency cepstral coefficients of the spectrogram to obtain a spectrum envelope; and extracting high-order features of the spectrum envelope based on the convolutional neural network to obtain high-dimensional features of the speech sample.

[0011] Furthermore, the preset synthesis model includes a synthesis model based on a Transformer structure.

[0012] Furthermore, the speech feature sequence of the text to be cloned is input into a speech synthesis model based on an improved WaveNet structure to generate a target speech waveform, including: inputting the speech feature sequence of the text to be cloned into the WaveNet structure for modulation, and generating an initial speech waveform through a conditional convolutional network; performing noise filtering and enhancement processing on the initial speech waveform to obtain the target speech waveform.

[0013] In a second aspect, an embodiment of the present invention further provides a personalized voice cloning system based on deep learning, comprising: a data acquisition module, a feature extraction module, a feature encoding module, a synthesis module, a training module and a speech synthesis module; wherein the data acquisition module is used to collect a speech sample of a target user; the feature extraction module is used to extract high-dimensional features of the speech sample based on a convolutional neural network; the feature encoding module is used to encode the high-dimensional features into a low-dimensional vector based on a variational autoencoder; the synthesis module is used to convert the low-dimensional vector into a speech feature sequence using a preset synthesis model; the training module is used to train the preset synthesis model using the text corresponding to the speech sample and the speech feature sequence as training samples to obtain a trained synthesis model; the synthesis module is also used to extract the speech feature sequence of the text to be cloned based on the trained synthesis model; the speech synthesis module is used to input the speech feature sequence of the text to be cloned into a speech synthesis model based on an improved WaveNet structure to generate a target speech waveform.

[0014] Furthermore, the training module is also used to train the preset synthesis model based on few-sample learning and self-supervised learning methods.

[0015] Furthermore, the feature extraction module is also used to: perform short-time Fourier transform on the speech sample to generate a spectrogram; extract Mel-frequency cepstral coefficients of the spectrogram to obtain a spectrum envelope; and extract high-order features of the spectrum envelope based on the convolutional neural network to obtain high-dimensional features of the speech sample.

[0016] In a third aspect, an embodiment of the present invention further provides an electronic device, comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the method described in the first aspect when executing the computer program.

[0017] In a fourth aspect, an embodiment of the present invention further provides a computer-readable storage medium, wherein the computer-readable storage medium stores computer instructions, and when the computer instructions are executed by a processor, the method described in the first aspect above is implemented.

[0018] The present invention provides a personalized voice cloning method and system based on deep learning, which solves the shortcomings of existing voice cloning technology and meets the diversified needs of the market by reducing data requirements, improving voice naturalness and personalization, enhancing environmental adaptability, and improving computing efficiency. It not only improves user experience, but also provides effective technical support for a wide range of application fields, and has significant market potential and social value. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] In order to more clearly illustrate the implementation methods of the present application or the technical solutions in the prior art, the drawings required for use in the specific implementation methods or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some implementation methods of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0020] Figure 1 A flowchart of a personalized voice cloning method based on deep learning provided by an embodiment of the present invention;

[0021] Figure 2 A schematic diagram of a personalized voice cloning system based on deep learning provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0022] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0023] Embodiment 1: Figure 1 is a flow chart of a personalized voice cloning method based on deep learning according to an embodiment of the present invention. Figure 1 As shown, the method specifically comprises the following steps:

[0024] Step S102: collecting a voice sample of the target user.

[0025] Preferably, the user's voice sample is collected through a smart device (such as a mobile phone, a computer), and then the collected voice sample is preprocessed, such as removing silence, cutting too long voice segments, and standardizing the volume.

[0026] In some optional implementations provided in the embodiments of the present invention, the recording format of the collected voice samples includes: PCM (pulse code modulation) or WAV, and the sampling rate is usually set to 16kHz or higher; the preprocessing tools used include: using Python libraries such as librosa or pydub for audio preprocessing.

[0027] Step S104, extracting high-dimensional features of the speech sample based on a convolutional neural network. Specifically, the steps include:

[0028] Step S1041, performing short-time Fourier transform on the speech sample to generate a spectrogram;

[0029] Step S1042, extracting the Mel-frequency cepstral coefficients of the spectrogram to obtain a spectrum envelope;

[0030] Step S1043, extracting high-order features of the spectrum envelope based on a convolutional neural network to obtain high-dimensional features of the speech sample.

[0031] Preferably, the librosa library is used for Short Time Fourier Transform (STFT) and Mel Frequency Cepstral Coefficient (MFCC) extraction.

[0032] Preferably, the convolutional neural network (CNNs) architecture adopts a three-layer convolutional network, each layer has a convolution kernel size of 3×3, uses a ReLU activation function, and a max pooling operation followed by batch normalization.

[0033] Step S106: Encode the high-dimensional features into low-dimensional vectors based on a variational autoencoder.

[0034] In an embodiment of the present invention, a variational autoencoder (VAE) is used to encode the extracted speech features into low-dimensional vectors. These vectors represent the core features of the speech and provide a compressed representation for the generation step. Specifically, the variational autoencoder includes an encoder, a reparameterization technique, and a decoder, wherein:

[0035] Encoder: Encodes high-dimensional feature input into a low-dimensional vector in the latent space;

[0036] Re-parameterization technique: The encoder output has a certain controllability and stability through re-parameterization technique;

[0037] Decoder: Reconstructs speech features from the latent space.

[0038] Preferably, in an embodiment of the present invention, the VAE architecture includes: the encoder and the decoder each use a two-layer fully connected network, and the latent space dimension is usually set to 64 or 128;

[0039] Activation function: ReLU for hidden layers, Sigmoid for output layers;

[0040] Reparameterization technique: Use Gaussian distribution for sampling. The reparameterization technique formula is: z=μ+σ*ε, where ε is standard normal distribution random noise.

[0041] Step S108, using a preset synthesis model, converting the low-dimensional vector into a speech feature sequence.

[0042] Preferably, the preset synthesis model includes a synthesis model based on a Transformer structure.

[0043] Specifically, in an embodiment of the present invention, the synthesis model adopts a Transformer-based architecture to convert the encoded features into a speech feature sequence. The model can capture complex patterns and dependencies in speech. The synthesis model specifically includes:

[0044] Input embedding layer: embeds low-dimensional vectors into high-dimensional representations;

[0045] Multi-head self-attention mechanism: captures long-range dependencies between speech features;

[0046] Position coding: adding time information through position coding;

[0047] Feedforward neural network: further processes the attention output to generate the final speech feature sequence.

[0048] Preferably, in an embodiment of the present invention, the Transformer architecture includes: based on the original Transformer design, including 6 encoding layers and 6 decoding layers, each layer has 8 attention heads;

[0049] Input embedding: Use a linear layer to embed and map low-dimensional features to a 512-dimensional representation;

[0050] Activation function: GELU (Gaussian Error Linear Unit) is used;

[0051] Optimizer: AdamW, learning rate is set to 1e-4.

[0052] Step S110, using the text and speech feature sequence corresponding to the speech sample as training samples, training a preset synthesis model to obtain a trained synthesis model.

[0053] Specifically, the preset synthesis model is trained based on few-sample learning and self-supervised learning methods.

[0054] Step S112: extracting the speech feature sequence of the text to be cloned based on the trained synthesis model.

[0055] Step S114, inputting the speech feature sequence of the text to be cloned into a speech synthesis model based on the improved WaveNet structure to generate a target speech waveform.

[0056] Specifically, step S114 also includes the following steps:

[0057] Step S1141, inputting the speech feature sequence of the text to be cloned into the WaveNet structure for modulation, and generating an initial speech waveform through a conditional convolutional network;

[0058] Step S1142, performing noise filtering and enhancement processing on the initial speech waveform to obtain a target speech waveform.

[0059] Specifically, the present invention implements the process of converting the generated speech features into a natural speech waveform, and adopts an improved WaveNet structure to generate a high-quality speech waveform. Specifically, the improved WaveNet structure includes:

[0060] Feature modulation: The generated speech features are input into WaveNet for modulation;

[0061] Waveform generation: The speech waveform is gradually generated through the conditional convolution network (Causal Convolution);

[0062] Post-processing: Noise filtering and enhancement of the generated speech.

[0063] Preferably, in an embodiment of the present invention, the WaveNet architecture: comprises 30 layers of residual blocks, each layer comprising 1-dimensional causal convolution and gated activation unit (GLU);

[0064] Sampling rate: 16kHz or 24kHz;

[0065] Post-processing tools: Noise filtering and speech enhancement using sox or custom filters.

[0066] In some optional implementations provided in the embodiments of the present invention, the improved WaveNet structure includes at least one of the following improvements to the WaveNet structure:

[0067] (1) Network structure optimization. Including:

[0068] Increase or decrease the number of convolutional layers to find the best configuration to balance model complexity and computational efficiency; adjust the number of convolution kernels in each layer so that the model can better capture the detailed features in the speech signal;

[0069] Introduce more residual connections to alleviate the gradient vanishing problem in deep network training; add skip connections to share information between different layers and improve the expressiveness of the model.

[0070] (2) Improvements in data preprocessing. Including:

[0071] Perform spectrum enhancement processing on the input data, such as Fourier transform and Mel spectrum processing, to extract richer frequency domain information;

[0072] Noise filtering and denoising are added in the data preprocessing stage to improve the purity of the speech signal and ensure the high quality of the model training data.

[0073] (3) Training strategy optimization. Including:

[0074] Use data augmentation techniques to increase the diversity of training data and prevent model overfitting. For example, by transforming speech signals in the time domain and frequency domain, more diverse training samples can be generated.

[0075] Use an adaptive learning rate optimizer (such as Adam or RMSprop) to improve training efficiency and model convergence speed.

[0076] (4) Generation process optimization. Including:

[0077] The parallel generation technology is introduced to reduce the time it takes for WaveNet to generate speech waveforms. The traditional WaveNet generates samples step by step, and the improved model can generate in parallel, improving generation efficiency.

[0078] The generated speech waveform is quantized and compressed to reduce storage and transmission costs while maintaining high-quality speech synthesis effects.

[0079] (5) Improvement of loss function. Including:

[0080] The perceptual loss function is introduced so that the model not only considers the waveform similarity of the synthesized speech, but also the perceptual quality of the speech, making the synthesized speech more natural and realistic.

[0081] Combine multiple loss functions (such as L1 loss, L2 loss, perceptual loss) to optimize various aspects of the model's performance during training.

[0082] Through the above improvements, the embodiments of the present invention can significantly improve the performance of the WaveNet-based speech synthesis model, making the generated speech more natural and fluent, and optimizing the computational efficiency.

[0083] From the above description, it can be seen that the embodiment of the present invention provides a personalized voice cloning method based on deep learning, which has the following technical effects compared with the prior art:

[0084] 1. Highly natural speech synthesis:

[0085] The present invention utilizes a multi-layer neural network structure, including convolutional neural networks (CNNs), variational autoencoders (VAEs), and a Transformer-based synthesis model to generate highly natural speech. Specific effects include:

[0086] Smooth speech generation: The synthesized speech is close to natural human speech in terms of speech rhythm and timbre changes, avoiding the mechanical feeling common in traditional speech synthesis.

[0087] Rich emotional expression: The model can capture subtle emotional changes in speech, such as ups and downs in tone and emphasis, making the generated speech more realistic in human expression.

[0088] 2. Powerful personalization capabilities:

[0089] Through feature extraction and encoding technology, the present invention can accurately retain the personalized characteristics of the target user, including timbre, intonation and speaking style. Specific effects include:

[0090] Preserve user voice: The generated voice can clearly reflect the unique voice of the target user, ensuring that the synthesized voice is highly similar to the user's original voice.

[0091] Imitate speaking style: The speaking speed, pauses, etc. can be consistent with the target user's speaking style, providing personalized voice output.

[0092] 3. Data efficiency is significantly improved:

[0093] The present invention uses few-sample learning and self-supervised learning technology to achieve high-quality voice cloning even with very little voice data. Specific effects include:

[0094] Efficient learning with small samples: Only a small amount of voice data (such as a few minutes of recording) is needed to train an effective voice cloning model, greatly reducing the cost and complexity of data collection.

[0095] Fast model adaptation: The model can quickly adapt to the speech characteristics of different users, reducing training time and data requirements.

[0096] 4. Strong environmental adaptability:

[0097] The system has enhanced its adaptability to different speech environments (such as noisy environments) and multiple accents through improved feature extraction and generation technology. Specific effects include:

[0098] Noise robustness: Ability to generate clear speech and maintain speech quality in an environment with background noise.

[0099] Multi-language support: The model structure can adapt to different languages ​​and accents and is widely applicable to global users.

[0100] 5. Real-time generation capability:

[0101] By optimizing the model structure and calculation process, the present invention demonstrates efficient computing capabilities in the speech generation process, making real-time speech synthesis possible. Specific effects include:

[0102] Low-latency generation: The latency of the speech synthesis process is extremely low, which can meet the needs of real-time applications such as online voice assistants and real-time translation.

[0103] High computing efficiency: It can maintain efficient speech generation performance even with low computing resources.

[0104] Embodiment 2: Figure 2 Schematic diagram of a personalized voice cloning system based on deep learning according to an embodiment of the present invention. Figure 2 As shown, the system includes: a data acquisition module 10, a feature extraction module 20, a feature encoding module 30, a synthesis module 40, a training module 50 and a speech synthesis module 60.

[0105] Specifically, the data collection module 10 is used to collect voice samples of the target user.

[0106] The feature extraction module 20 is used to extract high-dimensional features of the speech sample based on a convolutional neural network.

[0107] The feature encoding module 30 is used to encode high-dimensional features into low-dimensional vectors based on a variational autoencoder.

[0108] The synthesis module 40 is used to convert the low-dimensional vector into a speech feature sequence by using a preset synthesis model. Specifically, the preset synthesis model includes a synthesis model based on a Transformer structure.

[0109] The training module 50 is used to train a preset synthesis model using the text and speech feature sequence corresponding to the speech sample as training samples to obtain a trained synthesis model.

[0110] The synthesis module 40 is also used to extract the speech feature sequence of the text to be cloned based on the trained synthesis model.

[0111] The speech synthesis module 60 is used to input the speech feature sequence of the text to be cloned into the speech synthesis model based on the improved WaveNet structure to generate a target speech waveform.

[0112] Specifically, the training module 50 is also used to train the preset synthesis model based on few-sample learning and self-supervised learning methods.

[0113] Specifically, the feature extraction module 20 is also used for:

[0114] Perform short-time Fourier transform on the speech sample to generate a spectrogram;

[0115] Extract the Mel-frequency cepstral coefficients of the spectrogram to obtain the spectrum envelope;

[0116] Based on the convolutional neural network, the high-order features of the spectrum envelope are extracted to obtain the high-dimensional features of the speech samples.

[0117] Specifically, the speech synthesis module 60 is further used for:

[0118] The speech feature sequence of the text to be cloned is input into the WaveNet structure for modulation, and the initial speech waveform is generated through the conditional convolutional network;

[0119] The initial speech waveform is subjected to noise filtering and enhancement processing to obtain the target speech waveform.

[0120] Embodiment 3: The system architecture of the present invention includes five core modules: data acquisition module, feature extraction module, feature encoding module, synthesis model, and speech synthesis module. The specific implementation of each module is as follows:

[0121] 1. Data acquisition module:

[0122] The data collection module is responsible for collecting voice samples of target users. The specific implementation steps are as follows:

[0123] Voice sample collection: Users record a series of short voice samples through smart devices (such as mobile phones or computers) to ensure that the collected voice samples contain a variety of emotions and contexts.

[0124] Sample preprocessing: Preprocess the recorded speech samples, including removing silence, trimming long segments, normalizing volume, etc. You can use open source audio processing libraries such as librosa or pydub.

[0125] Implementation details:

[0126] The recommended recording format is WAV or FLAC, and the sampling rate is set to 16kHz or 24kHz.

[0127] Use librosa for audio cropping and de-silencing.

[0128] Sample code:

[0129] import librosaimport numpy as np

[0130] # Read audio file

[0131] y, sr = librosa.load('sample.wav', sr=16000)

[0132] # Remove mute processing

[0133] trimmed_audio, index = librosa.effects.trim(y)

[0134] 2. Feature extraction module:

[0135] The feature extraction module extracts high-dimensional features from speech data. These features are used to represent the timbre, pitch and rhythm of speech. The specific implementation steps are as follows:

[0136] Short-time Fourier transform (STFT): Converts speech signals into frequency domain representation and generates a spectrogram.

[0137] Mel-frequency cepstral coefficients (MFCC): Mel-frequency cepstral coefficients are extracted from the spectrogram to capture the spectral envelope of speech.

[0138] Convolutional Neural Networks (CNNs): CNNs are used to further extract high-level features.

[0139] Implementation details:

[0140] Compute STFT and MFCC using librosa.

[0141] The CNNs architecture uses a 3-layer convolutional network, each layer has a convolution kernel size of 3×3, uses a ReLU activation function, and a max pooling operation followed by batch normalization.

[0142] Python example code:

[0143] import librosaimport librosa.displayimport matplotlib.pyplot as plt

[0144] # Calculate STFT

[0145] D = librosa.stft(trimmed_audio)

[0146] # Calculate MFCC

[0147] mfcc = librosa.feature.mfcc(y=trimmed_audio, sr=sr, n_mfcc=13)

[0148] Example of CNNs architecture:

[0149] Python example code:

[0150] import torchimport torch.nn as nn

[0151] class CNNFeatureExtractor(nn.Module):

[0152] def __init__(self):

[0153] super(CNNFeatureExtractor, self).__init__()

[0154] self.conv1 = nn.Conv2d(1, 32, kernel_size=3, stride=1,padding=1)

[0155] self.conv2 = nn.Conv2d(32, 64, kernel_size=3, stride=1,padding=1)

[0156] self.conv3 = nn.Conv2d(64, 128, kernel_size=3, stride=1,padding=1)

[0157] self.pool = nn.MaxPool2d(2, 2)

[0158] self.relu = nn.ReLU()

[0159] self.bn1 = nn.BatchNorm2d(32)

[0160] self.bn2 = nn.BatchNorm2d(64)

[0161] self.bn3 = nn.BatchNorm2d(128)

[0162] def forward(self, x):

[0163] x = self.pool(self.relu(self.bn1(self.conv1(x))))

[0164] x = self.pool(self.relu(self.bn2(self.conv2(x))))

[0165] x = self.pool(self.relu(self.bn3(self.conv3(x))))

[0166] return x

[0167] 3. Feature Encoding Module:

[0168] The feature encoding module uses a variational autoencoder (VAE) to encode the extracted speech features into a low-dimensional vector. The specific implementation steps are as follows:

[0169] Encoder: Encodes high-dimensional feature input into a low-dimensional vector in the latent space.

[0170] Reparameterization technique: The encoder output is made controllable and stable through reparameterization technique.

[0171] Decoder: Reconstructs speech features from the latent space.

[0172] Implementation details:

[0173] The encoder and decoder each use a two-layer fully connected network, and the latent space dimension is set to 64 or 128.

[0174] Activation function: ReLU for hidden layers and Sigmoid for output layers.

[0175] Reparameterization technique formula: z = μ + σ * ε, where ε is the standard normally distributed random noise.

[0176] VAE architecture example:

[0177] Python example code:

[0178] class VAE(nn.Module):

[0179] def __init__(self, input_dim, hidden_dim, latent_dim):

[0180] super(VAE, self).__init__()

[0181] # Encoder

[0182] self.fc1 = nn.Linear(input_dim, hidden_dim)

[0183] self.fc21 = nn.Linear(hidden_dim, latent_dim)

[0184] self.fc22 = nn.Linear(hidden_dim, latent_dim)

[0185] # Decoder

[0186] self.fc3 = nn.Linear(latent_dim, hidden_dim)

[0187] self.fc4 = nn.Linear(hidden_dim, input_dim)

[0188] def encode(self, x):

[0189] h1 = torch.relu(self.fc1(x))

[0190] return self.fc21(h1), self.fc22(h1)

[0191] def reparameterize(self, mu, logvar):

[0192] std = torch.exp(0.5*logvar)

[0193] eps = torch.randn_like(std)

[0194] return mu + eps*std

[0195] def decode(self, z):

[0196] h3 = torch.relu(self.fc3(z))

[0197] return torch.sigmoid(self.fc4(h3))

[0198] def forward(self, x):

[0199] mu, logvar = self.encode(x)

[0200] z = self.reparameterize(mu, logvar)

[0201] return self.decode(z), mu, logvar

[0202] 4. Synthetic Model:

[0203] The synthesis model uses a Transformer-based architecture to convert the encoded features into a speech feature sequence. The specific implementation steps are as follows:

[0204] Input Embedding Layer: Embeds low-dimensional vectors into high-dimensional representations.

[0205] Multi-head self-attention mechanism: Capturing long-range dependencies between speech features.

[0206] Positional encoding: Adding time information through positional encoding.

[0207] Feedforward neural network: further processes the attention output to generate the final speech feature sequence.

[0208] Implementation details:

[0209] The Transformer architecture is based on the original design and consists of 6 encoding layers and 6 decoding layers, with 8 attention heads in each layer.

[0210] Input Embedding: Embedding is done using a linear layer to map low-dimensional features to a 512-dimensional representation.

[0211] Activation function: GELU (Gaussian Error Linear Unit) is used.

[0212] Optimizer: AdamW, learning rate is set to 1e-4.

[0213] Transformer model example:

[0214] Python example code:

[0215] from transformers import TransformerModel

[0216] class TransformerSynthesizer(nn.Module):

[0217] def __init__(self, input_dim, d_model, nhead, num_layers):

[0218] super(TransformerSynthesizer, self).__init__()

[0219] self.embedding = nn.Linear(input_dim, d_model)

[0220] self.transformer = TransformerModel(d_model, nhead, num_layers)

[0221] self.fc = nn.Linear(d_model, input_dim)

[0222] def forward(self, x):

[0223] x = self.embedding(x)

[0224] x = self.transformer(x)

[0225] return self.fc(x)

[0226] 5. Speech synthesis module:

[0227] The speech synthesis module converts the generated speech features into natural speech waveforms. It uses an improved WaveNet structure to generate high-quality speech waveforms. The specific implementation steps are as follows:

[0228] Feature modulation: The generated speech features are input into WaveNet for modulation.

[0229] Generating waveforms: Generate speech waveforms step by step through a conditional convolutional network.

[0230] Post-processing: Noise filtering and enhancement of the generated speech.

[0231] Implementation details:

[0232] The WaveNet architecture consists of 30 layers of residual blocks, each of which contains 1D causal convolutions and gated activation units (GLUs).

[0233] Sampling rate: 16kHz or 24kHz.

[0234] Post-processing tools: Noise filtering and speech enhancement using sox or custom filters.

[0235] WaveNet architecture example:

[0236] Python example code:

[0237] class WaveNet(nn.Module):

[0238] def __init__(self, input_dim, residual_channels, dilation_channels, skip_channels, num_layers):

[0239] super(WaveNet, self).__init__()

[0240] self.residual_channels = residual_channels

[0241] self.dilation_channels = dilation_channels

[0242] self.skip_channels = skip_channels

[0243] self.num_layers = num_layers

[0244] self.receptive_field = 2**num_layers

[0245] self.dilations = [2**i for i in range(num_layers)]

[0246] self.causal_convs = nn.ModuleList([

[0247] nn.Conv1d(input_dim, dilation_channels, kernel_size=2,dilation=d)

[0248] for d in self.dilations ])

[0250] self.residual_convs = nn.ModuleList([

[0251] nn.Conv1d(dilation_channels, residual_channels, kernel_size=1)

[0252] for _ in range(num_layers) ])

[0254] self.skip_convs = nn.ModuleList([

[0255] nn.Conv1d(dilation_channels, skip_channels, kernel_size=1)

[0256] for _ in range(num_layers) ])

[0258] self.final_conv = nn.Conv1d(skip_channels, 1, kernel_size=1)

[0259] def forward(self, x):

[0260] skip_connections = []

[0261] for i in range(self.num_layers):

[0262] y = torch.relu(self.causal_convs[i](x))

[0263] x = torch.relu(self.residual_convs[i](y)) + x[:, :, -y.size(2):]

[0264] skip_connections.append(self.skip_convs[i](y))

[0265] return self.final_conv(torch.sum(torch.stack(skip_connections), dim=0))

[0266] The specific operation process of the present invention comprises the following steps:

[0267] Voice sample collection: Users provide a small amount of voice samples and upload them to the system through smart devices.

[0268] Feature extraction: The feature extraction module processes the speech samples and extracts high-dimensional speech features.

[0269] Feature encoding: The feature encoding module encodes the extracted high-dimensional features into low-dimensional vectors.

[0270] Feature synthesis: The synthesis model converts the encoded features into a target speech feature sequence.

[0271] Speech Generation: The speech synthesis module converts the speech feature sequence into a natural speech waveform and performs post-processing.

[0272] Voice output: The final generated voice is played or downloaded through the system for users to use.

[0273] The implementation conditions of the system provided by the embodiment of the present invention include:

[0274] Hardware configuration: It is recommended to use a GPU-accelerated computing device (such as NVIDIA Tesla V100 or A100) to support efficient training and inference of deep learning models.

[0275] Software environment: Use PyTorch or TensorFlow framework for model development and librosa for audio processing.

[0276] Data requirements: Each user's voice data samples should include a variety of emotions and contexts, covering different tones such as normal conversations, questions, and exclamations.

[0277] An embodiment of the present invention further provides an electronic device, comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the method in the above-mentioned embodiment 1 when executing the computer program.

[0278] An embodiment of the present invention further provides a computer-readable storage medium, which stores computer instructions. When the computer instructions are executed by a processor, the method in the above-mentioned embodiment 1 is implemented.

[0279] It will be apparent to those skilled in the art that the invention is not limited to the details of the exemplary embodiments described above and that the invention can be implemented in other specific forms without departing from the spirit or essential features of the invention. Therefore, the embodiments should be considered exemplary and non-limiting in all respects, and the scope of the invention is defined by the appended claims rather than the foregoing description, and it is intended that all variations falling within the meaning and scope of the equivalent elements of the claims be included in the invention. Any reference numeral in a claim should not be considered as limiting the claim to which it relates.

[0280] In addition, it should be understood that although the present specification is described according to implementation modes, not every implementation mode contains only one independent technical solution. This description of the specification is only for the sake of clarity. Those skilled in the art should regard the specification as a whole. The technical solutions in each embodiment may also be appropriately combined to form other implementation modes that can be understood by those skilled in the art.

Claims

1. A personalized voice cloning method based on deep learning, characterized in that: include: Collect voice samples of target users; Based on a convolutional neural network, extracting high-dimensional features of the speech sample, wherein the high-dimensional features are used to represent the timbre, pitch and rhythm of the speech; The step of extracting high-dimensional features of the speech sample based on a convolutional neural network includes: Performing short-time Fourier transform on the speech sample to generate a spectrogram; Extracting Mel-frequency cepstral coefficients of the spectrogram to obtain a spectrum envelope; Extracting high-order features of the spectrum envelope based on the convolutional neural network to obtain high-dimensional features of the speech sample; Based on a variational autoencoder, the high-dimensional features are encoded into low-dimensional vectors; using a preset synthesis model, the low-dimensional vectors are converted into speech feature sequences; The synthesis model specifically includes: an input embedding layer for embedding low-dimensional vectors into high-dimensional representations; a multi-head self-attention mechanism for capturing long-distance dependencies between speech features; a positional encoding for adding time information through positional encoding; and a feedforward neural network for further processing the attention output to generate a final speech feature sequence. Using the text corresponding to the speech sample and the speech feature sequence as training samples, training the preset synthesis model to obtain a trained synthesis model; Extracting the speech feature sequence of the text to be cloned based on the trained synthesis model; The speech feature sequence of the text to be cloned is input into a speech synthesis model based on an improved WaveNet structure to generate a target speech waveform.

2. The method according to claim 1, characterized in that: The preset synthesis model is trained, including: training the preset synthesis model based on a few-sample learning and a self-supervised learning method.

3. The method according to claim 1, characterized in that: The preset synthesis model includes a synthesis model based on a Transformer structure.

4. The method according to claim 1, characterized in that: Inputting the speech feature sequence of the text to be cloned into a speech synthesis model based on an improved WaveNet structure to generate a target speech waveform, including: Inputting the speech feature sequence of the text to be cloned into the WaveNet structure for modulation, and generating an initial speech waveform through a conditional convolutional network; The initial speech waveform is subjected to noise filtering and enhancement processing to obtain the target speech waveform.

5. A personalized voice cloning system based on deep learning, used to implement the method according to any one of claims 1 to 4, characterized in that: include: Data collection module, feature extraction module, feature encoding module, synthesis module, training module and speech synthesis module; among them, The data collection module is used to collect voice samples of target users; The feature extraction module is used to extract high-dimensional features of the speech sample based on a convolutional neural network; The feature encoding module is used to encode the high-dimensional features into low-dimensional vectors based on a variational autoencoder; The synthesis module is used to convert the low-dimensional vector into a speech feature sequence using a preset synthesis model; The training module is used to train the preset synthesis model using the text corresponding to the speech sample and the speech feature sequence as training samples to obtain a trained synthesis model; The synthesis module is further used to extract the speech feature sequence of the text to be cloned based on the trained synthesis model; The speech synthesis module is used to input the speech feature sequence of the text to be cloned into a speech synthesis model based on an improved WaveNet structure to generate a target speech waveform.

6. The system according to claim 5, characterized in that: The training module is also used to train the preset synthesis model based on few-sample learning and self-supervised learning methods.

7. The system according to claim 5, characterized in that: The feature extraction module is further used for: Performing short-time Fourier transform on the speech sample to generate a spectrogram; Extracting Mel-frequency cepstral coefficients of the spectrogram to obtain a spectrum envelope; Based on the convolutional neural network, high-order features of the spectrum envelope are extracted to obtain high-dimensional features of the speech sample.

8. An electronic device, characterized in that: include: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the method according to any one of claims 1 to 4 when executing the computer program.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer instructions, and when the computer instructions are executed by a processor, the method according to any one of claims 1 to 4 is implemented.

Images