Voice cloning method, voice cloning system, storage medium and program product

Abstract

The invention discloses a voice cloning method, a voice cloning system, a storage medium and a program product, and relates to the field of artificial intelligence, and the method comprises the steps: collecting the original voice of a target speaker; performing preprocessing operation on the original voice to obtain optimized voice data; extracting voice feature data from the optimized voice data; training the voice feature data through a preset deep learning model to obtain a trained voice clone model; based on the voice clone model, generating clone voice according to the input text content; calculating the similarity between the cloned voice and the original voice; and when the similarity exceeds a preset similarity threshold, outputting the cloned voice. By implementing the method, the accuracy of voice cloning is improved.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence, and in particular to a voice cloning method, a voice cloning system, a storage medium and a program product. Background Art

[0002] With the rapid development of artificial intelligence, speech synthesis technology has been widely used in many fields, such as intelligent customer service, audio books, voice navigation, etc. Among them, voice cloning, as an important branch of speech synthesis, can imitate the voice characteristics of a specific speaker and has important value in personalized voice services.

[0003] At present, voice cloning technology mainly generates voice through an acoustic rule library, which contains voice units such as tone, intonation, and speaking speed. In the voice generation stage, the voice cloning system selects corresponding voice units from the acoustic rule library based on the text, combines them, and adjusts them according to the preset voice rules to finally generate voice.

[0004] Since the acoustic rule base matching method mainly relies on preset speech rules, the generated speech often has an obvious mechanical feel. When different emotions or tones need to be expressed, it is difficult to achieve natural speech performance. Summary of the invention

[0005] The present application provides a voice cloning method, a voice cloning system, a storage medium and a program product, which are used to improve the accuracy of voice cloning.

[0006] In a first aspect, the present application provides a voice cloning method, which is applied to a voice cloning system, the method comprising: collecting an original voice of a target speaker; performing a preprocessing operation on the original voice to obtain optimized voice data, the preprocessing operation comprising a noise reduction process, an audio quality standardization process, and a sentence segmentation process; extracting voice feature data from the optimized voice data, the voice feature data comprising a pitch feature, a timbre feature, a duration feature, and an intensity feature; training the voice feature data through a preset deep learning model to obtain a trained voice cloning model, the voice cloning model being used to clone the voice of the target speaker; generating a cloned voice based on the voice cloning model and input text content; calculating the similarity between the cloned voice and the original voice; and outputting the cloned voice when the similarity exceeds a preset similarity threshold.

[0007] By adopting the above technical solution, the voice cloning system uses deep learning technology to obtain a personalized voice cloning model, so that a cloned voice with a voice effect close to that of the target speaker can be synthesized based on any text content. This method does not rely on the traditional acoustic rule library, but extracts multiple voice features from the original language of the target speaker for model training to capture the personalized characteristics of the target speaker's voice, improve the anthropomorphism and realism of the cloned voice, effectively alleviate the problem of unnatural voice synthesis caused by using preset voice rules, and achieve personalized and realistic voice cloning effects.

[0008] In combination with some embodiments of the first aspect, in some embodiments, the original speech is preprocessed to obtain optimized speech data, and the preprocessing operation includes noise reduction, audio quality standardization and sentence segmentation, specifically including: performing noise reduction on the original speech signal to obtain noise-reduced speech data; unifying the audio parameters of the noise-reduced speech data to obtain standardized speech data; based on a preset pause duration threshold, dividing the standardized speech data into multiple speech segments; deleting speech segments whose duration is less than the preset duration threshold to obtain the optimized speech data.

[0009] By adopting the above technical solutions, the voice cloning system can effectively remove the noise components in the original voice, eliminate the difference in audio quality, obtain clear standardized voice data, and eliminate invalid short-term voices. The optimized voice data after preprocessing is of higher quality, which can provide a high-quality input source for subsequent feature extraction and model training, and improve the effect of voice cloning.

[0010] In combination with some embodiments of the first aspect, in some embodiments, speech feature data is extracted from the optimized speech data, and the speech feature data includes pitch feature, timbre feature, duration feature and intensity feature, specifically including: performing fundamental frequency detection on the optimized speech data to obtain the pitch feature; extracting the timbre feature in the optimized speech data through spectrum analysis; determining the duration feature in the optimized speech data based on the duration of the phoneme and the speaking rate; performing energy analysis on the optimized speech data to obtain the intensity feature; and constructing the speech feature data based on the pitch feature, the timbre feature, the duration feature and the intensity feature.

[0011] By adopting the above technical solutions, the voice cloning system performs fundamental frequency detection, spectrum analysis, duration calculation and energy analysis to obtain the pitch characteristics, timbre characteristics, duration characteristics and intensity characteristics of the voice respectively. These four types of features can fully describe the characteristics of the voice. The pitch characteristics reflect the pitch of the voice, the timbre characteristics reflect the pronunciation structure, the duration characteristics reflect the rhythm of the voice, and the intensity characteristics reflect the strength of the voice. The voice cloning system constructs voice feature data based on these feature data, which can accurately characterize the voice style and personality of the target speaker. Using these rich voice feature data as input during model training can capture the unique voice pattern of the target speaker, greatly improve the realism of voice cloning, and achieve personalized and accurate speech synthesis.

[0012] In combination with some embodiments of the first aspect, in some embodiments, the speech feature data is trained by a preset deep learning model to obtain a trained speech cloning model, and the speech cloning model is used to clone the speech of the target speaker. The preset deep learning model includes an encoder and a decoder, and specifically includes: dividing the speech feature data into a training data set and a verification data set; inputting the training data set into the encoder to obtain hidden features of the target speaker; fusing the hidden features with text feature data corresponding to the speech feature data to obtain comprehensive features; inputting the comprehensive features into the decoder to generate speech prediction features; calculating a loss function based on the speech prediction features and the speech feature data in the verification data set; and obtaining the speech cloning model when the loss function is less than a preset loss function threshold.

[0013] By adopting the above technical solution, the voice cloning system uses a preset deep learning model to train voice features and obtain a voice cloning model. This deep learning-based model structure can establish a mapping relationship between the voice features of the target speaker and the text semantics, achieve voice personalization, do not rely on subjective experience, and are more reliable and effective.

[0014] In combination with some embodiments of the first aspect, in some embodiments, the hidden layer features are fused with the text feature data corresponding to the speech feature data to obtain a comprehensive feature, which specifically includes: converting the text feature data into a word vector sequence; calculating the correlation weight between the hidden layer features and the word vector sequence; based on the correlation weight, performing a weighted combination of the hidden layer features and the word vector sequence to obtain a feature combination vector; and performing a dimensional transformation on the feature combination vector to obtain the comprehensive feature.

[0015] By adopting the above technical solution, the voice cloning system can capture the intrinsic relationship between text semantic information and voice style information, and organically combine the two to form the input of voice decoding. The text word vector reflects the semantic content, and the voice hidden features reflect the personalized style. The fusion of the two as the decoder input can make the output voice contain the style characteristics of the target speaker while retaining the text semantics, thereby generating a cloned voice with both semantic accuracy and voice personalization.

[0016] In combination with some embodiments of the first aspect, in some embodiments, after the step of outputting the cloned voice when the similarity exceeds a preset similarity threshold, the method further includes: performing sentiment analysis on the text content to obtain sentiment feature parameters; performing sentiment adjustment on the cloned voice based on the sentiment feature parameters to obtain an optimized cloned voice; calculating the degree of match between the optimized cloned voice and a preset sentiment voice template; and when the degree of match is greater than a preset degree of match threshold, using the optimized cloned voice as the final output voice.

[0017] By adopting the above technical solution, the voice cloning system optimizes the emotional expression of the cloned voice by analyzing the emotional characteristic parameters of the text content, adjusting the intonation curve, speech speed and pauses of the cloned voice, and outputting a cloned voice that meets the expected emotion. This method can make the cloned voice achieve better emotional expression, rather than just imitating the voice characteristics. Different text content can correspond to different emotions, and the cloned voice can also change to different tones, improving the richness, diversity and authenticity of the voice, greatly expanding the application scope of voice cloning technology.

[0018] In combination with some embodiments of the first aspect, in some embodiments, the emotion adjustment of the cloned voice based on the emotion feature parameters to obtain the optimized cloned voice specifically includes: determining the target emotion model according to a preset emotion model library and the emotion feature parameters; based on the target emotion model, adjusting the pitch curve, speaking speed, pause duration and volume intensity of the cloned voice to obtain the optimized cloned voice.

[0019] By adopting the above technical solution, the voice cloning system combines the preset emotion model library and finds the matching target emotion model according to the emotion characteristics of the text content, so as to adjust the voice parameters and optimize the emotion expression. This method can quickly determine the direction and basis of emotion adjustment, effectively guide the generation of emotional speech, avoid distortion caused by blind adjustment, enhance the emotion expression ability of the voice cloning system, and make the cloned speech rich in emotion while retaining the speech characteristics of the target speaker.

[0020] In a second aspect, an embodiment of the present application provides a voice cloning system, the voice cloning system comprising: one or more processors and a memory; the memory is coupled to the one or more processors, the memory is used to store computer program code, the computer program code includes computer instructions, the one or more processors call the computer instructions to enable the voice cloning system to perform the method described in the first aspect and any possible implementation manner of the first aspect.

[0021] In a third aspect, an embodiment of the present application provides a computer program product comprising instructions. When the computer program product is run on a voice cloning system, the voice cloning system performs the method described in the first aspect and any possible implementation manner of the first aspect.

[0022] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, comprising instructions. When the instructions are executed on a voice cloning system, the voice cloning system performs the method described in the first aspect and any possible implementation manner of the first aspect.

[0023] It can be understood that the voice cloning system provided in the second aspect, the computer program product provided in the third aspect, and the computer storage medium provided in the fourth aspect are all used to execute the method provided in the embodiment of the present application. Therefore, the beneficial effects that can be achieved can refer to the beneficial effects in the corresponding method, which will not be repeated here.

[0024] One or more technical solutions provided in the embodiments of the present application have at least the following technical effects or advantages: 1. By adopting the above technical solution, the voice cloning system uses deep learning technology to obtain a personalized voice cloning model, so that it can synthesize a cloned voice with a voice effect close to that of the target speaker based on any text content. This method does not rely on the traditional acoustic rule library, but extracts multiple voice features from the original language of the target speaker for model training to capture the personalized characteristics of the target speaker's voice, improve the anthropomorphism and realism of the cloned voice, effectively alleviate the problem of unnatural voice synthesis caused by using preset voice rules, and achieve personalized and realistic voice cloning effects.

[0025] 2. By adopting the above technical solutions, the voice cloning system performs fundamental frequency detection, spectrum analysis, duration calculation and energy analysis to obtain the pitch characteristics, timbre characteristics, duration characteristics and intensity characteristics of the voice respectively. These four types of features can fully describe the characteristics of the voice. The pitch characteristics reflect the pitch of the voice, the timbre characteristics reflect the pronunciation structure, the duration characteristics reflect the rhythm of the voice, and the intensity characteristics reflect the strength of the voice. The voice cloning system constructs voice feature data based on these feature data, which can accurately characterize the voice style and personality of the target speaker. Using these rich voice feature data as input during model training can capture the unique voice pattern of the target speaker, greatly improve the realism of voice cloning, and achieve personalized and accurate speech synthesis.

[0026] 3. By adopting the above technical solution, the voice cloning system can capture the intrinsic relationship between text semantic information and voice style information, and organically combine the two to form the input of voice decoding. The text word vector reflects the semantic content, and the voice hidden features reflect the personalized style. The fusion of the two as the decoder input can make the output voice contain the style characteristics of the target speaker on the basis of retaining the text semantics, thereby generating a cloned voice with both semantic accuracy and voice personalization. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] Figure 1 It is a flow chart of the voice cloning method in the embodiment of the present application; Figure 2 is another flow chart of the voice cloning method in the embodiment of the present application; Figure 3 It is a schematic diagram of a physical device structure of the voice cloning system in the embodiment of the present application. DETAILED DESCRIPTION

[0028] The terms used in the following embodiments of the present application are only for the purpose of describing specific embodiments, and are not intended to be used as limitations to the present application. As used in the specification of the present application, the singular expressions "one", "a kind of", "above", "the" and "this" are intended to also include plural expressions, unless there is a clear indication to the contrary in the context. It should also be understood that the term "and / or" used in the present application refers to any or all possible combinations comprising one or more of the listed items.

[0029] In the following, the terms "first" and "second" are used for descriptive purposes only and are not to be understood as suggesting or implying relative importance or implicitly indicating the number of the indicated technical features. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of the features, and in the description of the embodiments of the present application, unless otherwise specified, "plurality" means two or more.

[0030] The following is a description of the process of the method provided by this implementation. Figure 1 , which is a flow chart of the voice cloning method in an embodiment of the present application.

[0031] S101, collecting the original speech of the target speaker; Among them, the target speaker refers to the specific person whose voice needs to be cloned; the original voice refers to the real voice signal emitted by the target speaker without any processing; and acquisition refers to the process of obtaining the original voice of the target speaker through a recording device.

[0032] Specifically, the voice cloning system uses high-quality microphones and other recording equipment to record the target speaker's voice reading a preset text in a quiet environment. During the recording process, it is necessary to ensure that the sampling rate is not less than 16kHz, the bit depth is not less than 16bit, and the recording time is usually more than 30 minutes to obtain sufficient voice samples. The recording content should contain rich voice materials, covering different tones, intonations and emotional expressions.

[0033] S102, performing a preprocessing operation on the original speech to obtain optimized speech data, wherein the preprocessing operation includes noise reduction processing, audio quality standardization processing and sentence segmentation processing; Among them, the preprocessing operation refers to the process of preliminary processing of the original speech; optimizing speech data refers to speech data with higher quality after preprocessing; noise reduction processing refers to removing background noise from the original speech; audio quality standardization processing refers to unifying audio parameters; sentence segmentation processing refers to dividing continuous speech into independent sentence segments.

[0034] Specifically, first, the voice cloning system uses a noise reduction algorithm to filter out interference signals such as environmental noise and current noise in the original voice; then, the voice cloning system performs audio quality standardization on the original voice after noise reduction, adjusts the sampling rate to 16kHz, and standardizes the volume to -23LUFS; then, based on acoustic features and pause duration (usually 200ms is used as the preset pause duration threshold), the voice cloning system divides the original voice after audio quality standardization into multiple independent sentence segments; finally, the voice cloning system deletes voice segments that are too short (such as less than 500ms) or of unqualified quality to obtain clear, standardized and optimized voice data.

[0035] Optionally, in general, the original speech is preprocessed to obtain optimized speech data. The preprocessing operation includes noise reduction, audio quality standardization and sentence segmentation, which can be achieved in the following ways, which are not limited here: noise reduction is performed on the original speech signal to obtain noise-reduced speech data; audio parameters of the noise-reduced speech data are unified to obtain standardized speech data; based on a preset pause duration threshold, the standardized speech data is divided into multiple speech segments; and speech segments with a duration less than the preset duration threshold are deleted to obtain the optimized speech data.

[0036] S103, extracting voice feature data from the optimized voice data, the voice feature data including pitch feature, timbre feature, duration feature and intensity feature; Among them, speech feature data is a numerical parameter used to describe the characteristics of speech; pitch feature refers to the fundamental frequency change characteristics of speech; timbre feature refers to the spectral envelope characteristics of speech; duration feature refers to the duration characteristics of speech; intensity feature refers to the energy distribution characteristics of speech.

[0037] Specifically, first, the voice cloning system extracts the fundamental frequency curve of the optimized voice data through the YIN algorithm or the WORLD vocoder to obtain the pitch feature; then, the voice cloning system uses the Mel-frequency cepstral coefficient (MFCC) or linear prediction coefficient (LPC) to analyze the spectrum envelope of the optimized voice data and extract the timbre features that reflect the vocal tract characteristics; then, the voice cloning system calculates the duration and overall speech rate of each phoneme to obtain the duration feature; finally, the voice cloning system obtains the intensity feature of the optimized voice data through short-time energy analysis. All features are converted into a numerical vector format that can be used for deep learning model training to form complete voice feature data.

[0038] Optionally, in general, voice feature data is extracted from the optimized voice data, and the voice feature data includes pitch feature, timbre feature, duration feature and intensity feature, which can be achieved in the following ways, which are not limited here: performing fundamental frequency detection on the optimized voice data to obtain the pitch feature; extracting the timbre feature in the optimized voice data through spectrum analysis; determining the duration feature in the optimized voice data based on the duration of the phoneme and the speaking rate; performing energy analysis on the optimized voice data to obtain the intensity feature; and constructing the voice feature data based on the pitch feature, the timbre feature, the duration feature and the intensity feature.

[0039] Among them, pitch features are used to represent the fundamental frequency and pitch changes of speech; timbre features are used to represent the harmonic structure and sound quality characteristics of speech; duration features are used to represent the duration and speaking speed of phoneme pronunciation; intensity features are used to represent the loudness and energy distribution of speech; fundamental frequency detection refers to the process of extracting the basic frequency of speech through signal processing methods; spectrum analysis refers to the process of decomposing and analyzing speech in the frequency domain; phoneme refers to the smallest segment unit in speech; energy analysis refers to the process of quantifying speech intensity and energy changes.

[0040] Specifically, first, the voice cloning system extracts multi-dimensional features from the input optimized voice data: fundamental frequency detection is performed through methods such as short-time Fourier transform to obtain the fundamental frequency curve reflecting the pitch and intonation of the target speaker; spectral feature extraction methods such as Mel-frequency cepstral coefficients (MFCC) are used to analyze the spectral envelope of the voice to obtain harmonic structure information that reflects the timbre characteristics; based on the phoneme alignment results of speech recognition, the duration and overall speaking speed of each phoneme are calculated to obtain the duration feature; energy analysis is performed by calculating parameters such as short-time energy and zero-crossing rate to obtain the loudness and energy contour features of the voice. Finally, the voice cloning system integrates and normalizes these data to construct complete voice feature data, providing a basis for subsequent acoustic modeling.

[0041] S104, training the speech feature data by using a preset deep learning model to obtain a trained speech cloning model, where the speech cloning model is used to clone the speech of the target speaker; Among them, the preset deep learning model represents a pre-designed neural network structure; the encoder refers to a network module that converts input features into hidden layer representations; the decoder refers to a network module that reconstructs hidden layer features into target outputs; the training data set represents data samples used for model training; the validation data set refers to independent data samples used to evaluate model performance; the hidden layer features represent the high-dimensional feature representations extracted by the encoder; the loss function refers to a mathematical function that measures the model prediction error; and the voice cloning model represents a trained model that can be used to generate cloned voices.

[0042] Specifically, the voice cloning system first divides the voice feature data into a training set and a validation set in a ratio of 8:2; then the training data is input into an encoder (usually using a multi-layer CNN or Transformer structure) to extract the voice feature encoding of the target speaker; the extracted features are then fused with the corresponding text features (usually word vectors) through an attention mechanism; the fused features are passed through a decoder (usually using an autoregressive structure) to generate voice prediction features; the system calculates the mean square error and other loss indicators between the predicted features and the true features; when the loss value is lower than a preset threshold (such as 0.01) or the number of training rounds reaches the upper limit, the final voice cloning model is obtained.

[0043] Optionally, in general, the speech feature data is trained by a preset deep learning model to obtain a trained speech cloning model, which is used to clone the speech of the target speaker. The preset deep learning model includes an encoder and a decoder, which can be implemented in the following manner, which is not limited here: dividing the speech feature data into a training data set and a verification data set; inputting the training data set into the encoder to obtain hidden features of the target speaker; fusing the hidden features with text feature data corresponding to the speech feature data to obtain comprehensive features; inputting the comprehensive features into the decoder to generate speech prediction features; calculating a loss function based on the speech prediction features and the speech feature data in the verification data set; and obtaining the speech cloning model when the loss function is less than a preset loss function threshold.

[0044] Among them, the preset deep learning model is a pre-designed neural network structure for learning and modeling speech features; the training data set refers to a set of speech feature samples used for model training; the validation data set represents a set of test sample speech feature samples used to evaluate model performance; the encoder refers to a neural network module that converts input features into potential representations; the decoder refers to a neural network module that converts potential representations back into target domain features; the hidden layer feature refers to the abstract feature representation extracted by the encoder; the text feature data refers to the linguistic features of the text corresponding to the speech; the comprehensive feature refers to the feature representation that integrates acoustic and linguistic information; the speech prediction feature represents the target speech feature generated by the model; the loss function is used to represent the error measure between the model prediction results and the real data; the preset loss function threshold refers to the error limit value used to determine whether the model training has converged.

[0045] Specifically, first, the voice cloning system randomly divides the extracted voice feature data into a training data set and a validation data set in a ratio of 8:2, while ensuring the balance of data distribution. During the training process, the encoder (usually using a multi-layer convolutional neural network or Transformer structure) receives the training data set, extracts the voiceprint features and acoustic features of the target speaker through multi-layer nonlinear transformations, and forms a low-dimensional hidden layer representation. The voice cloning system also extracts the corresponding linguistic features such as phoneme sequences, tones, and stresses through the text analysis module, and fuses these linguistic features with the hidden layer features in the feature space under the guidance of the attention mechanism to generate comprehensive features containing acoustic and linguistic information. The decoder (usually using a neural network based on an autoregressive structure) receives these comprehensive features and reconstructs the target voice features by generating them frame by frame. In each round of training iteration, the voice cloning system uses a weighted combination of multiple loss terms such as mean square error and KL divergence to calculate the difference between the model prediction results and the real voice features in the validation data set. When the loss function of multiple consecutive iterations is less than the preset loss function threshold (such as 0.01), the model is considered to have converged and the training process is completed. The entire training process usually uses the Adam optimizer, and uses techniques such as learning rate decay and gradient clipping to ensure training stability.

[0046] Optionally, in general, the hidden layer feature is fused with the text feature data corresponding to the speech feature data to obtain a comprehensive feature, which can be achieved in the following way, which is not limited here: converting the text feature data into a word vector sequence; calculating the correlation weight between the hidden layer feature and the word vector sequence; based on the correlation weight, performing a weighted combination of the hidden layer feature and the word vector sequence to obtain a feature combination vector; and performing a dimensional transformation on the feature combination vector to obtain the comprehensive feature.

[0047] Among them, the word vector sequence refers to the sequence of words in the text converted into continuous vector representations; the correlation weight refers to the numerical value that measures the degree of correlation between acoustic features and language features; the feature combination vector is used to represent the intermediate representation that combines acoustic and language features.

[0048] Specifically, first, the voice cloning system uses a pre-trained word embedding model (such as Word2Vec or BERT) to convert each word in the text feature data into a word vector of a fixed dimension (such as 256 dimensions) to form a word vector sequence. Then, the voice cloning system calculates the correlation between the hidden features (acoustic features) and the word vector sequence through a multi-head attention mechanism. Each attention head independently calculates the similarity score of the query-key-value pair and performs softmax normalization to obtain the attention weight. Based on the obtained attention weight, the voice cloning system performs a weighted sum operation on the hidden features and the word vector sequence to achieve adaptive fusion of acoustic information and language information and generate a feature combination vector. Finally, the voice cloning system uses a multi-layer perceptron or convolutional neural network to perform nonlinear transformation and dimensionality adjustment on the feature combination vector (usually including dimensionality reduction operations, such as compressing 512 dimensions to 256 dimensions) to ensure that the output comprehensive features have dimensions and structures suitable for decoder processing.

[0049] S105, generating a cloned voice based on the voice cloning model and the input text content; Among them, text content refers to text information that needs to be converted into speech; cloned speech refers to the synthetic speech generated by the speech cloning model that imitates the target speaker.

[0050] Specifically, first, the voice cloning system performs natural language processing such as word segmentation and part-of-speech tagging on the input text content to extract the linguistic features of the text. Then, the voice cloning system inputs the linguistic features of the text into the encoder of the language cloning model to obtain the latent representation of the text. Next, the voice cloning system adaptively fuses the latent representation of the text with the acoustic features of the target speaker. Finally, the decoder converts the fused features into speech waveform parameters, which are synthesized into the final cloned speech through the vocoder.

[0051] S106, calculating the similarity between the cloned voice and the original voice; Among them, similarity refers to the degree of closeness between the acoustic features of the cloned speech and the original speech; acoustic features refer to various parameters that describe the acoustic properties of speech.

[0052] Specifically, the voice cloning system uses a multi-dimensional evaluation method to calculate the similarity: first, the acoustic feature vectors of the cloned voice and the original voice are extracted, including the fundamental frequency contour, spectral envelope, energy distribution, etc.; then, the cosine similarity, Euclidean distance and other metrics between these feature vectors are calculated; at the same time, the pre-trained voiceprint recognition model is used to calculate the voiceprint similarity of the two voice segments; finally, the weighted average of each indicator is taken to obtain the comprehensive similarity.

[0053] S107: When the similarity exceeds a preset similarity threshold, output the cloned voice.

[0054] The preset similarity threshold is a standard value for determining the quality of cloned voices; and output refers to the process of saving or playing qualified cloned voices.

[0055] Specifically, the voice cloning system compares the calculated similarity with a preset similarity threshold (usually set to 0.85). If the similarity exceeds the preset similarity threshold, it means that the cloned voice has reached an acceptable quality standard, and the voice cloning system saves the cloned voice in WAV format with a 16kHz sampling rate and 16bit bit depth. If the similarity does not exceed the preset similarity threshold, then the model training link is returned for optimization, or the generation parameters are adjusted to regenerate the cloned voice until a satisfactory result is obtained.

[0056] By adopting the above technical solution, the voice cloning system uses deep learning technology to obtain a personalized voice cloning model, so that a cloned voice with a voice effect close to that of the target speaker can be synthesized based on any text content. This method does not rely on the traditional acoustic rule library, but extracts multiple voice features from the original language of the target speaker for model training to capture the personalized characteristics of the target speaker's voice, improve the anthropomorphism and realism of the cloned voice, effectively alleviate the problem of unnatural voice synthesis caused by using preset voice rules, and achieve personalized and realistic voice cloning effects.

[0057] The following is a more detailed description of the process of the method provided by this implementation. Figure 2 , is another flow chart of the voice cloning method in the embodiment of the present application.

[0058] After step S107, the following steps may or may not be performed, which is not limited here: S201, performing sentiment analysis on the text content to obtain sentiment feature parameters; Among them, sentiment analysis refers to the process of identifying and quantifying the emotional tendencies contained in text content; sentiment feature parameters refer to numerical indicators that describe the emotional attributes of text content; and sentiment tendencies refer to the emotional colors expressed by the text content, such as joy, anger, sadness, etc.

[0059] Specifically, first, the voice cloning system uses natural language processing technology to segment and tag the text content; then, the voice cloning system identifies emotional vocabulary and modal particles in the text content based on the emotional dictionary and deep learning model; then, the voice cloning calculates the emotional polarity (positive, negative, neutral), emotional category (such as joy level 0.8, anger level 0.2), and emotional intensity (a numerical value between 0-1) of the text, where emotional intensity refers to the degree of emotional expression; finally, the voice cloning system combines these parameters into a multidimensional vector and outputs it as emotional feature parameters.

[0060] S202, performing emotion adjustment on the cloned voice based on the emotion feature parameter to obtain an optimized cloned voice; Among them, emotion adjustment refers to adjusting the acoustic parameters of the cloned voice according to the emotion characteristic parameters; optimized cloned voice refers to the voice after emotion adjustment.

[0061] Specifically, first, the voice cloning system selects the target emotion model that best matches the current emotion feature parameters from the preset emotion model library. The preset emotion model library contains acoustic parameter patterns of multiple typical emotions. The target emotion model refers to a pre-trained model used to guide the adjustment of voice emotion. Then, the voice cloning system adjusts the acoustic parameters of the cloned voice according to the specific values ​​of the emotion feature parameters: for example, joy will increase the pitch (an average increase of 20-50Hz), speed up the speech rate (increase by 10-30%), shorten the pauses (reduce by 20-40%), and increase the volume fluctuation range. Finally, the voice cloning system re-synthesizes the adjusted voice through the vocoder to generate an optimized cloned voice with the target emotion characteristics.

[0062] Optionally, in general, emotion adjustment is performed on the cloned voice based on the emotion feature parameters to obtain an optimized cloned voice, which can be achieved in the following manner, which is not limited here: determining a target emotion model based on a preset emotion model library and the emotion feature parameters; based on the target emotion model, adjusting the pitch curve, speech speed, pause duration and volume intensity of the cloned voice to obtain the optimized cloned voice.

[0063] Among them, the pitch curve is used to indicate the change of the fundamental frequency of speech over time; the speaking rate is used to indicate the pronunciation speed of speech; the pause duration is used to indicate the duration of the pause between sentences; and the volume intensity is used to indicate the change of the loudness of speech.

[0064] S203, calculating the matching degree between the optimized cloned voice and the preset emotional voice template; The preset emotional speech template refers to a pre-set standard emotional speech sample library; the matching degree refers to the consistency between the optimized cloned speech and the preset emotional speech template in emotional expression.

[0065] Specifically, first, the voice cloning system selects standard voice samples corresponding to the emotional category from the preset emotional voice template library; then, the voice cloning system extracts the acoustic feature vectors of the optimized cloned voice and the standard language sample respectively, including emotion-related features such as pitch contour, energy envelope, and spectral tilt; then, the voice cloning system calculates the similarity of the two sets of acoustic feature vectors in various dimensions, including Pearson correlation coefficient, dynamic time warping distance, etc.; finally, the voice cloning system weighted averages the similarities of each dimension to obtain the overall matching degree.

[0066] S204: When the matching degree is greater than a preset matching degree threshold, the optimized cloned voice is used as the final output voice.

[0067] Among them, the preset matching threshold is the standard value for determining the quality of emotional expression; the final output voice refers to the cloned voice that has been emotion-optimized and of qualified quality.

[0068] Specifically, the voice cloning system compares the calculated matching degree with the preset matching degree threshold (usually set to 0.8). If the matching degree is greater than the preset matching degree threshold, it indicates that the emotional expression of the optimized cloned voice has achieved the expected effect. Then the voice cloning system uses the optimized cloned voice as the final output result and saves it as a high-quality audio file (24bit / 48kHz WAV format). If the matching degree does not reach the preset matching degree threshold, the voice cloning system returns to the emotional adjustment link and optimizes by fine-tuning the acoustic parameters until a satisfactory emotional expression effect is obtained.

[0069] The following describes the voice cloning system in the embodiment of the present invention from the perspective of hardware processing. Figure 3 , is a schematic diagram of a physical device structure of a voice cloning system in an embodiment of the present application.

[0070] It should be noted that Figure 3 The structure of the voice cloning system shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present invention.

[0071] like Figure 3 As shown, the voice cloning system includes a central processing unit (CPU) 301, which can perform various appropriate actions and processes according to the program stored in the read-only memory (ROM) 302 or the program loaded from the storage part 308 to the random access memory (RAM) 303, such as executing the method described in the above embodiment. In the RAM 303, various programs and data required for system operation are also stored. The CPU 301, the ROM 302 and the RAM 303 are connected to each other through the bus 304. The input / output (I / O) interface 305 is also connected to the bus 304.

[0072] The following components are connected to the I / O interface 305: an input section 306 including an audio input device, a button switch, etc.; an output section 307 including a liquid crystal display (LCD) and an audio output device, an indicator light, etc.; a storage section 308 including a hard disk, etc.; and a communication section 309 including a network interface card such as a LAN (Local Area Network) card, a modem, etc. The communication section 309 performs communication processing via a network such as the Internet. A drive 310 is also connected to the I / O interface 305 as needed. A removable medium 311, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 310 as needed so that a computer program read therefrom is installed into the storage section 308 as needed.

[0073] In particular, according to an embodiment of the present invention, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present invention includes a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes a computer program for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network through the communication part 309, and / or installed from a removable medium 311. When the computer program is executed by the central processing unit (CPU) 301, various functions defined in the present invention are performed.

[0074] It should be noted that specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM), a flash memory, an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In the present invention, a computer-readable storage medium may be any tangible medium containing or storing a program that may be used by or in combination with an instruction execution system, apparatus, or device.

[0075] The flowcharts and block diagrams in the accompanying drawings illustrate the possible architecture, functions and operations of the systems, methods and computer program products according to various embodiments of the present invention. Each box in the flowchart or block diagram may represent a module, a program segment, or a part of a code, and the above-mentioned module, program segment, or a part of a code contains one or more executable instructions for implementing the specified logical functions. It should also be noted that in some alternative implementations, the functions marked in the box may also occur in an order different from that marked in the accompanying drawings.

[0076] Specifically, the voice cloning system of this embodiment includes a processor and a memory. The memory stores a computer program. When the computer program is executed by the processor, the voice cloning method provided in the above embodiment is implemented.

[0077] As another aspect, the present invention further provides a computer-readable storage medium, which may be included in the voice cloning system described in the above embodiment; or may exist independently without being assembled into the voice cloning system. The above storage medium carries one or more computer programs, and when the above one or more computer programs are executed by a processor of the voice cloning system, the voice cloning system implements the voice cloning method provided in the above embodiment.

[0078] As described above, the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present application.

[0079] As used in the above embodiments, the term "when..." may be interpreted to mean "if..." or "after..." or "in response to determining..." or "in response to detecting...", depending on the context. Similarly, the phrases "upon determining..." or "if (the stated condition or event) is detected" may be interpreted to mean "if determining..." or "in response to determining..." or "upon detecting (the stated condition or event)" or "in response to detecting (the stated condition or event)", depending on the context.

[0080] Those skilled in the art can understand that to implement all or part of the processes in the above-mentioned embodiments, the processes can be completed by computer programs to instruct related hardware, and the programs can be stored in computer-readable storage media. When the programs are executed, they can include the processes of the above-mentioned method embodiments. The aforementioned storage media include: ROM or random access memory RAM, magnetic disk or optical disk and other media that can store program codes.

Claims

1. A voice cloning method, characterized in that: Applied to a voice cloning system, the method comprises: Collect the original speech of the target speaker; Performing a preprocessing operation on the original speech to obtain optimized speech data, wherein the preprocessing operation includes noise reduction processing, audio quality standardization processing and sentence segmentation processing; Extracting speech feature data from the optimized speech data, wherein the speech feature data includes pitch feature, timbre feature, duration feature and intensity feature; The speech feature data is trained by a preset deep learning model to obtain a trained speech cloning model, wherein the speech cloning model is used to clone the speech of the target speaker; Based on the voice cloning model, generating a cloned voice according to the input text content; Calculating the similarity between the cloned voice and the original voice; When the similarity exceeds a preset similarity threshold, the cloned voice is output.

2. The method according to claim 1, characterized in that The preprocessing operation is performed on the original speech to obtain optimized speech data, wherein the preprocessing operation includes noise reduction processing, audio quality standardization processing and sentence segmentation processing, and specifically includes: De-noising the original speech signal to obtain de-noised speech data; Unifying the audio parameters of the noise-reduced speech data to obtain standardized speech data; Based on a preset pause duration threshold, dividing the standardized voice data into a plurality of voice segments; The voice segments whose duration is less than a preset duration threshold are deleted to obtain the optimized voice data.

3. The method according to claim 1, characterized in that The voice feature data is extracted from the optimized voice data, wherein the voice feature data includes pitch feature, timbre feature, duration feature and intensity feature, and specifically includes: Performing fundamental frequency detection on the optimized speech data to obtain the pitch feature; Extracting the timbre features in the optimized speech data by spectrum analysis; Determining the duration feature in the optimized speech data according to the duration of the phoneme and the speech rate; Performing energy analysis on the optimized speech data to obtain the intensity feature; The speech feature data is constructed based on the pitch feature, the timbre feature, the duration feature and the intensity feature.

4. The method according to claim 1, characterized in that: The speech feature data is trained by a preset deep learning model to obtain a trained speech cloning model, wherein the speech cloning model is used to clone the speech of the target speaker, and the preset deep learning model includes an encoder and a decoder, specifically including: Dividing the speech feature data into a training data set and a verification data set; Inputting the training data set into the encoder to obtain the hidden features of the target speaker; The hidden layer features are combined with the text feature data corresponding to the speech feature data to obtain a comprehensive feature; Inputting the comprehensive features into the decoder to generate speech prediction features; Calculating a loss function based on the speech prediction features and the speech feature data in the verification data set; When the loss function is less than a preset loss function threshold, the voice cloning model is obtained.

5. The method according to claim 4, characterized in that The step of fusing the hidden layer features with the text feature data corresponding to the speech feature data to obtain comprehensive features specifically includes: Convert the text feature data into a word vector sequence; Calculating the correlation weight between the hidden layer features and the word vector sequence; Based on the correlation weight, weighted combination is performed on the hidden layer features and the word vector sequence to obtain a feature combination vector; The dimension of the feature combination vector is transformed to obtain the comprehensive feature.

6. The method according to claim 1, characterized in that After the step of outputting the cloned voice when the similarity exceeds a preset similarity threshold, the method further includes: Performing sentiment analysis on the text content to obtain sentiment feature parameters; Performing emotion adjustment on the cloned voice based on the emotion characteristic parameter to obtain an optimized cloned voice; Calculating the matching degree between the optimized cloned voice and the preset emotional voice template; When the matching degree is greater than a preset matching degree threshold, the optimized cloned voice is used as the final output voice.

7. The method according to claim 6, characterized in that The step of performing emotion adjustment on the cloned voice based on the emotion feature parameter to obtain an optimized cloned voice specifically includes: Determining a target emotion model according to a preset emotion model library and the emotion feature parameters; Based on the target emotion model, the pitch curve, speech speed, pause duration and volume intensity of the cloned voice are adjusted to obtain the optimized cloned voice.

8. A voice cloning system, characterized in that: The voice cloning system comprises: one or more processors and a memory; the memory is coupled to the one or more processors, the memory is used to store computer program code, the computer program code comprises computer instructions, and the one or more processors call the computer instructions to enable the voice cloning system to perform the method according to any one of claims 1 to 7.

9. A computer-readable storage medium comprising instructions, characterized in that: When the instruction is executed on the voice cloning system, the voice cloning system is caused to execute the method according to any one of claims 1 to 7.

10. A computer program product, characterized in that When the computer program product is run on a voice cloning system, the voice cloning system is enabled to perform the method according to any one of claims 1 to 7.

Images