Voice cloning method and device based on tts speech technology and storage medium

Abstract

This invention relates to the field of speech synthesis technology, and more particularly to a voice cloning method, apparatus, and storage medium based on TTS (Text-to-Speech) technology. The method includes: acquiring target audio data and preprocessing the target audio data; extracting target acoustic features from the preprocessed target audio data and constructing a target acoustic parameter database based on the extracted target acoustic features; acquiring target text and its sentiment tags, and generating a phoneme sequence and a target parameter sequence; traversing the phoneme sequence to locate vowel nodes, and optimizing the target parameter sequence based on the location results; generating synthesized audio based on the target parameter sequence, and performing a voiceprint consistency check on the synthesized audio. This invention significantly reduces the dependence on original recording data and production costs, and greatly improves the efficiency of voice generation.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of speech synthesis, in particular to a voice cloning method and device based on TTS speech technology and a storage medium. BACKGROUND

[0002] Currently, although text-to-speech (TTS) technology has become increasingly mature, it still faces significant challenges when applied to fields such as audiobooks and radio dramas that require high artistic expressiveness. Traditional speech synthesis methods either generate mechanical sounds that lack individuality and emotion or require the recording of a large amount of voice data for each character, resulting in high production costs and long production cycles.

[0003] Although existing technologies attempt to clone voices using a small number of samples, the generated voices often lack consistency in tone, naturalness, and emotional expressiveness, and may exhibit unstable tone quality, emotional detachment, or harsh transitions, making it difficult to meet the market's urgent demand for high-quality, efficient, and personalized emotional voice generation. Therefore, there is an urgent need for an innovative solution that can quickly clone personalized voices that are both faithful and expressive based on limited data. SUMMARY

[0004] The present application aims to provide a voice cloning method and device based on TTS speech technology and a storage medium to address at least one of the problems in the prior art.

[0005] To achieve the above-mentioned purpose, the present application adopts the following technical solutions:

[0006] A voice cloning method based on TTS speech technology, comprising:

[0007] Collecting target audio data and pre-processing the target audio data;

[0008] Extracting target acoustic features of the pre-processed target audio data and constructing a target acoustic parameter database based on the extracted target acoustic features;

[0009] Obtaining a target text and its emotional label, and generating a phoneme sequence and a target parameter sequence;

[0010] Traversing the phoneme sequence to locate vowel nodes and optimizing the target parameter sequence based on the location results;

[0011] Generating synthesized audio based on the target parameter sequence and performing voiceprint consistency verification on the synthesized audio.

[0012] Further, the pre-processing process includes noise reduction, silence segment removal, volume normalization, and accurate phoneme-level timestamp annotation based on specific text.

[0013] Furthermore, the target audio data is divided into the smallest phoneme units based on the phoneme-level timestamps of the target audio data, and the acoustic features of the audio segments of each phoneme unit are extracted. The acoustic features include the fundamental frequency, formant parameters, energy value, and duration of the audio segment. The fundamental frequency is the average value of the fundamental frequency of the audio segment, the formant parameters are the center values ​​of the first three formant frequencies of the audio segment, the energy value is the root mean square energy of the audio segment, and the duration is the duration of the audio segment.

[0014] Furthermore, the audio segments are labeled with emotional tags based on the emotional tags described for each phoneme unit in a specific text;

[0015] Calculate the average acoustic features of audio segments of the same phoneme unit under the same emotion tag in the target audio data, and denote the average fundamental frequency value of the audio segment of the i-th phoneme unit as F0i, the average formant parameters of the audio segment of the i-th phoneme unit as (F1i, F2i, F3i), where F1i is the first formant peak of the audio segment of the i-th phoneme unit, F2i is the second formant peak of the audio segment of the i-th phoneme unit, F3i is the third formant peak of the audio segment of the i-th phoneme unit, the average energy value of the audio segment of the i-th phoneme unit is denoteed as Ei, and the average duration of the audio segment of the i-th phoneme unit is denoteed as Di. {F0i, F1i, F2i, F3i, Ei, Di} are the acoustic features of phoneme i under the emotion tag.

[0016] A target acoustic parameter database is constructed using {phoneme i, emotion marker} as the key and {F0i, F1i, F2i, F3i, Ei, Di} as the value.

[0017] Furthermore, the target text is segmented and converted into pinyin to obtain a phoneme sequence. The phoneme sequence of the target text is traversed, and the acoustic features of the corresponding phoneme are queried from the target acoustic parameter database using each phoneme and the sentiment tag of the target text as the key.

[0018] Arrange the acoustic features of all the retrieved phonemes in order to form the target fundamental frequency sequence, target formant sequence, target energy sequence, and target duration sequence.

[0019] Furthermore, the phoneme sequence is traversed to identify all vowel nodes. For each vowel node vj, if all adjacent phonemes are vowel nodes, it is taken as the target node. The second resonant peak F2(j-1) of the vowel node v(j-1) preceding the target node is obtained, and the second resonant peak F2(j+1) of the vowel node v(j+1) following the target node is obtained. The second resonant peak F2j of the target node is optimized and set as F2j', where F2j' = w1 × F2j + w2 × F2(j-1) + w3 × F2(j+1). Here, w1 is the weight factor of the current vowel itself, w2 is the forward influence weight factor, and w3 is the backward influence weight factor.

[0020] Furthermore, the target parameter sequence is input into a pre-trained parametric speech synthesizer, which synthesizes speech waveforms segment by segment according to the phoneme sequence and corresponding acoustic parameters, and outputs synthesized audio.

[0021] Furthermore, a pre-trained deep neural network voiceprint embedding model is used to extract the voiceprint feature vectors of the original speech sample of the target character and the voiceprint feature vectors of the synthesized audio, respectively. The cosine similarity between the voiceprint feature vectors of the original speech sample of the target character and the voiceprint feature vectors of the synthesized audio is calculated as the voiceprint consistency score. If the voiceprint consistency score is greater than or equal to 0.85, the voiceprint consistency test is deemed to have passed; otherwise, the voiceprint consistency test is deemed to have failed.

[0022] According to another aspect of this application, a voice cloning device based on TTS voice technology is provided, comprising:

[0023] The preprocessing unit is used to acquire target audio data and preprocess the target audio data;

[0024] The database construction unit is used to extract the target acoustic features of the preprocessed target audio data and construct a target acoustic parameter database based on the extracted target acoustic features.

[0025] The sequence generation unit is used to acquire the target text and its sentiment tags, and generate a phoneme sequence and a target parameter sequence;

[0026] An optimization unit is used to traverse the phoneme sequence to locate vowel nodes and optimize the target parameter sequence based on the location results.

[0027] The verification unit is used to generate synthesized audio based on the target parameter sequence and to verify the consistency of the synthesized audio with voiceprints.

[0028] According to another aspect of this application, a computer-readable storage medium is provided, the computer-readable storage medium storing a computer program, wherein the computer program is used to control the electronic device in which the computer-readable storage medium is located to execute the aforementioned voice cloning method based on TTS voice technology during runtime.

[0029] The beneficial effects of this invention are as follows: First, rigorous data acquisition and preprocessing lay a high-quality data foundation for voice cloning; second, through refined acoustic feature analysis and database construction, the unique voiceprint features and emotional expression patterns of the target character are accurately captured and stored; third, combined with intelligent parameter sequence optimization and advanced neural network synthesis technology, it is possible to quickly generate voices that are highly consistent with the target character's timbre, emotionally rich, and naturally fluent. The overall solution significantly reduces reliance on original recording data and production costs, greatly improves generation efficiency, and provides strong technical support for audio content creation, enabling the large-scale, high-quality creation of highly attractive personalized character voices, effectively enhancing the immersive experience and artistic value of the work. Attached Figure Description

[0030] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0031] Figure 1 This is a flowchart illustrating the voice cloning method based on TTS voice technology in this embodiment.

[0032] Figure 2 This is a flowchart illustrating the method for constructing the acoustic parameter database in this embodiment.

[0033] Figure 3 This is a flowchart illustrating the voiceprint consistency verification method in this embodiment.

[0034] Figure 4 This is a schematic diagram of the sound cloning device based on TTS voice technology in this embodiment. Detailed Implementation

[0035] To more clearly illustrate the present invention, the following description, in conjunction with preferred embodiments and accompanying drawings, further explains the invention. Similar components in the drawings are indicated by the same reference numerals. Those skilled in the art should understand that the specific description below is illustrative rather than restrictive and should not be construed as limiting the scope of protection of the present invention.

[0036] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate for the embodiments of this application described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or elements is not necessarily limited to those explicitly listed, but may include other steps or elements not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0037] Specifically, the voice cloning method based on TTS speech technology described in this application is used to quickly generate highly consistent and expressive personalized voices for specific characters in audiobooks or radio dramas.

[0038] Please see Figure 1 As shown, it is a flowchart illustrating the voice cloning method based on TTS voice technology in this embodiment, including:

[0039] Step S101: Collect target audio data and preprocess the target audio data; the target audio data is the original voice sample data of the target character, which is a specific text read by the voice actor of the target character in a professional recording studio environment; the specific text contains all phonemes, covers different tones of high, medium and low, and contains four basic emotional sentences of calm, joy, sadness and anger.

[0040] Specifically, target audio data is collected using a high-fidelity microphone and sound card. After collection, the target audio is preprocessed. This embodiment does not impose specific limitations on the data collection method, and those skilled in the art can freely set it according to their needs.

[0041] Specifically, the preprocessing process involves denoising the target audio data, removing silent segments, normalizing the volume, and performing precise phoneme-level timestamp annotation based on the read-out text to form a time-aligned set of (text, audio segment) data pairs. The phoneme-level timestamp annotation is the process of determining the precise start and end times of each phoneme in the original audio. The phonemes are the initials and finals in Chinese Pinyin. For initials, they are defined as consonant units without tone, such as: b, p, m, f, d, t, n, l, ... For finals, they are considered as an inseparable whole with tone, and each final with tone is considered an independent phoneme, such as: ā, á, ǎ, à, which are four different phonemes.

[0042] Specifically, the preprocessing implementation process in this embodiment is as follows: The acquired raw target audio data is first imported into professional audio processing software or scripts, and noise reduction is performed using spectral subtraction or a deep learning-based noise reduction model. Silence segments are removed based on energy threshold detection, and volume is normalized. Then, a forced alignment tool, such as MontrealForced Aligner, is used to generate phoneme-level timestamps accurate to the millisecond level based on the text being read. Finally, a set of data pairs consisting of numerous time-series strictly aligned (text, audio segments) is output. This embodiment does not specifically limit the implementation process of preprocessing; those skilled in the art can freely set it according to their needs.

[0043] Specifically, by capturing pristine raw speech in a high-standard recording environment and performing preprocessing such as noise reduction, cropping, and standardization, a high-quality data foundation is laid for the entire system. This ensures that the objects of subsequent processing and analysis are clear, consistent, and interference-free speech units.

[0044] Please see Figure 1 As shown, the voice cloning method based on TTS speech technology also includes:

[0045] Step S102: Extract the target acoustic features from the preprocessed target audio data, and construct a target acoustic parameter database based on the extracted target acoustic features.

[0046] Please see Figure 2 As shown, the method for constructing the target acoustic parameter database includes:

[0047] Step S201: Perform multi-dimensional acoustic feature analysis on the preprocessed target audio data to extract target acoustic features.

[0048] Specifically, the target audio data is divided into the smallest phoneme units based on the phoneme-level timestamps, and acoustic features are extracted from the audio segments of each phoneme unit. The acoustic features include the fundamental frequency, formant parameters, energy value, and duration of the audio segment. The fundamental frequency is the average value of the fundamental frequency of the audio segment, the formant parameters are the center values ​​of the first three formant frequencies of the audio segment, the energy value is the root mean square energy of the audio segment, and the duration is the duration of the audio segment.

[0049] Specifically, the acoustic feature extraction process involves precisely dividing the preprocessed target audio data into independent phoneme unit audio segments corresponding to the smallest unit of articulation based on the phoneme-level timestamps obtained through a forced alignment tool. Subsequently, for each phoneme unit audio segment, signal processing algorithms are used to extract its multidimensional acoustic features: the average value of the fundamental frequency of the audio segment is calculated using an autocorrelation algorithm or a deep learning-based prediction model as its fundamental frequency value; the center values ​​of the first three formant frequencies in the spectral envelope of the audio segment are obtained using a linear predictive coding (LPC) algorithm or cepstral analysis as formant parameters, and the root mean square (RMS) energy of the audio segment signal is calculated as its energy value; and the duration of the phoneme is directly calculated based on the start and end time difference of the timestamp as its duration value. In this embodiment, the above settings are not specifically limited, and those skilled in the art can freely set them according to their needs.

[0050] Specifically, by performing extremely detailed acoustic analysis of speech, decomposing speech into the smallest phoneme units and extracting multidimensional features, it is possible to accurately and quantitatively capture the most essential vocal characteristics of the speaker, such as timbre, pitch, articulation strength, and rhythm, laying a solid foundation for building a detailed acoustic model.

[0051] Please continue reading. Figure 2 As shown, the method for constructing the target acoustic parameter database includes:

[0052] Step S202: Construct a target acoustic parameter database based on the extracted target acoustic features.

[0053] Specifically, emotional tags are assigned to audio segments based on the emotional labels described for each phoneme unit in a specific text;

[0054] Calculate the average acoustic features of the audio segments of the same phoneme unit under the same emotion tag in the target audio data, and denote the average fundamental frequency value of the audio segment of the i-th phoneme unit as F0i, the average formant parameters of the audio segment of the i-th phoneme unit as (F1i, F2i, F3i), where F1i is the first formant peak of the audio segment of the i-th phoneme unit, F2i is the second formant peak of the audio segment of the i-th phoneme unit, F3i is the third formant peak of the audio segment of the i-th phoneme unit, the average energy value of the audio segment of the i-th phoneme unit is denoteed as Ei, and the average duration of the audio segment of the i-th phoneme unit is denoteed as Di. {F0i, F1i, F2i, F3i, Ei, Di} are the acoustic features of phoneme i under the emotion tag.

[0055] A target acoustic parameter database DB is constructed using {phoneme i, emotion marker} as the key and {F0i, F1i, F2i, F3i, Ei, Di} as the value.

[0056] Specifically, scattered acoustic feature data are categorized, averaged, and stored according to phonemes and emotions, forming a knowledge base that the system can quickly access. This ensures that each pronunciation has a basis in fact during speech synthesis, thereby efficiently reproducing the stable and expressive voice of the target character.

[0057] Please see Figure 1 As shown, the voice cloning method based on TTS speech technology also includes:

[0058] Step S103: Obtain the target text and its sentiment tags, and generate a phoneme sequence and a target parameter sequence, wherein the target text is the text of the speech to be synthesized.

[0059] Specifically, the target text is segmented and converted into pinyin to obtain a phoneme sequence. The phoneme sequence of the target text is traversed, and the acoustic features of the corresponding phoneme are queried from the target acoustic parameter database using each phoneme and the sentiment tag of the target text as keys.

[0060] Arrange the acoustic features of all the retrieved phonemes in order to form the target fundamental frequency sequence, target formant sequence, target energy sequence, and target duration sequence.

[0061] Specifically, it realizes intelligent mapping from text to acoustic parameters. By querying the database, it automatically matches the most suitable acoustic parameters to be synthesized, thereby ensuring that the rhythm and emotional color of the synthesized speech can be highly consistent with the intention of the text content, so that the generated speech is not only similar in timbre, but also more accurate and vivid in its tone and emotion.

[0062] Please continue reading. Figure 1 As shown, the voice cloning method based on TTS speech technology also includes:

[0063] Step S104: Traverse the phoneme sequence to locate vowel nodes, and optimize the target parameter sequence based on the location results.

[0064] Specifically, the phoneme sequence is traversed to identify all vowel nodes. For each vowel node vj, if all adjacent phonemes are vowel nodes, it is taken as the target node. The second resonant peak F2(j-1) of the vowel node v(j-1) preceding the target node is obtained, and the second resonant peak F2(j+1) of the vowel node v(j+1) following the target node is obtained. The second resonant peak F2j of the target node is optimized and set as F2j', where F2j' = w1 × F2j + w2 × F2(j-1) + w3 × F2(j+1). Here, w1 is the weight factor of the current vowel itself, w2 is the forward influence weight factor, w3 is the backward influence weight factor, and w1 + w2 + w3 = 1.

[0065] Specifically, in this embodiment, the current vowel weight factor is the contribution of the current vowel's target value to the final result, with a value of 0.7; the forward influence weight factor is the influence strength of the previous vowel on the current vowel, with a value of 0.25; and the backward influence weight factor is the influence strength of the next vowel on the current vowel, with a value of 0.05.

[0066] Specifically, the vowel nodes are all phoneme units classified as vowels in the phoneme sequence obtained from the target text, including simple vowels (a, o, e, i, u, ü), compound vowels (such as ai, ou), and nasal vowels (such as an, eng).

[0067] Specifically, the implementation process of traversing the phoneme sequence to locate vowel nodes is as follows: First, obtain the phoneme sequence obtained in step S103, and traverse each phoneme unit in the sequence from beginning to end. In this embodiment, a set of Chinese vowel phonemes is preset (containing all possible simple vowels, compound vowels, and nasal vowel phonemes). By matching the current phoneme with this set, it is determined whether it belongs to a vowel. If the match is successful, the index position of the phoneme in the sequence is immediately marked as a vowel node. This process is repeated until the end of the sequence, and finally an ordered list of vowel nodes containing the index positions of all vowel phonemes is output. This list provides a clear processing object and order for the formant smoothing optimization of vowel nodes in the subsequent step S104.

[0068] Specifically, in this embodiment, the optimization process of the second resonance peak of the target node is carried out sequentially according to the order of vowel nodes. That is, if the second resonance peak of a vowel node has been optimized, the optimized second resonance peak of the vowel node is used in the subsequent optimization process of the second resonance peak of the vowel node.

[0069] Specifically, by intelligently smoothing the formant transition between vowels, it effectively eliminates the abruptness and mechanical feeling that may be produced by phoneme splicing, making the connection between syllables as smooth and pleasant as a real person's voice, and significantly enhancing the auditory experience.

[0070] Please continue reading. Figure 1 As shown, the voice cloning method based on TTS speech technology also includes:

[0071] Step S105: Generate synthesized audio based on the target parameter sequence, and perform voiceprint consistency verification on the synthesized audio.

[0072] Please see Figure 3 As shown, the voiceprint consistency verification method includes:

[0073] Step S301: Generate synthesized audio based on the target parameter sequence.

[0074] Specifically, the target parameter sequence is input into a pre-trained parametric speech synthesizer, and the speech waveform is synthesized segment by segment according to the phoneme sequence and corresponding acoustic parameters. During the synthesis process, the duration of each phoneme segment is controlled according to the duration sequence, the pitch contour is adjusted according to the fundamental frequency sequence, the spectral envelope is shaped according to the formant sequence, and the amplitude variation is adjusted according to the energy sequence, and finally the synthesized audio is output.

[0075] Specifically, the pre-trained parametric speech synthesizer described in this application can be a neural network acoustic model. In the pre-training stage, a high-quality speech dataset containing hundreds of hours of data and covering a large number of different speakers, along with their corresponding texts, is used. This dataset needs to undergo refined preprocessing and phoneme annotation similar to step S101. During training, the model takes the phoneme sequence of the text and its prosodic boundaries as input, and the corresponding acoustic features, such as Mel spectrum, fundamental frequency F0, energy, spectral envelope, or glottal impulse, as the learning target. It models the complex nonlinear relationship between the input sequence and the output acoustic feature sequence through deep neural networks such as Tacotron2, FastSpeech2, or Transformer-based architectures. The training objective is to minimize the loss function between the predicted acoustic features and the true features, such as L1 or L2 loss. In this embodiment, the above settings are not specifically limited, and those skilled in the art can freely set them according to their needs.

[0076] Specifically, by employing a pre-trained neural network model, the system can mimic and reproduce the complex variation patterns of human voices, thereby ensuring that the synthesized speech meets high standards in terms of sound quality, fluency, and naturalness. This is the core technology for achieving high-quality speech synthesis.

[0077] Please continue reading. Figure 3 As shown, the voiceprint consistency verification method further includes:

[0078] Step S302: Perform a voiceprint consistency check on the synthesized audio.

[0079] Specifically, a pre-trained deep neural network voiceprint embedding model is used to extract the voiceprint feature vectors of the original speech sample of the target character and the voiceprint feature vectors of the synthesized audio, respectively. The cosine similarity between the voiceprint feature vectors of the original speech sample of the target character and the voiceprint feature vectors of the synthesized audio is calculated as the voiceprint consistency score. If the voiceprint consistency score is greater than or equal to 0.85, the voiceprint consistency test is deemed to have passed; otherwise, the voiceprint consistency test is deemed to have failed.

[0080] If the voiceprint consistency check fails, a warning will be sent to the user, along with a request for additional data.

[0081] Specifically, in this embodiment, the supplementary data prompt refers to the voice actors recording more specific text, especially sentences containing more different phoneme combinations and intonations.

[0082] Specifically, in this embodiment, the voiceprint feature vector refers to a high-dimensional numerical vector extracted from a speech signal using a pre-trained deep neural network voiceprint model, such as ECAPA-TDNN, to characterize the speaker's identity. This vector is a fixed-dimensional array of real numbers, which can effectively capture the speaker's essential features such as physiological and pronunciation habits. In this verification process, the original speech sample of the target character and the synthesized audio to be verified are input into the above model to extract the corresponding voiceprint feature vector for subsequent similarity calculation and consistency determination.

[0083] Specifically, advanced voiceprint recognition technology is used to automatically verify the similarity between the synthesized speech and the target voice, ensuring the reliability of the output results. When deviations are detected, the system can proactively provide improvement solutions, effectively enhancing the system's usability and user trust.

[0084] Please see Figure 4 As shown, the voice cloning device based on TTS voice technology includes:

[0085] The preprocessing unit is used to acquire target audio data and preprocess the target audio data;

[0086] The database construction unit is used to extract the target acoustic features of the preprocessed target audio data and construct a target acoustic parameter database based on the extracted target acoustic features.

[0087] The sequence generation unit is used to acquire the target text and its sentiment tags, and generate a phoneme sequence and a target parameter sequence;

[0088] An optimization unit is used to traverse the phoneme sequence to locate vowel nodes and optimize the target parameter sequence based on the location results.

[0089] The verification unit is used to generate synthesized audio based on the target parameter sequence and to verify the consistency of the synthesized audio with voiceprints.

[0090] The voice cloning device based on TTS speech technology provided in this application can execute the voice cloning method based on TTS speech technology provided in any embodiment of this application, and has the corresponding functional modules and beneficial effects of the execution method.

[0091] This application also provides a computer-readable storage medium, which is a tangible physical storage medium that can store the aforementioned computer program and various types of data used in the program; the physical storage medium includes, but is not limited to, existing physical storage media or combinations thereof, such as random access memory, read-only memory, optical disk, and hard disk.

[0092] Those skilled in the art will understand that all or some of the steps and systems in the methods disclosed above can be implemented as software, firmware, hardware, and suitable combinations thereof. Some or all of the physical components can be implemented as software executed by a processor, such as a central processing unit, digital signal processor, or microprocessor, or as hardware, or as an integrated circuit, such as an application-specific integrated circuit. Such software can be distributed on a computer-readable medium, which can include computer storage media (or non-transitory media) and communication media (or transient media). As is known to those skilled in the art, the term computer storage media includes volatile and non-volatile, removable and non-removable media implemented in any method or technique for storing information (such as computer-readable programs, data structures, program modules, or other data). Furthermore, it is known to those skilled in the art that communication media typically contain computer-readable programs, data structures, program modules, or other data in modulated data signals such as carrier waves or other transmission mechanisms, and can include any information delivery medium.

[0093] Obviously, the above embodiments of the present invention are merely examples for clearly illustrating the present invention, and are not intended to limit the implementation of the present invention. For those skilled in the art, other variations or modifications can be made based on the above description. It is impossible to exhaustively list all the implementation methods here. All obvious variations or modifications derived from the technical solutions of the present invention are still within the protection scope of the present invention.

Claims

1. A voice cloning method based on TTS speech technology, characterized in that, include: Collect target audio data and preprocess the target audio data; Extract the target acoustic features from the preprocessed target audio data, and construct a target acoustic parameter database based on the extracted target acoustic features; Obtain the target text and its sentiment tags, and generate a phoneme sequence and a target parameter sequence; Traverse the phoneme sequence to locate vowel nodes, and optimize the target parameter sequence based on the location results; Synthetic audio is generated based on the target parameter sequence, and the synthesized audio is subjected to voiceprint consistency verification. The audio segment is marked with emotion tags according to the emotion tags described by each phoneme unit in a specific text. The specific text is a sentence containing all phonemes, covering different tones from high to low, and containing four basic emotions: calm, joy, sadness, and anger. Calculate the average acoustic features of audio segments of the same phoneme unit under the same emotion marker in the target audio data, and denote the average fundamental frequency value of the audio segment of the i-th phoneme unit as F0i, the average formant parameters of the audio segment of the i-th phoneme unit as {F1i, F2i, F3i}, where F1i is the first formant peak of the audio segment of the i-th phoneme unit, F2i is the second formant peak of the audio segment of the i-th phoneme unit, F3i is the third formant peak of the audio segment of the i-th phoneme unit, the average energy value of the audio segment of the i-th phoneme unit is denoteed as Ei, and the average duration of the audio segment of the i-th phoneme unit is denoteed as Di. {F0i, F1i, F2i, F3i, Ei, Di} are the acoustic features of phoneme i under the emotion marker. A target acoustic parameter database is constructed using {phoneme i, emotion marker} as the key and {F0i, F1i, F2i, F3i, Ei, Di} as the value; Traverse the phoneme sequence and identify all vowel nodes. For each vowel node vj, if all adjacent phonemes are vowel nodes, then it is taken as the target node. Obtain the second resonance peak F2(j-1) of the vowel node v(j-1) preceding the target node and the second resonance peak F2(j+1) of the vowel node v(j+1) following the target node. Optimize the second resonance peak F2j of the target node and set the optimized second resonance peak of the target node as F2j', where F2j' = w1 × F2j + w2 × F2(j-1) + w3 × F2(j+1); where w1 is the weight factor of the current vowel itself, w2 is the forward influence weight factor, and w3 is the backward influence weight factor.

2. The voice cloning method based on TTS speech technology according to claim 1, characterized in that, The preprocessing process involves noise reduction, silent segment removal, volume normalization, and precise phoneme-level timestamp annotation of the target audio data based on the specific text.

3. The voice cloning method based on TTS speech technology according to claim 2, characterized in that, The target audio data is segmented into the smallest phoneme units based on the phoneme-level timestamps, and the acoustic features of each phoneme unit's audio segment are extracted. The acoustic features include the fundamental frequency, formant parameters, energy value, and duration of the audio segment. The fundamental frequency is the average value of the fundamental frequency of the audio segment. The formant parameters are the center values ​​of the first three formant frequencies of the audio segment. The energy value is the root mean square energy of the audio segment. The duration is the duration of the audio segment.

4. The voice cloning method based on TTS speech technology according to claim 3, characterized in that, The target text is segmented and converted into pinyin to obtain a phoneme sequence. The phoneme sequence of the target text is traversed, and the acoustic features of the corresponding phoneme are queried from the target acoustic parameter database using each phoneme and the sentiment tag of the target text as the key. Arrange the acoustic features of all the retrieved phonemes in order to form the target fundamental frequency sequence, target formant sequence, target energy sequence, and target duration sequence.

5. The voice cloning method based on TTS speech technology according to claim 4, characterized in that, The target parameter sequence is input into a pre-trained parametric speech synthesizer, which synthesizes speech waveforms segment by segment according to the phoneme sequence and corresponding acoustic parameters, and outputs synthesized audio.

6. The voice cloning method based on TTS speech technology according to claim 5, characterized in that, A pre-trained deep neural network voiceprint embedding model is used to extract the voiceprint feature vectors of the original speech sample of the target character and the voiceprint feature vector of the synthesized audio. The cosine similarity between the voiceprint feature vector of the original speech sample of the target character and the voiceprint feature vector of the synthesized audio is calculated as the voiceprint consistency score. If the voiceprint consistency score is greater than or equal to 0.85, the voiceprint consistency test is deemed to have passed; otherwise, the voiceprint consistency test is deemed to have failed.

7. A voice cloning device based on TTS speech technology, applied to the voice cloning method based on TTS speech technology as described in any one of claims 1-6, characterized in that, include: The preprocessing unit is used to acquire target audio data and preprocess the target audio data; The database construction unit is used to extract the target acoustic features of the preprocessed target audio data and construct a target acoustic parameter database based on the extracted target acoustic features. The sequence generation unit is used to acquire the target text and its sentiment tags, and generate a phoneme sequence and a target parameter sequence; An optimization unit is used to traverse the phoneme sequence to locate vowel nodes and optimize the target parameter sequence based on the location results. The verification unit is used to generate synthesized audio based on the target parameter sequence and to verify the consistency of the synthesized audio with voiceprints.

8. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program, wherein the computer program is used to control the electronic device on which the computer-readable storage medium is located to execute the voice cloning method based on TTS voice technology as described in any one of claims 1-6 during runtime.

Images