Voice cloning timbre and rhythm adjustment method and system based on voice endpoint detection

Abstract

This invention relates to a method and system for adjusting the timbre and rhythm of voice cloning based on speech endpoint detection. The method includes acquiring the speech signal of a target speaker; resampling the speech signal at a preset sampling rate and performing noise reduction processing through a denoising autoencoder; dividing the pre-processed speech signal into silence segments and vocal segments; calculating the rhythm parameters of the speech signal and adjusting the rhythm parameters proportionally; adjusting the timbre of the speech signal through a preset filter function; adjusting and splicing audio segments; inputting the sound into a voice cloning model; and finally outputting natural speech. This invention adds a segmentation mechanism for silence segments and a strict rhythm control mechanism, fully utilizing silence and pause information to enhance the naturalness and auditory expressiveness of the speech.

Description

Technical Field

[0001] This invention belongs to the field of speech synthesis and speech processing technology, and specifically relates to a method and system for adjusting the timbre and rhythm of voice cloning based on speech endpoint detection. Background Technology

[0002] With the rise of applications such as virtual humans, intelligent customer service, immersive interaction, and metaverse, users' demands for voice experience have moved beyond simply "hearing clearly and understanding," pursuing higher standards of "natural, emotional, rhythmic, and personalized" delivery. Currently, voice cloning technology has made significant progress in speaker timbre reproduction and few-shot / zero-shot scenarios, but its controllability and fine-grained performance in rhythm, pauses, speech rate, and prosody expression remain severely lacking. Although most systems can achieve near-identical timbre and clear pronunciation, problems such as unnatural pauses and rhythms, speech rate drift, and limited expressiveness often occur in long / complex sentence structures.

[0003] While current mainstream voice cloning technologies achieve high fidelity in speaker timbre reproduction and short-shot / zero-shot replication, they still have significant shortcomings when generating entire speech segments, complex sentences, or long texts. First, these systems often lack fine-grained recognition of silence and speech segment boundaries. Prosodic features such as pause duration, utterance speed ranging, and intra- and inter-sentence rhythmic consistency are often weakened or ignored, resulting in generated speech that sounds flat, with abrupt pauses or rhythmic imbalances. Second, when the reference speech segment is short or has a high proportion of silence, the model struggles to obtain sufficient rhythmic / prosodic cues, leading to a significant decrease in the expressiveness and naturalness of the synthesized speech. Furthermore, when the training data includes samples with high noise levels, large differences in utterance speed, and uneven pause distribution, these highly variable samples introduce interference with rhythm / pause pattern estimation, degrading the model's rhythm control and pause quality in real-world use. Summary of the Invention

[0004] This invention provides a method and system for adjusting the timbre and rhythm of sound cloning based on speech endpoint detection, aiming to solve at least one of the technical problems existing in the prior art.

[0005] The technical solution of this invention relates to a method for adjusting the timbre and rhythm of sound cloning based on speech endpoint detection. This method is applied to a system for adjusting the timbre and rhythm of sound cloning based on speech endpoint detection. The system includes a speech acquisition module, a preprocessing module, a silence detection module, a rhythm adjustment module, a timbre adjustment module, and a speech splicing module connected sequentially. The method for adjusting the timbre and rhythm of sound cloning based on speech endpoint detection includes the following steps:

[0006] S100, The voice acquisition module acquires the voice signal x(t) of the target speaker;

[0007] S200, The preprocessing module resamples the speech signal x(t) at a preset sampling rate and performs noise reduction processing through a denoising autoencoder (DAE).

[0008] S300, the silence segment detection module distinguishes the preprocessed speech signal into silence segments and vocal segments;

[0009] S400. The rhythm adjustment module calculates the rhythm parameters of the speech signal and adjusts the rhythm parameters proportionally according to the preset rhythm target.

[0010] S500. The timbre adjustment module adjusts the timbre of the speech signal through a preset filter function without changing the speech rate and rhythm, thereby obtaining a speech signal with adjusted timbre.

[0011] S600, the speech splicing module adjusts and splices audio segments, inputs the sound into the speech cloning model, and finally outputs natural speech.

[0012] Furthermore, step S300 includes:

[0013] S310. Based on a preset time interval, measure the short-time energy Et and zero-crossing rate of the preprocessed speech signal. If the short-time energy Et is greater than a preset threshold and the zero-crossing rate is greater than a preset threshold, it is determined to be a vocal segment; otherwise, it is determined to be a silent segment.

[0014] S320. Based on the segmented producing and silent segments, output the boundary sequence B.

[0015] ,

[0016] Where i is the segment number of the vocal segment. Let be the start time of the i-th vocal segment. Let be the end time of the i-th vocal segment.

[0017] Furthermore, step S400 includes:

[0018] S410. Calculate the rhythm parameters of the speech signal, wherein the rhythm parameters include the duration of each vocal segment. Adjacent silence intervals and speech rate ratio ;

[0019] S420. Based on the preset rhythm target, the rhythm parameters are proportionally adjusted to obtain the duration of each vocal segment after adjustment. Adjusted adjacent silence intervals and the adjusted speech rate ratio The preset rhythm objectives include at least adjusting speech rate, adjusting pause time, and adjusting tone intensity;

[0020] S430, based on the adjusted duration of each vocal segment Adjusted adjacent silence intervals and the adjusted speech rate ratio The time scale is modified by stretching or compressing the length of silent and vocal segments of a speech signal using the Waveform Similarity Overlap-Add (WSOLA) algorithm.

[0021] Furthermore, in step S410,

[0022] Duration of each vocal segment for:

[0023] ,

[0024] Adjacent silence intervals for:

[0025] ,

[0026] Speech rate ratio for:

[0027] ,

[0028] Where i is the segment number of the vocal segment. Let be the start time of the i-th vocal segment. Let be the end time of the i-th vocal segment. The number of characters in the corresponding text clause.

[0029] Furthermore, in step S420,

[0030] Adjusted duration of each vocal segment for:

[0031] ,

[0032] The adjusted adjacent silence interval is as follows:

[0033] ,

[0034] The adjusted speech rate ratio is as follows:

[0035] ,

[0036] wherein, is the adjustment coefficient of the duration of each sounding segment of, is the adjustment coefficient of the adjacent silence interval of, is the number of characters of the corresponding text clause.

[0037] Furthermore, in step S500, the voice signal after tone adjustment is expressed as:

[0038] ,

[0039] wherein, h[n] is the impulse response of the FIR filter, x[n] is the discrete sampling sequence of the input voice signal, y[n] is the filtered output sequence, n is the discrete time index, k is the convolution summation index, M is the filter order (or length minus 1), and the symbol "*" represents the discrete convolution operation.

[0040] Furthermore, for the high-frequency enhanced voice signal for bright tone, the preset filter transfer function is:

[0041] ,

[0042] The enhanced signal is:

[0043] ,

[0044]

[0045] wherein, [f1, f2] is the high-frequency enhancement frequency band interval value, f1 is the lower limit, f2 is the upper limit, and f1 < f2, with the unit of Hz; is the high-frequency enhancement coefficient, with the value range (0, 1), used to control the enhancement intensity; H_bright(f) is the frequency response of the high-frequency enhancement filter, h_bright[n] is its time-domain impulse response; x[n] is the original voice sequence, and y[n] is the enhanced output sequence;

[0045] If it is used for enhancing the low-frequency of a voice signal with a thick timbre, the preset filter transfer function is:

[0046] ,

[0047] The enhanced signal is:

[0048] ,

[0049] where, [f3, f4] is the low-frequency enhancement frequency band interval value, f3 is the lower limit, f4 is the upper limit, and f3 < f4, with the unit of Hz; is the low-frequency enhancement coefficient, used to control the thickness; H_thick(f) is the frequency response of the low-frequency enhancement filter, h_thick[n] is its time-domain impulse response; x[n] is the original voice sequence, and y[n] is the enhanced output sequence.

[0050] Furthermore, if it is used to improve clarity and remove nasal sounds from the voice signal, it is achieved through a band-stop filter, and the preset filter transfer function is:

[0051] ,

[0052] The enhanced signal is:

[0053] ,

[0054] where, [f5, f6] is the frequency band interval value of the band-stop filter, f5 is the lower limit, f6 is the upper limit, and f5 < f6, with the unit of Hz; H_notch(f) is the frequency response of the band-stop filter, h_notch[n] is its time-domain impulse response; x[n] is the original voice sequence, and y[n] is the filtered output sequence.

[0055] Furthermore, the present invention also proposes a voice cloning timbre and rhythm adjustment system based on voice endpoint detection, used to implement the voice cloning timbre and rhythm adjustment method based on voice endpoint detection. The voice cloning timbre and rhythm adjustment system based on voice endpoint detection includes, connected in sequence:

[0056] A voice acquisition module, used to acquire the voice signal x(t) of the target speaker;

[0057] A preprocessing module, used to preprocess the voice signal x(t) acquired by the voice acquisition module, including resampling and denoising operations;

[0058] A silent segment detection module, which distinguishes the preprocessed voice signal into silent segments and sounding segments by combining energy determination, zero-crossing rate analysis, and a deep endpoint detection model;

[0059] The rhythm adjustment module is used to calculate and adjust rhythm parameters.

[0060] The timbre adjustment module adjusts the timbre of the rhythm-adjusted speech signal based on a preset filter.

[0061] The voice splicing module is used to adjust and splice audio segments.

[0062] Furthermore, the present invention also proposes a computer-readable storage medium storing program instructions thereon, which, when executed by a processor, implement the aforementioned sound cloning timbre and rhythm adjustment method based on speech endpoint detection.

[0063] Compared with existing technologies, the present invention has the following characteristics:

[0064] The aforementioned method and system for sound cloning timbre and rhythm adjustment based on speech endpoint detection adds a segmentation mechanism for silent segments and a strict rhythm control mechanism, making full use of silence and pause information to enhance the naturalness and auditory expressiveness of speech. This improves the rhythmic naturalness, rhythmic consistency, and expressiveness of generated speech in long / complex sentence structures. Attached Figure Description

[0065] Figure 1 This is a flowchart of a sound cloning timbre and rhythm adjustment method based on speech endpoint detection. Detailed Implementation

[0066] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0067] The following will provide a clear and complete description of the concept, specific structure, and technical effects of the present invention in conjunction with the embodiments and accompanying drawings, so as to fully understand the purpose, solution, and effects of the present invention.

[0068] It should be noted that, unless otherwise specified, when a feature is referred to as "fixed" or "connected" to another feature, it can be directly fixed or connected to the other feature, or indirectly fixed or connected to the other feature. The singular forms "a," "described," and "the" used herein are also intended to include the plural forms, unless the context clearly indicates otherwise. Furthermore, unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art. The terminology used in this specification is for the purpose of describing particular embodiments only and not for limiting the invention. The term "and / or" as used herein includes any combination of one or more of the associated listed items.

[0069] It should be understood that although the terms first, second, third, etc., may be used in this disclosure to describe various elements, these elements should not be limited to these terms. These terms are only used to distinguish elements of the same type from each other. For example, a first element may also be referred to as a second element without departing from the scope of this disclosure, and similarly, a second element may also be referred to as a first element. Any and all instances or exemplary language (“e.g.,” “such as,” etc.) used herein are intended only to better illustrate embodiments of the invention and, unless otherwise required, do not impose a limitation on the scope of the invention. Furthermore, the industry term “pose” as used herein refers to the position and orientation of an element relative to a spatial coordinate system.

[0070] Reference Figure 1 This invention provides a method and system for adjusting the timbre and rhythm of voice cloning based on voice endpoint detection. The method is applied to a system that includes a voice acquisition module, a preprocessing module, a silence detection module, a rhythm adjustment module, a timbre adjustment module, and a voice splicing module connected sequentially. The method comprises the following steps:

[0071] S100, The voice acquisition module acquires the voice signal x(t) of the target speaker;

[0072] S200, The preprocessing module resamples the speech signal x(t) at a preset sampling rate and performs noise reduction processing through a denoising autoencoder (DAE).

[0073] S300, the silence segment detection module distinguishes the preprocessed speech signal into silence segments and vocal segments;

[0074] S400. The rhythm adjustment module calculates the rhythm parameters of the speech signal and adjusts the rhythm parameters proportionally according to the preset rhythm target.

[0075] S500. The timbre adjustment module adjusts the timbre of the speech signal through a preset filter function without changing the speech rate and rhythm, thereby obtaining a speech signal with adjusted timbre.

[0076] S600, the speech splicing module adjusts and splices audio segments, inputs the sound into the speech cloning model, and finally outputs natural speech.

[0077] Compared with existing technologies, the present invention has the following characteristics:

[0078] The aforementioned method and system for sound cloning timbre and rhythm adjustment based on speech endpoint detection adds a segmentation mechanism for silent segments and a strict rhythm control mechanism, making full use of silence and pause information to enhance the naturalness and auditory expressiveness of speech. This improves the rhythmic naturalness, rhythmic consistency, and expressiveness of generated speech in long / complex sentence structures.

[0079] Specifically, this invention addresses these issues by proposing VAD endpoint detection as a front-end segmentation mechanism to identify the boundaries between silent and vocal segments. During the voice cloning process, these boundaries are used as anchor points to adjust rhythm (e.g., pause duration, speech rate ratio) and timbre (timbre brightness, formant characteristics, etc.), thereby improving the prosodic naturalness, rhythmic consistency, and expressiveness of the generated speech in long / complex sentence structures. This invention is particularly suitable for demanding scenarios such as virtual humans, intelligent agents, and multimodal voice interaction, where the speaker's rhythm and timbre control directly impact user experience and speech naturalness.

[0080] Compared with the prior art, the advantages of the present invention are as follows:

[0081] 1. This invention introduces Voice Activity Detection (VAD) to accurately segment speech signals into phonation and silence segments, improving the precision of rhythm adjustment. Compared to most current voice cloning systems, which generally model the entire speech segment without considering the boundary information between silence and phonation segments, this invention can identify the boundaries between silence and phonation and then adjust rhythmic characteristics such as pause duration and speech rate ratio with anchor points. This results in generated speech with a more natural sense of rhythm and pause positions in long / complex sentence structures that better match human speaking habits.

[0082] Voice Activity Detection (VAD) technology, as a fundamental module in speech signal processing, has been widely and maturely implemented, enabling the segmentation of silence and vocalization segments. It is widely used in speech recognition, communication systems, and noise suppression. Combining VAD with voice cloning provides natural boundary information for rhythm control, allowing the system to clearly know where to stop and where to speed up / slow down. This boundary-driven adjustment method has not yet been systematically utilized in most existing cloning systems. Employing this technology not only improves the naturalness of the sound and the fluency of the intonation but also maintains consistency in multilingual and multi-speaker environments. This is particularly important for scenarios with high requirements for rhythm and timbre, such as virtual humans, intelligent agents, dialogue systems, and broadcasting / dubbing. Therefore, this invention proposes using VAD technology as a front-end silence / voice segmentation method, combined with silence segment pruning and rhythm parameter adjustment strategies, to improve the overall performance of the voice cloning system in terms of rhythmic naturalness, intonation, and expressiveness, thereby meeting the higher demands of future user experiences.

[0083] 2. Enhanced timbre control. After the silence segmentation, this invention not only adjusts static timbre characteristics such as timbre brightness and formant features, but also, because the silence segment is standardized, the timbre modeling of the vocal segment is more focused and cleaner, avoiding interference from silence or noise on timbre expression and speaker characteristic reproduction.

[0084] 3. The system framework is scalable to multilingual and multi-speaker scenarios, exhibiting strong versatility and adaptability. In multilingual / multi-speaker voice cloning tasks, some existing works have demonstrated the feasibility of cross-language speaker adaptation, but significant shortcomings remain in rhythm and pause consistency. The silence segmentation + boundary rhythm control mechanism provided by this invention can be directly generalized to different languages / speakers without requiring a redesign of rhythm strategies for each language or speaker, thereby reducing data requirements and training complexity.

[0085] 4. Improved naturalness and user experience. The speech generated by this invention in long sentences and natural contexts will be more rhythmic and fluent than systems with static rhythms or no silence segmentation, thus significantly improving user evaluation (naturalness / expressiveness). One of the factors users are most sensitive to regarding the naturalness of a speech synthesis system is rhythmic characteristics such as prosody, pauses, and speech rate.

[0086] 5. Reduce invalid input and model resource waste. By standardizing silent segments, this invention reduces the computational and learning burden on the model in meaningless silent parts, making model training and inference more efficient.

[0087] Furthermore, refer to Figure 1 Step S300 includes:

[0088] S310. Based on a preset time interval, measure the short-time energy Et and zero-crossing rate of the preprocessed speech signal. If the short-time energy Et is greater than a preset threshold and the zero-crossing rate is greater than a preset threshold, it is determined to be a vocal segment; otherwise, it is determined to be a silent segment.

[0089] S320. Based on the segmented producing and silent segments, output the boundary sequence B.

[0090] ,

[0091] Where i is the segment number of the vocal segment. Let be the start time of the i-th vocal segment. Let be the end time of the i-th vocal segment.

[0092] Furthermore, refer to Figure 1 Step S400 includes:

[0093] S410. Calculate the rhythm parameters of the speech signal, wherein the rhythm parameters include the duration of each vocal segment. Adjacent silence intervals and speech rate ratio ;

[0094] S420. Based on the preset rhythm target, the rhythm parameters are proportionally adjusted to obtain the duration of each vocal segment after adjustment. Adjusted adjacent silence intervals and the adjusted speech rate ratio The preset rhythm objectives include at least adjusting speech rate, adjusting pause time, and adjusting tone intensity;

[0095] S430, based on the adjusted duration of each vocal segment Adjusted adjacent silence intervals and the adjusted speech rate ratio The time scale is modified by stretching or compressing the length of silent and vocal segments of a speech signal using the Waveform Similarity Overlap-Add (WSOLA) algorithm.

[0096] Furthermore, refer to Figure 1 In step S410,

[0097] Duration of each vocal segment for:

[0098] ,

[0099] Adjacent silence intervals for:

[0100] ,

[0101] Speech rate ratio for:

[0102] ,

[0103] Where i is the segment number of the vocal segment. Let be the start time of the i-th vocal segment. Let be the end time of the i-th vocal segment. The number of characters in the corresponding text clause.

[0104] Specifically, and The boundary sequence B obtained from steps S310-S320 can be extracted and converted into time or sample point number according to the sampling rate. The input text is segmented into clauses by the text processing module according to punctuation or semantic pauses, and then aligned with the i-th vocal segment. Specifically, coarse alignment based on duration ratio or fine alignment using a forced alignment model can be employed. The number of characters (or phonemes / syllables) in the clause is then counted as the threshold. .

[0105] Furthermore, refer to Figure 1 In step S420,

[0106] Adjusted duration of each vocal segment for:

[0107] ,

[0108] Adjusted adjacent silence interval for:

[0109] ,

[0110] Adjusted speech rate ratio for:

[0111] ,

[0112] in, Duration of each vocal segment Adjustment coefficient, Adjacent silence intervals Adjustment coefficient, The number of characters in the corresponding text clause.

[0113] Specifically, The scaling factor used to control the duration of the i-th vocal segment is determined by a preset speech rate target or a speech rate ratio set by the user, for example, based on the ratio of the target speech rate r_target to the original speech rate r_i. Used to control the scaling of adjacent silence intervals, its value is obtained by mapping from a preset pause target or emotional intensity parameter; The method for obtaining it is the same as above, which is the number of characters in the text clause corresponding to the i-th vocal segment.

[0114] Furthermore, refer to Figure 1 In step S500, the voice signal after timbre adjustment Represented as:

[0115] ,

[0116] Where h[n] is the impulse response of the FIR filter, x[n] is the discrete sampling sequence of the input speech signal, y[n] is the filtered output sequence, n is the discrete time index, k is the convolution summation index, M is the filter order (or length minus 1), and the symbol "*" represents the discrete convolution operation.

[0117] Specifically, to clarify the origin of the above filter formula, the target amplitude response H_des(f) is first given in the frequency domain, and then the impulse response h[n] is obtained from the frequency domain to the time domain. The sampling rate is Fs, the frequency variable f takes the value [0, Fs / 2], and the discrete-time frequency response satisfies H(f) = sum_{n=0}^{M} h[n] e^{-j 2pi fn / Fs}. For high-frequency enhancement of bright timbre, H_bright(f) is 1 in f1 to f2 and 0 in the rest; for low-frequency enhancement of rich timbre, H_thick(f) is 1 in f3 to f4 and 0 in the rest; for band-stop removal of nasality, H_notch(f) is 0 in f5 to f6 and 1 in the rest.

[0118] The impulse response of an ideal bandpass filter, obtained by inverse Fourier transform, is h_bp[n] = (2f2 / Fs)*sinc(2f2n / Fs) - (2f1 / Fs)*sinc(2f1n / Fs), where sinc(x) = sin(pix) / (pix). An ideal bandstop filter can be obtained by h_notch_ideal[n] = delta[n] - h_bp[n], where delta[n] is the unit impulse. Therefore, h_ideal[n] is an infinitely long sequence.

[0119] Since h_ideal[n] is infinitely long and cannot be directly implemented, the window truncation method is used to obtain finite-length FIR coefficients. A window function w[n] of length N is selected, such as the Hamming window w[n]=0.54 - 0.46 cos(2pi n / (N - 1)) or the Hann window w[n]=0.5 - 0.5 cos(2pi n / (N - 1)). Let h[n]=h_ideal[n]*w[n], and arrange h[n] in a midpoint-symmetric manner to ensure linear phase. The filter order M = N - 1. The larger N is, the narrower the transition band, but the higher the computational complexity.

[0120] When implementing filtering, the convolution y[n]=x[n]*h[n] can be directly performed on the entire speech sequence; to reduce latency and computational complexity, frame processing can also be used. Let the frame length be Nw and the frame shift be H. Usually, Nw is taken as 20 - 40 ms and H is taken as 10 - 20 ms. The speech is framed into x_m[n]=x[n + mH]*w_f[n], n = 0..Nw - 1, where w_f[n] is the analysis window.

[0121] When filtering each frame, it can be implemented in the frequency domain: X_m(k)=FFT{x_m[n]}, Y_m(k)=X_m(k)*H(k), y_m[n]=IFFT{Y_m(k)}, and then the overlap-and-add (OLA) or overlap-save (OLS) method is used to restore the time-domain sequence; amplitude compensation is performed on the synthesis window when needed to ensure that the amplitude in the overlapping region is not distorted.

[0122] To avoid distortion caused by excessive enhancement, the passband gain of h[n] can be normalized (for example, making the average passband gain equal to 1), and 、 an upper limit is set; f1 to f6 can be obtained from the target timbre or speech spectrum statistics, for example, by estimating the formant and energy distribution through the short-time Fourier transform (STFT) and then setting them. The above steps complete a closed-loop process for filter design, window truncation, frame shifting, and splicing restoration.

[0123] Furthermore, if it is used for high-frequency enhanced speech signals with a bright timbre, the preset filter transfer function is:

[0124] ,

[0125] The enhanced signal is:

[0126] ,

[0127] where [f1,f2] is the interval value of the high-frequency enhancement band, f1 is the lower limit, f2 is the upper limit, and f1 < f2, with the unit of Hz; is the high-frequency enhancement coefficient, with a value range of (0, 1), used to control the enhancement intensity; H_bright(f) is the frequency response of the high-frequency enhancement filter, and h_bright[n] is its impulse response in the time domain; x[n] is the original speech sequence, and y[n] is the enhanced output sequence;

[0128] If it is used for the low-frequency enhanced speech signal with a thick timbre, the preset filter transfer function is:

[0129] ,

[0130] The enhanced signal is:

[0131] ,

[0132] where, [f3, f4] is the low-frequency enhancement frequency band interval value, f3 is the lower limit, f4 is the upper limit, and f3 < f4, with the unit of Hz; is the low-frequency enhancement coefficient, used to control the thickness; H_thick(f) is the frequency response of the low-frequency enhancement filter, and h_thick[n] is its impulse response in the time domain; x[n] is the original speech sequence, and y[n] is the enhanced output sequence.

[0133] In a specific embodiment,

[0134] When high-frequency enhancement (bright timbre) is required, such as increasing the energy in the 3 - 8 kHz frequency band, the filter transfer function is:

[0135]

[0136] Then the enhanced signal is:

[0137]

[0138] where controls the enhancement intensity.

[0139] When low-frequency enhancement (thick timbre) is required, such as enhancing the energy in the 80 - 300 Hz frequency band:

[0140]

[0141] Enhancement formula:

[0142] .

[0143] Further, referring to Figure 1 , if it is used to improve clarity and remove nasal sounds from the speech signal, it is achieved through a band-stop filter, and the preset filter transfer function is:

[0144] ,

[0145] The enhanced signal is:

[0146] ,

[0147] where [f5, f6] is the frequency band interval value of the band-stop filter, f5 is the lower limit, f6 is the upper limit, and f5 < f6, with the unit of Hz; H_notch(f) is the frequency response of the band-stop filter, and h_notch[n] is its impulse response in the time domain; x[n] is the original speech sequence, and y[n] is the filtered output sequence.

[0148] In a specific embodiment, when it is necessary to remove nasal sounds (improve clarity), such as removing the resonance in the 250–500 Hz band, through the band-stop filter:

[0149]

[0150] Then:

[0151] .

[0152] In a specific embodiment, for step S600, when splicing sound segments, it is necessary to obtain new parameters re-obtained according to S500, and then add preset silence to the sound segments. Specifically, first read the filter parameters and coefficients obtained from S500, and perform consistent timbre processing on each sounding segment; then calculate the target duration and silence duration of each segment according to d_i', Δ_i', and r_i' obtained from S420, and determine the position and duration of the silence to be inserted or reduced; crop the sounding segments according to the target duration or stretch / compress them through WSOLA to ensure consistent duration; insert zero-amplitude silence or noise floor silence between adjacent sounding segments, and use 5–20 ms cross-fading at the connection to avoid sudden changes; finally, perform overall loudness normalization and peak limiting, and after checking for no obvious breaks, send it to the voice cloning model to generate the final speech.

[0153] Furthermore, the present invention also proposes a voice cloning timbre and rhythm adjustment system based on voice endpoint detection, which is used to implement the voice cloning timbre and rhythm adjustment method based on voice endpoint detection. The voice cloning timbre and rhythm adjustment system based on voice endpoint detection includes, connected in sequence:

[0154] A voice acquisition module, which is used to acquire the voice signal x(t) of the target speaker;

[0155] A preprocessing module, which is used to preprocess the voice signal x(t) collected by the voice acquisition module, including resampling and denoising operations;

[0156] The silence segment detection module distinguishes the preprocessed speech signal into silence segments and vocal segments by combining energy determination, zero-crossing rate analysis and deep endpoint detection model.

[0157] The rhythm adjustment module is used to calculate and adjust rhythm parameters.

[0158] The timbre adjustment module adjusts the timbre of the rhythm-adjusted speech signal based on a preset filter.

[0159] The voice splicing module is used to adjust and splice audio segments.

[0160] Furthermore, refer to Figure 1 The present invention also proposes a computer-readable storage medium storing program instructions thereon, which, when executed by a processor, implement the aforementioned sound cloning timbre and rhythm adjustment method based on speech endpoint detection.

[0161] The innovative aspects that this invention aims to protect:

[0162] 1. This invention utilizes the silence segment information detected by VAD to trim or scale redundant silences in the speech data stream, so as to avoid the silence segments being incorrectly modeled in the sound cloning and synthesis stage, thereby reducing rhythm distortion and unnatural speech problems.

[0163] 2. By adding a "silence specification - rhythm constraint" processing step to the front end of the cloning synthesis, this invention significantly improves the fluency and rhythm of the generated speech while maintaining the consistency of the original timbre features, ensuring that the synthesis result is more in line with human auditory perception habits.

[0164] 3. The method proposed in this invention has universality and scalability, and can be seamlessly embedded into any deep learning-based sound cloning framework. It can improve the naturalness of sound quality and reduce the computational burden caused by invalid segments, providing a more efficient solution for scenarios such as intelligent agents, virtual humans and voice interaction.

[0165] The innovative points that this invention aims to protect are as follows:

[0166] 1. This invention utilizes the silence segment information detected by VAD to trim or scale redundant silences in the speech data stream, so as to avoid the silence segments being incorrectly modeled in the sound cloning and synthesis stage, thereby reducing rhythm distortion and unnatural speech problems.

[0167] 2. By adding a "silence specification - rhythm constraint" processing step to the front end of the cloning synthesis, this invention significantly improves the fluency and rhythm of the generated speech while maintaining the consistency of the original timbre features, ensuring that the synthesis result is more in line with human auditory perception habits.

[0168] 3. The method proposed in this invention has universality and scalability, and can be seamlessly embedded into any deep learning-based sound cloning framework. It can improve the naturalness of sound quality and reduce the computational burden caused by invalid segments, providing a more efficient solution for scenarios such as intelligent agents, virtual humans and voice interaction.

[0169] It should be understood that the method steps in the embodiments of the present invention can be implemented or carried out by computer hardware, a combination of hardware and software, or by computer instructions stored in a non-transitory computer-readable storage medium. The method can use standard programming techniques. Each program can be implemented in a high-level procedural or object-oriented programming language to communicate with the computer system. However, if necessary, the program can be implemented in assembly or machine language. In any case, the language can be a compiled or interpreted language. Furthermore, for this purpose, the program can run on a programmed application-specific integrated circuit (ASIC).

[0170] Furthermore, the procedures described herein may be performed in any suitable order unless otherwise indicated herein or otherwise clearly contradicted by the context. The procedures described herein (or variations and / or combinations thereof) may be executed under the control of one or more computer systems configured with executable instructions, and may be implemented by hardware or a combination thereof as code (e.g., executable instructions, one or more computer programs, or one or more applications) that commonly executes on one or more processors. The computer program comprises a plurality of instructions executable by one or more processors.

[0171] Furthermore, the method can be implemented operatively connected to any suitable type of computing platform, including but not limited to personal computers, minicomputers, mainframes, workstations, networked or distributed computing environments, standalone or integrated computer platforms, or communicating with charged particle tools or other imaging systems, etc. Aspects of the invention can be implemented as machine-readable code stored on a non-transitory storage medium or device, whether removable or integrated into a computing platform, such as a hard disk, optical read and / or write storage medium, RAM, ROM, etc., such that it is readable by a programmable computer, and when the storage medium or device is read by the computer, it can be used to configure and operate the computer to perform the processes described herein. Furthermore, the machine-readable code, or portions thereof, can be transmitted via wired or wireless networks. The invention described herein includes these and other different types of non-transitory computer-readable storage media when such media comprises instructions or programs that implement the steps described above in conjunction with a microprocessor or other data processor. When programmed according to the methods and techniques described in the invention, the invention may also include the computer itself.

[0172] A computer program can be applied to input data to perform the functions described herein, thereby transforming the input data to generate output data stored in non-volatile memory. The output information can also be applied to one or more output devices, such as a display. In a preferred embodiment of the invention, the transformed data represents physical and tangible objects, including specific visual depictions of physical and tangible objects generated on the display.

[0173] The above description is merely a preferred embodiment of the present invention. The present invention is not limited to the above-described embodiments. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention, as long as they achieve the technical effects of the present invention by the same means, should be included within the scope of protection of the present invention. Within the scope of protection of the present invention, the technical solutions and / or implementation methods can have various modifications and variations.

Claims

1. A method for voice cloning timbre and rhythm adjustment based on voice endpoint detection, characterized in that, The described method for adjusting timbre and rhythm of voice cloning based on voice endpoint detection is applied to a system for adjusting timbre and rhythm of voice cloning based on voice endpoint detection. The system for adjusting timbre and rhythm of voice cloning based on voice endpoint detection includes a voice acquisition module, a preprocessing module, a silent segment detection module, a rhythm adjustment module, a timbre adjustment module, and a voice splicing module that are connected in sequence. The method for adjusting timbre and rhythm of voice cloning based on voice endpoint detection includes the following steps: S100. The voice acquisition module acquires the voice signal x(t) of the target speaker. S200. The preprocessing module resamples the voice signal x(t) at a preset sampling rate and performs noise reduction processing through a denoising autoencoder. S300. The silent segment detection module differentiates the preprocessed voice signal into silent segments and sounding segments. S400. The rhythm adjustment module calculates the rhythm parameters of the voice signal and performs proportional adjustment on the rhythm parameters according to a preset rhythm target. S500. The timbre adjustment module adjusts the timbre of the voice signal through a preset filter function without changing the speech rate and rhythm, and obtains the voice signal after timbre adjustment. S600. The voice splicing module adjusts and splices the audio segments, and inputs the voice into a voice cloning model, and finally outputs natural speech.

2. The method for adjusting the timbre and rhythm of sound cloning based on speech endpoint detection according to claim 1, characterized in that, Step S300 includes: S310. Based on a preset time interval, measure the short-time energy Et and zero-crossing rate of the preprocessed voice signal. If the short-time energy Et is greater than a preset threshold and the zero-crossing rate is greater than a preset threshold, it is determined as a sounding segment; otherwise, it is determined as a silent segment. S320. Based on the segmented sounding segments and silent segments, output the boundary sequence B. ， Where i is the segment number of the vocal segment. Let be the start time of the i-th vocal segment. Let be the end time of the i-th vocal segment.

3. The method for adjusting the timbre and rhythm of sound cloning based on speech endpoint detection according to claim 1, characterized in that, Step S400 includes: S410. Calculate the rhythm parameters of the speech signal, wherein the rhythm parameters include the duration of each vocal segment. Adjacent silence intervals and speech rate ratio ; S420. Based on the preset rhythm target, the rhythm parameters are proportionally adjusted to obtain the duration of each vocal segment after adjustment. Adjusted adjacent silence intervals and the adjusted speech rate ratio The preset rhythm objectives include at least adjusting speech rate, adjusting pause time, and adjusting tone intensity; S430, based on the adjusted duration of each vocal segment Adjusted adjacent silence intervals and the adjusted speech rate ratio The time scale is modified by stretching or compressing the length of silent and vocal segments of a speech signal using a waveform similarity superposition algorithm.

4. The method for adjusting the timbre and rhythm of sound cloning based on speech endpoint detection according to claim 3, characterized in that, In step S410, Duration of each vocal segment for: ， Adjacent silence intervals for: ， speech rate ratio for: ， Where i is the segment number of the vocal segment. Let be the start time of the i-th vocal segment. Let be the end time of the i-th vocal segment. The number of characters in the corresponding text clause.

5. The method for adjusting the timbre and rhythm of sound cloning based on speech endpoint detection according to claim 1, characterized in that, In step S420, Adjusted duration of each vocal segment for: ， Adjusted adjacent silence interval for: ， Adjusted speech rate ratio for: ， in, Duration of each vocal segment Adjustment coefficient, Adjacent silence intervals Adjustment coefficient, The number of characters in the corresponding text clause.

6. The method for adjusting the timbre and rhythm of sound cloning based on speech endpoint detection according to claim 1, characterized in that, In step S500, the voice signal after timbre adjustment Represented as: ， Where, h[n] is the impulse response of the FIR filter, x[n] is the discrete sampling sequence of the input voice signal, y[n] is the filtered output sequence, n is the discrete time index, k is the convolution summation index, M is the filter order, and the symbol "*" represents the discrete convolution operation.

7. According to the method for adjusting timbre and rhythm of voice cloning based on voice endpoint detection as claimed in claim 6, wherein If it is a high-frequency enhanced voice signal for bright timbre, the preset filter transfer function is: ， The enhanced signal is: ， Among them, [f1, f2] is the high-frequency enhancement frequency band interval value, where f1 is the lower limit and f2 is the upper limit, and f1 < f2, with the unit of Hz; α is the high-frequency enhancement coefficient, and its value range is (0, 1), which is used to control the enhancement intensity; H_bright(f) is the frequency response of the high-frequency enhancement filter, and h_bright[n] is its time-domain impulse response; x[n] is the original speech sequence, and y[n] is the enhanced output sequence; If it is a low-frequency enhanced voice signal for thick timbre, the preset filter transfer function is: ， The enhanced signal is: ， Among them, [f3, f4] is the interval value of the low-frequency enhancement frequency band, f3 is the lower limit, f4 is the upper limit, and f3 < f4, with the unit of Hz; is the low-frequency enhancement coefficient, which is used to control the thickness; H_thick(f) is the frequency response of the low-frequency enhancement filter, h_thick[n] is its impulse response in the time domain; x[n] is the original speech sequence, and y[n] is the enhanced output sequence.

8. According to the method for adjusting timbre and rhythm of voice cloning based on voice endpoint detection as claimed in claim 6, wherein If it is used to improve clarity and perform nasal removal processing on the voice signal, it is implemented through a band-stop filter, and the preset filter transfer function is: ， The enhanced signal is: ， Where, [f5, f6] is the frequency band interval value of the band-stop filter, f5 is the lower limit, f6 is the upper limit, and f5 < f6, with the unit of Hz; H_notch(f) is the frequency response of the band-stop filter, h_notch[n] is its time-domain impulse response; x[n] is the original voice sequence, and y[n] is the filtered output sequence.

9. A voice endpoint detection-based sound cloning timbre and rhythm adjustment system, used to implement the voice endpoint detection-based sound cloning timbre and rhythm adjustment method as described in any one of claims 1 to 8, characterized in that, The voice endpoint detection-based sound cloning timbre and rhythm adjustment system comprises, in sequence: The voice acquisition module is used to acquire the voice signal x(t) of the target speaker; The preprocessing module is used to preprocess the speech signal x(t) acquired by the speech acquisition module, including resampling and noise reduction operations; The silence segment detection module distinguishes the preprocessed speech signal into silence segments and vocal segments by combining energy determination, zero-crossing rate analysis and deep endpoint detection model. The rhythm adjustment module is used to calculate and adjust rhythm parameters. The timbre adjustment module adjusts the timbre of the rhythm-adjusted speech signal based on a preset filter. The voice splicing module is used to adjust and splice audio segments.

10. A computer-readable storage medium, characterized in that, It stores program instructions, which, when executed by a processor, implement the sound cloning timbre and rhythm adjustment method based on voice endpoint detection as described in any one of claims 1 to 8.

Images