Voice conversion method and system for protecting timbre

Abstract

The invention discloses a voice conversion method for protecting timbre, and relates to the technical field of voice signal processing. The method comprises a timbre feature extraction module, a timbre keeping voice conversion module, a timbre consistency evaluation module and an output reconstruction module. The method comprises the following steps: firstly, acquiring and analyzing a source voice signal through a timbre feature extraction module, and extracting core feature parameters representing the timbre of a speaker; and then, the tone-keeping voice conversion module uses the trained deep learning model to take the extracted tone characteristics as condition input in the voice content conversion process, so that the voice containing the target content and keeping the original tone characteristics is generated. And then, the tone consistency evaluation module screens and evaluates the conversion results by calculating the tone similarity and setting a credibility threshold value, so that the tone fidelity is ensured. And finally, the output reconstruction module carries out synthesis and post-processing on the qualified voice to generate the final high-quality voice.

Description

A speech conversion method and system that preserves timbre Technical Field

[0001] This invention relates to the field of speech signal processing and speech synthesis technology, specifically to a speech conversion method and system that preserves timbre. Background Technology

[0002] Speech conversion technology is a key technology for altering speech content and has wide applications in media entertainment, privacy protection, and personalized speech synthesis. With advancements in deep learning, the quality and naturalness of speech conversion have significantly improved. However, most traditional or existing speech conversion methods inevitably alter or lose the unique timbre characteristics of the original speaker when converting speech content (such as text, accents, and speech), resulting in inconsistencies between the synthesized speech and the original speaker's voice, affecting the naturalness, realism, and continuity of speaker identity. Especially in scenarios requiring speaker identity protection or high timbre consistency (such as virtual idols, audiobooks, and secure communications), existing methods struggle to effectively preserve and protect the original timbre characteristics while efficiently converting content. Therefore, there is an urgent need for a technical solution that can actively protect and maintain the original timbre during speech conversion. Summary of the Invention

[0003] The purpose of this invention is to overcome the shortcomings of existing technologies and provide a speech conversion method and system that preserves timbre. It introduces a structured timbre feature extraction and protection mechanism, combined with a deep learning conversion model and timbre consistency evaluation feedback, aiming to achieve high-quality speech content conversion while ensuring maximum preservation of original timbre features. This system can accurately extract timbre features, generate target speech through intelligent conversion logic, and provide a rigorous evaluation and feedback mechanism to ensure the controllability and reliability of the conversion process.

[0004] To address the aforementioned technical problems, this invention provides the following technical solution: a voice conversion system that preserves timbre. This system includes: a timbre feature extraction module, used to acquire source speech signals, convert the acquired signals into digital signals, and perform preprocessing to extract feature parameters characterizing timbre; a timbre-preserving voice conversion module, receiving the feature parameters output by the timbre feature extraction module, using deep learning technology to construct and execute a timbre-preserving voice conversion model, converting the source speech content into target content while preserving the original timbre features; a timbre consistency evaluation module, which calculates the timbre similarity between the converted speech data output by the voice conversion module and the source speech using a preset evaluation algorithm, and performs credibility screening on the conversion results; and an output reconstruction module, responsible for synthesizing and post-processing the evaluated speech data to generate the final speech signal. The module also provides a real-time feedback mechanism, allowing operators or the system to understand the conversion progress and timbre fidelity results through an interface or API.

[0005] Furthermore, the timbre feature extraction module includes a microphone or audio interface device for acquiring speech, used to receive external speech input, acquire speech signals at certain time intervals, and then convert the acquired signals into digital signals and perform pre-emphasis and frame segmentation processing.

[0006] Furthermore, the specific steps of the timbre feature extraction module include preprocessing: preprocessing the digitized speech signal, using an adaptive filtering algorithm to filter the acquired speech data to remove environmental noise, thereby improving the accuracy of subsequent feature extraction. When performing noise suppression, an adaptive filtering algorithm is used, particularly the Least Mean Square (LMS) algorithm, to reduce or eliminate noise coexisting with the speech signal. The LMS algorithm minimizes the error between the desired signal and the filter output by adjusting the filter coefficients. The core calculation formula of the LMS algorithm is as follows: The filter output calculation formula is... ,in It is the order of the filter (i.e., the length of the weight vector). It is the weight vector at time t. The i-th element, The error signal is calculated as the i-th element of the reference input signal vector at time n. Weight vector update: .

[0007] Furthermore, the specific steps of the timbre feature extraction module also include feature extraction: extracting features representing timbre from the preprocessed signal, using Mel-frequency cepstral coefficients (MFCC), fundamental frequency, and spectral envelope. These features will be used for subsequent conversion and evaluation. The calculation steps for feature extraction using the MFCC algorithm are as follows: (1) Frame division and pre-emphasis: divide the speech signal into frames and pre-emphasize each frame. The purpose of pre-emphasis is to enhance the energy of the high-frequency part to compensate for the loss of the high-frequency part of the speech signal during transmission. The formula for pre-emphasis is: ,in It is the original speech signal. It is the signal after pre-emphasis. It is a constant, usually set to 0.97; (2) Fast Fourier Transform (FFT): Perform Fast Fourier Transform (FFT) on each frame of the pre-emphasized signal to convert it from the time domain to the frequency domain and obtain the spectrum; (3) Calculate the Mel frequency: Convert the result of FFT to the Mel frequency domain. The conversion relationship between Mel frequency and linear frequency is: ,in, (4) Apply Mel filter bank: Use a set of Mel filters to filter the spectrum. Each filter corresponds to a Mel frequency band. The output of the filter is the weighted sum of the original spectrum in the frequency band. (5) Take the logarithm: Take the logarithm of the output of the Mel filter to further compress the dynamic range. (6) Discrete cosine transform (DCT): Apply Discrete cosine transform (DCT) to the output of the Mel filter after taking the logarithm to obtain the MFCC coefficients. Usually, only the first few coefficients of the DCT are taken as MFCC features.

[0008] Furthermore, the specific steps of the timbre-preserving speech conversion module also include conversion model training and inference. Conversion model training: using a deep neural network (DNN) algorithm, the conversion model is trained using a speech dataset containing timbre labels, enabling the model to learn the decoupling and recombination relationship between speech content and timbre features. The calculation of the DNN algorithm mainly includes forward propagation and backward propagation. In forward propagation, from the input layer to the hidden layer: z = Wx + b, where z is the input of the hidden layer, W is the weight matrix, x is the input vector, and b is the bias vector. From the hidden layer to the output layer: the above formula is used again, only the weight matrix, input vector, and bias vector are different. Backpropagation adjusts the weights and biases according to the output error. The activation function is usually the ReLU or Sigmoid function, used to introduce non-linearity. Timbre-preserving inference: during the conversion stage, the source speech features to be converted and the target content prompts are input into the trained neural network model. The model will inject the extracted timbre features as conditions, and output an intermediate representation that meets the target content requirements while preserving the timbre features of the source speech.

[0009] Furthermore, the specific steps of the timbre consistency evaluation module are as follows: (1) Receiving evaluation data: The timbre consistency evaluation module receives the converted speech data from the voice conversion module for timbre preservation and the timbre feature data of the source speech from the timbre feature extraction module. (2) Feature comparison and similarity calculation: Using the timbre feature comparison algorithm, calculate the timbre feature vectors of the converted speech and the source speech to obtain the timbre similarity score. (3) Confidence screening: According to the preset confidence threshold, compare and screen the similarity scores to determine whether the conversion has successfully preserved the timbre. (4) Result feedback and processing: Feed back the evaluation results to the output reconstruction module or the conversion module. For the results that do not pass the screening, re-conversion or targeted optimization can be triggered.

[0010] Furthermore, the specific steps of the output reconstruction module usually involve the final generation process of the speech signal. The following is an overview of the steps: (1) Data reception and initialization: The output reconstruction module first receives the evaluated intermediate speech representation data and loads necessary speech synthesis components such as vocoders. (2) Speech waveform synthesis: Use the vocoder to synthesize the intermediate representation into a time-domain speech waveform signal. (3) Post-processing and enhancement: Perform post-processing such as denoising, smoothing, and volume normalization on the synthesized speech waveform to improve the auditory quality. (4) Result output and feedback: Output the processed final speech signal as an audio file or streaming data, and at the same time, feedback the synthesis status and quality information to the system interface. (5) Continuous monitoring and recording: The output reconstruction module continuously records the quality metrics and parameters of the output speech for system performance analysis and optimization.

[0011] Furthermore, the timbre consistency evaluation module involves a threshold comparison algorithm to analyze and compare the calculated timbre similarity data. By setting the timbre similarity threshold, this algorithm compares the real-time calculated similarity score with these thresholds to determine whether the voice conversion process has successfully preserved the timbre or whether there is timbre distortion.

[0012] Furthermore, the formula of the threshold comparison algorithm can be expressed as: If S < T, then trigger the warning or reprocessing process; where S represents the calculated timbre similarity score and T represents the preset confidence threshold.

[0013] Furthermore, a voice conversion method for preserving timbre is also provided, which includes the following steps: (1) Timbre feature extraction: collect source speech signal data, convert the collected signal into a digital signal and preprocess it to extract timbre feature parameters; (2) Timbre-preserving speech conversion: install and configure a timbre-preserving speech conversion module, send the extracted timbre feature parameters and target content prompts to the module, use deep learning technology to perform timbre-preserving speech conversion and generate an intermediate representation; (3) Timbre consistency evaluation: calculate the timbre similarity and perform credibility screening based on the converted speech data and source speech features through an evaluation algorithm; (4) Output reconstruction: perform speech synthesis and post-processing on the intermediate representation that has passed the evaluation to generate and output the final speech signal.

[0014] Compared with existing technologies, this voice-interactive machine tool control system and method have the following beneficial effects: First, by introducing a structured timbre feature extraction and protection mechanism and combining it with a deep learning conversion model, this invention can achieve high-quality conversion of speech content while effectively protecting the original timbre features. The system provides rigorous feedback and screening through a timbre consistency evaluation module, ensuring the controllability of the conversion process and the reliability of the output results. This solves the problem of timbre loss in traditional conversions and achieves an efficient balance between conversion quality and timbre fidelity.

[0015] Second, this invention accurately extracts timbre features using algorithms such as MFCC in the timbre feature extraction module. The timbre-preserving speech conversion module employs deep learning technology to learn the decoupling and recombination relationship between timbre features and speech content, preserving timbre during the conversion process through a conditional injection mechanism. The timbre consistency evaluation module uses a threshold comparison algorithm to filter the conversion results, ensuring that only speech with high timbre fidelity enters the final output stage. This closed-loop process significantly improves the practicality and naturalness of speech conversion in scenarios such as identity protection and personalized synthesis.

[0016] Other advantages, objectives and features of the invention will be set forth in part in the description which follows, and in part will be apparent to those skilled in the art from the following examination or study, or may be learned from the practice of the invention. Attached Figure Description

[0017] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the accompanying drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are merely some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without any creative effort.

[0018] Figure 1 is a structural diagram of a voice conversion system that protects the tone; Figure 2 is a flowchart of a voice conversion method that protects the tone. Detailed Implementation

[0019] The technical solutions in the embodiments of the present invention will be clearly and completely described below. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of the present invention.

[0020] Example 1: A voice conversion system for preserving timbre. This system includes a timbre feature extraction module for acquiring source speech signals, converting the acquired signals into digital signals, and performing preprocessing to extract feature parameters characterizing timbre. The timbre feature extraction module includes a microphone or audio interface device for acquiring speech, receiving external speech input, acquiring speech signals at certain time intervals, converting the acquired signals into digital signals, and performing basic processing such as pre-emphasis and framing to prepare for subsequent feature extraction.

[0021] A voice-preserving speech conversion system is disclosed. This system includes a voice-preserving speech conversion module that receives feature parameters output by a voice feature extraction module, utilizes deep learning technology to construct and execute a voice-preserving speech conversion model, converting source speech content into target content while preserving the original voice features. This module possesses strong learning and generalization capabilities, enabling it to learn and adapt to the complex mapping relationships between the voice features of different speakers and various conversion targets (such as different texts and different emotions), thereby improving the flexibility of conversion and the accuracy of voice preservation. Furthermore, the module is robust to input feature noise, ensuring stable operation under various speech quality conditions.

[0022] The specific steps of the timbre feature extraction module include preprocessing: preprocessing the digitized speech signal, using an adaptive filtering algorithm to filter the acquired speech data to remove environmental noise, thereby improving the accuracy of subsequent feature extraction; and using an adaptive filtering algorithm, particularly the Least Mean Square (LMS) algorithm, to reduce or eliminate noise coexisting with the speech signal during noise suppression. The LMS algorithm minimizes the error between the desired signal and the filter output by adjusting the filter coefficients, thus effectively improving the clarity of the speech signal.

[0023] The timbre feature extraction module also includes feature extraction: extracting features characterizing timbre from the preprocessed signal, such as Mel-frequency cepstral coefficients (MFCC), fundamental frequency, and spectral envelope. These features will be used for subsequent conversion and evaluation. The MFCC algorithm is used for feature extraction. MFCC is a commonly used feature extraction method that simulates the characteristics of human hearing and has good discriminative power for different timbre features. Its calculation steps are as described above, involving frame segmentation, pre-emphasis, FFT, Mel filtering, logarithmic calculation, and DCT, ultimately yielding MFCC coefficients that effectively characterize timbre.

[0024] The specific steps of the timbre-preserving speech conversion module also include conversion model training and inference. Conversion model training: Using a deep neural network (DNN) algorithm, the conversion model is trained on a speech dataset containing timbre labels, enabling the model to learn the decoupling and recombination relationship between speech content and timbre features. During training, the predicted output is calculated through forward propagation, and the network weights and biases are adjusted based on the error between the predicted output and the true labels through backpropagation, gradually optimizing the model parameters. Timbre-preserving inference: In the conversion stage, the source speech features to be converted and target content cues (such as the embedded representation of the target text) are input into the trained neural network model. The model's internal mechanism injects the extracted timbre features as conditional signals into the generation process, thereby guiding the model output to both meet the target content requirements and preserve the intermediate acoustic representation of the source speech timbre features (such as a Mel spectrogram) at the spectral and prosodic levels.

[0025] Example 2: A speech conversion system that preserves timbre. This system includes a timbre consistency evaluation module. Based on the converted speech data output by the speech conversion module, the system calculates the timbre similarity between the converted speech and the source speech using a preset evaluation algorithm, and performs reliability screening on the conversion results. This module can accurately quantify the degree of timbre preservation, promptly detect and filter conversion results with timbre distortion, ensuring the quality and reliability of the final output speech. Simultaneously, this module also has a feedback function, enabling the evaluation results to guide system optimization.

[0026] The specific steps of the timbre consistency evaluation module are as follows: First, the module receives the converted speech data (or its features) from the timbre-preserving speech conversion module and the timbre feature data of the source speech from the timbre feature extraction module. Next, using a timbre feature comparison algorithm (such as a cosine similarity-based or specially trained timbre embedding model), the timbre feature vectors of the converted speech and the source speech are calculated to obtain a quantified timbre similarity score. Then, based on a preset confidence threshold, this score is compared and filtered. If the score is higher than or equal to the threshold, timbre preservation is considered successful; if the score is lower than the threshold, timbre preservation is deemed unqualified. Finally, the module feeds back the evaluation results (including the score and a pass / fail indicator) to the output reconstruction module or conversion module. For unsuccessful results, the system can be triggered to retry the conversion or transferred to the post-processing stage for targeted timbre correction.

[0027] During the evaluation process, the timbre consistency evaluation module also performs logical verification on the instructions to ensure the correctness and completeness of the evaluation process. If the evaluation data is abnormal or feature extraction fails, the module may issue an error message according to preset rules or try to use alternative features for calculation to improve the robustness of the evaluation system.

[0028] Example 3 describes a voice conversion system that preserves timbre. This system includes an output reconstruction module responsible for synthesizing and post-processing the evaluated voice data to generate the final voice signal. This module is designed to be highly efficient, reliable, and easy to operate, enabling the system to quickly generate high-quality final voice. The module also provides a real-time feedback mechanism, allowing operators or the system to monitor the conversion progress and timbre fidelity results through an interface or API.

[0029] The specific steps of the output reconstruction module typically involve the final generation process of the speech signal. First, the output reconstruction module initializes, loading necessary speech synthesis components such as a vocoder (e.g., WaveNet, HiFi-GAN). Next, the module receives intermediate speech representation data (e.g., Mel spectrograms) that has passed timbre consistency evaluation. Then, the vocoder synthesizes the intermediate representation into a time-domain speech waveform signal. Afterward, the synthesized raw waveform undergoes post-processing and enhancement, such as applying denoising algorithms to eliminate subtle noise introduced during synthesis, and performing dynamic range compression or volume normalization to improve auditory consistency. Finally, the processed, high-quality speech signal is output as an audio file in formats such as WAV or MP3, or pushed as streaming data to the caller. Simultaneously, the module can record key parameters of this synthesis (such as time consumption and synthesis quality score) in a log for system performance monitoring and optimization.

[0030] A voice conversion system that protects timbre ensures controllable output quality through a timbre consistency evaluation module. This module involves a threshold comparison algorithm to analyze and compare calculated timbre similarity data. The algorithm sets a series of timbre similarity thresholds and compares the real-time calculated score with these thresholds to determine whether the voice conversion process successfully preserves the timbre or whether timbre distortion has occurred. Specifically, the security monitoring module first determines the key evaluation parameter of timbre similarity. Then, based on the quality requirements of the application scenario, historical conversion data, and experimental verification results, it sets one or more thresholds for this parameter. These thresholds typically represent the lower limit of acceptable timbre fidelity. Once the evaluation module starts running, it calculates the timbre similarity score for each conversion in real time and compares this score with the preset thresholds. If the score is lower than its corresponding threshold, the evaluation module considers the timbre fidelity of this conversion insufficient, thus triggering corresponding processing mechanisms (such as marking it as a failure, requesting a retry, or initiating an optimization process).

[0031] Example 4 describes a speech conversion method that preserves timbre. The specific control steps of this method are as follows: A timbre feature extraction module acquires the source speech signal using a microphone or audio file reading interface and converts the acquired signal into a digital signal. An adaptive filtering algorithm is used to filter and remove noise from the acquired speech data to improve the accuracy of subsequent feature extraction. Feature extraction algorithms such as MFCC are used to analyze the preprocessed speech signal and extract feature parameter vectors representing timbre.

[0032] The timbre-preserving speech conversion module receives feature parameters and target content prompts (such as text) output by the timbre feature extraction module. It performs format standardization and alignment on the received data. Based on a pre-defined deep learning model architecture and mapping relationship, it injects the source timbre features as conditions into the model to generate an intermediate acoustic representation that conforms to the target content and preserves the source timbre. The timbre consistency evaluation module receives the converted intermediate representation and extracts its timbre features again. It compares the extracted features with the source speech features and calculates a timbre similarity score. If the score is lower than a pre-defined threshold, an error handling mechanism is immediately activated, such as discarding the result, logging it, and potentially triggering a new round of conversion.

[0033] The output reconstruction module synthesizes the evaluated intermediate representation into a time-domain speech waveform using a vocoder. Post-processing, such as denoising and smoothing, is then applied to the synthesized waveform to improve the auditory quality of the output speech. Finally, a high-quality target speech file is output and saved, completing the voice-preserving speech conversion process. Operators can view the conversion progress, voice fidelity score, and final result through a monitoring interface.

[0034] Example 5 This example provides a specific and feasible method for constructing and operating a voice conversion system that preserves the timbre, to further illustrate the present invention. This example uses the conversion of speaker A's speech content into target text while preserving A's timbre as an example.

[0035] A timbre encoder pre-trained on a large-scale speaker verification dataset (such as VoxCeleb) is used. This encoder employs a deep neural network architecture based on ResNet34 or TDNN, trained with a generalized end-to-end (GE2E) loss function to learn a timbre embedding space capable of distinguishing different speakers. The encoder takes an 80-dimensional Mel spectrogram as input and outputs a 256-dimensional timbre embedding vector (SpeakerEmbedding, spk_emb). This timbre encoder has its parameters fixed in subsequent processes for extracting timbre features.

[0036] Then, an encoder-decoder model based on the attention mechanism is constructed as a speech conversion model that preserves timbre. The specific structure is as follows: Encoder: It consists of 3 layers of convolutional neural network (CNN) and downsampling layer, which is responsible for encoding the Mel spectrogram (denoted as mel_src) of the source speech into the hidden state sequence H_src.

[0037] The timbre conditional injection module: The source speech timbre embedding vector spk_emb extracted by the pre-trained timbre encoder is projected through a fully connected layer (LinearLayer), and then concatenated with the attention context vector at each step of the decoder as additional input conditions for the decoder.

[0038] Decoder: A single-layer recurrent neural network (RNN) based on LSTM or GRU. At each step t, the decoder receives the output of the previous step, the context vector c_t generated by the encoder's hidden state H_src through the attention mechanism, and a vector concatenated with timbre conditions, and predicts a frame mel_tgt_t of the target Mel spectrogram.

[0039] Training objective and loss function: Training is performed using a multi-task loss function that includes tone preservation constraints. in: The L1 spectrum reconstruction loss is calculated by determining the difference between the predicted mel_tgt and the true target spectrum.

[0040] To address timbre adversarial loss, a timbre classification discriminator is introduced to attempt to distinguish whether the generated speech belongs to the source speaker or the target speaker. The generator (transformation model) aims to "deceive" the discriminator into believing that the generated speech's timbre belongs to the source speaker. This forces the generator to preserve the source timbre when transforming the content. λ is a balancing hyperparameter, for example, set to 0.1.

[0041] Training data: Use a speech dataset that contains parallel corpora (the same speaker speaking different texts) or non-parallel corpora (such as the VCTK dataset) for training.

[0042] (3) Vocoder selection: HiFi-GAN, a publicly available and mature neural vocoder, was selected. This vocoder has been pre-trained on a large amount of speech data and can reconstruct the Mel spectrogram into a time-domain waveform with high quality.

[0043] Then, given a voice audio_src from source speaker A and a target text text_tgt, the system operates according to the following steps: Step 1: Extract timbre features; preprocess audio_src (pre-emphasis, framing, Hamming windowing).

[0044] Calculate its 80-dimensional Mel spectrogram mel_src.

[0045] Inputting mel_src into the pre-trained timbre encoder yields a 256-dimensional timbre embedding vector spk_emb_A.

[0046] Step 2: Voice conversion with timbre preservation; input mel_src and spk_emb_A into the trained conversion model.

[0047] Meanwhile, a text front-end processor (such as the text encoder in Tacotron2) is used to convert text_tgt into a phoneme sequence or character embedding sequence as a start signal for the decoder.

[0048] The conversion model runs, outputting the predicted target Mel spectrogram mel_tgt_pred. During this process, spk_emb_A continuously influences the decoding process through a conditional injection module, ensuring that the timbre characteristics of the generated spectrum match those of A.

[0049] Step 3: Timbre consistency evaluation; input the generated mel_tgt_pred back into the same pre-trained timbre encoder to obtain its timbre embedding vector spk_emb_gen.

[0050] Calculate the cosine similarity between spk_emb_A and spk_emb_gen as the timbre similarity score S: Compare S with the preset confidence threshold T=0.85.

[0051] If S≥0.85, the tone quality is deemed acceptable, and proceed to step four.

[0052] If S < 0.85, the result is deemed unqualified. The system will mark the generated result as low quality and can choose from the following strategies: a) discard the result and log it; b) fine-tune it using mel_tgt_pred and spk_emb_A through an additional timbre post-processing network and then evaluate it again.

[0053] Input the qualified mel_tgt_pred into the HiFi-GAN vocoder.

[0054] HiFi-GAN converts the Mel spectrogram into a time-domain speech waveform, audio_tgt.

[0055] Perform simple post-processing on audio_tgt, such as peak normalization, to ensure consistent output volume.

[0056] The system outputs the final audio file final_audio.wav and simultaneously displays the timbre similarity score S and the status "success" on the system interface.

[0057] To demonstrate the effectiveness of the system in this embodiment, a comparative experiment was conducted. Using the VCTK dataset, five randomly selected speech samples from speaker A were used as sources and converted into the same target text. The comparison method was the traditional CycleGAN-VC (a non-parallel corpus speech conversion method). Evaluation metrics included: an objective metric: timbre embedding cosine similarity (SV-Sim, higher is better).

[0058] Subjective metric: Mean Opinion Score (MOS), where testers are invited to rate the "timbre similarity" of the generated speech on a scale of 1 to 5.

[0059]

[0060] Experimental results show that the system of the present invention is significantly better than the traditional method in both objective and subjective timbre fidelity indicators, and effectively solves the problem of timbre loss in speech conversion.

[0061] In summary, by introducing a structured process of timbre feature extraction, protective conversion, rigorous evaluation, and reconstruction output, this invention can effectively protect the original timbre features while enabling flexible conversion of speech content, thereby enhancing the practical value and user experience of speech conversion technology in fields such as identity protection and personalized media creation.

[0062] It will be apparent to those skilled in the art that the present invention is not limited to the details of the exemplary embodiments described above, and that the invention can be implemented in other specific forms without departing from its spirit or essential characteristics. Therefore, the embodiments should be considered in all respects as exemplary and non-limiting, and the scope of the invention is defined by the appended claims rather than the foregoing description. Thus, all variations falling within the meaning and scope of the same elements of the claims are intended to be included within the present invention. No reference numerals in the claims should be construed as limiting the scope of the claims.

Claims

1. A voice conversion system that preserves timbre, characterized in that, The system includes: a timbre feature extraction module, used to acquire source speech signals, convert the acquired signals into digital signals and preprocess them to extract feature parameters characterizing timbre; a timbre-preserving speech conversion module, which receives the feature parameters output by the timbre feature extraction module, uses deep learning technology to construct and execute a timbre-preserving speech conversion model, converts the source speech content into target content, and preserves the original timbre features in the process; a timbre consistency evaluation module, which calculates the timbre similarity between the converted speech data output by the speech conversion module and the source speech using a preset evaluation algorithm, and performs credibility screening on the conversion results; and an output reconstruction module, responsible for synthesizing and post-processing the evaluated speech data to generate the final speech signal; the modules also provide a real-time feedback mechanism, allowing operators or the system to understand the conversion progress and timbre fidelity results through an interface or API.

2. The voice conversion system for protecting timbre according to claim 1, characterized in that, The timbre feature extraction module includes a microphone or audio interface device for acquiring speech, used to receive external speech input, acquire speech signals at certain time intervals, and then convert the acquired signals into digital signals and perform pre-emphasis and frame segmentation processing.

3. The voice conversion system for protecting timbre according to claim 1, characterized in that, The specific steps of the timbre feature extraction module include preprocessing: preprocessing the digitized speech signal, using an adaptive filtering algorithm to filter the acquired speech data to remove environmental noise, thereby improving the accuracy of subsequent feature extraction. When performing noise suppression, an adaptive filtering algorithm is used, particularly the Least Mean Square (LMS) algorithm, to reduce or eliminate noise present alongside the speech signal. The LMS algorithm minimizes the error between the desired signal and the filter output by adjusting the filter coefficients. The core calculation formula of the LMS algorithm is as follows: The filter output calculation formula is... ,in It is the order of the filter (i.e., the length of the weight vector). It is the weight vector at time t. The i-th element, The error signal is calculated as the i-th element of the reference input signal vector at time n. Weight vector update: 。 4. A voice conversion system for protecting timbre according to claim 3, characterized in that, The specific steps of the timbre feature extraction module are as follows: Feature extraction includes: extracting features representing timbre from the preprocessed signal using Mel-frequency cepstral coefficients (MFCC), fundamental frequency, and spectral envelope. These features will be used for subsequent conversion and evaluation. The calculation steps for feature extraction using the MFCC algorithm are as follows: (1) Framing and pre-emphasis: the speech signal is framed, and each frame is pre-emphasized. The purpose of pre-emphasis is to enhance the energy of the high-frequency part to compensate for the loss of the high-frequency part of the speech signal during transmission. The formula for pre-emphasis is: ,in It is the original speech signal. It is the signal after pre-emphasis. It is a constant, usually set to 0.97; (2) Fast Fourier Transform (FFT): Perform Fast Fourier Transform (FFT) on each frame of the pre-emphasized signal to convert it from the time domain to the frequency domain and obtain the spectrum; (3) Calculate the Mel frequency: Convert the result of FFT to the Mel frequency domain. The conversion relationship between Mel frequency and linear frequency is: ,in, (4) Apply Mel filter bank: Use a set of Mel filters to filter the spectrum. Each filter corresponds to a Mel frequency band. The output of the filter is the weighted sum of the original spectrum in the frequency band. (5) Take the logarithm: Take the logarithm of the output of the Mel filter to further compress the dynamic range. (6) Discrete cosine transform (DCT): Apply Discrete cosine transform (DCT) to the output of the Mel filter after taking the logarithm to obtain the MFCC coefficients. Usually, only the first few coefficients of the DCT are taken as MFCC features.

5. A voice conversion system for protecting timbre according to claim 4, characterized in that, The specific steps of the timbre-preserving speech conversion module also include conversion model training and inference. Conversion model training: Using a deep neural network (DNN) algorithm, the conversion model is trained on a speech dataset containing timbre labels, enabling the model to learn the decoupling and recombination relationship between speech content and timbre features. The calculation of the DNN algorithm mainly includes forward propagation and backward propagation. In forward propagation, from the input layer to the hidden layer: z = Wx + b, where z is the input of the hidden layer, W is the weight matrix, x is the input vector, and b is the bias vector. From the hidden layer to the output layer: the above formula is used, but the weight matrix, input vector, and bias vector are different. Backpropagation adjusts the weights and biases according to the output error. The activation function is usually ReLU or Sigmoid to introduce non-linearity. Timbre-preserving inference: In the conversion stage, the source speech features to be converted and the target content prompts are input into the trained neural network model. The model will inject the extracted timbre features as conditions and output an intermediate representation that meets the target content requirements while preserving the timbre features of the source speech.

6. A voice conversion system for protecting timbre according to claim 1, characterized in that, The specific steps of the timbre consistency evaluation module are as follows: (1) Receive evaluation data: The timbre consistency evaluation module receives the converted speech data from the timbre-preserving speech conversion module and receives the timbre feature data of the source speech from the timbre feature extraction module; (2) Feature comparison and similarity calculation: Using the timbre feature comparison algorithm, the timbre feature vectors of the converted speech and the source speech are calculated to obtain the timbre similarity score; (3) Credibility screening: According to the preset credibility threshold, the similarity scores are compared and screened to determine whether the conversion has successfully preserved the timbre; (4) Result feedback and processing: The evaluation results are fed back to the output reconstruction module or the conversion module. For results that fail the screening, re-conversion or targeted optimization can be triggered.

7. A voice conversion system for protecting timbre according to claim 1, characterized in that, The specific steps of the output reconstruction module usually involve the final generation process of the speech signal. The following is an overview of the steps: (1) Data reception and initialization: The output reconstruction module first receives the speech intermediate representation data that has passed the evaluation and loads the necessary speech synthesis components such as the vocoder; (2) Speech waveform synthesis: The intermediate representation is synthesized into a time-domain speech waveform signal using the vocoder; (3) Post-processing and enhancement: The synthesized speech waveform is post-processed such as denoising, smoothing, and volume normalization to improve the auditory quality; (4) Result output and feedback: The final processed speech signal is output as an audio file or streaming data, and the synthesis status and quality information can be fed back to the system interface; (5) Continuous monitoring and recording: The output reconstruction module continuously records the quality indicators and parameters of the output speech for system performance analysis and optimization.

8. A voice conversion system for protecting timbre according to claim 1, characterized in that, The timbre consistency evaluation module involves a threshold comparison algorithm to analyze and compare the calculated timbre similarity data. This algorithm sets timbre similarity thresholds and compares the real-time calculated similarity scores with these thresholds to determine whether the speech conversion process has successfully maintained the timbre or whether timbre distortion has occurred.

9. A voice conversion system for protecting timbre according to claim 8, characterized in that, The formula of the threshold comparison algorithm can be expressed as: if S < T, then trigger the warning or reprocessing process; where S represents the calculated timbre similarity score, and T represents the preset credibility threshold.

10. A speech conversion method that preserves timbre, characterized in that, This method includes the following steps: (1) Timbre feature extraction: Collect the source speech signal data, convert the collected signal into a digital signal and perform preprocessing, and extract the timbre feature parameters; (2) Timbre-preserving voice conversion: Install and configure the timbre-preserving voice conversion module, input the extracted timbre feature parameters and the target content prompt into this module, and use deep learning technology to perform timbre-preserving voice conversion to generate an intermediate representation; (3) Timbre consistency evaluation: According to the converted speech data and the source speech features, calculate the timbre similarity through the evaluation algorithm and perform credibility screening; (4) Output reconstruction: Perform speech synthesis and postprocessing on the intermediate representation that has passed the evaluation, and generate and output the final speech signal.

Images