Sound replication method and related apparatus

Abstract

The application provides a sound replication method and related device, and relates to the technical field of sound processing. The reference audio is subjected to audio verification to obtain first audio, the first audio is subjected to a speech enhancement operation to achieve the purposes of noise reduction, dereverberation and improvement of the signal-to-noise ratio of the audio, thereby obtaining second audio, the second audio is subjected to a speech activity detection and segment division operation to obtain a candidate speech segment, a target speech segment meeting the sound replication requirement is selected from the candidate speech segment, the optimal segment with clear timbre, high signal-to-noise ratio and stable pronunciation is selected from the candidate speech segment for timbre embedding extraction and speech generation, the probability of timbre deviation of the generated speech is reduced, the accuracy of the speech generation is improved, and the user experience is improved.

Description

Technical Field

[0001] This application relates to the field of sound processing technology, and in particular to a sound replication method and related apparatus. Background Technology

[0002] Zero-Shot Voice Cloning is a cutting-edge field in voice cloning technology, aiming to achieve high-fidelity personalized speech generation using only a small amount (or even a single segment) of reference audio, without requiring historical training data of the target speaker.

[0003] When the reference audio contains background noise, reverberation, or non-speech segments (such as breathing sounds), the voiceprint encoder will process the audio segments containing interference, and the extracted timbre embedding vector will be mixed with noise features or lack key spectral information, ultimately resulting in timbre deviation in the generated speech and affecting the user experience. Summary of the Invention

[0004] In view of the above problems, this application provides a sound replication method and related apparatus to improve the accuracy of the generated speech. The specific solution is as follows:

[0005] The first aspect of this application provides a sound replication method, including:

[0006] The reference audio is verified to obtain the first audio, and the first audio is enhanced to obtain the second audio.

[0007] The second audio is subjected to speech activity detection and segmentation to obtain candidate speech segments, and the target speech segment that meets the sound reproduction requirements is selected from the candidate speech segments.

[0008] The target speech segment is subjected to speech recognition to obtain transcribed text;

[0009] Perform text standardization on the target text input by the user to obtain standardized text;

[0010] Based on the transcribed text, the timbre embedding of the target speech segment is extracted;

[0011] Based on the timbre embedding, the target speech corresponding to the standardized text is generated.

[0012] In one possible implementation, audio verification is performed on the reference audio to obtain the first audio, which includes:

[0013] If the audio format of the reference audio is a specified format, calculate the effective duration of the reference audio.

[0014] If the effective duration of the audio is within a preset duration range, the sampling rate of the reference audio is determined;

[0015] If the sampling rate of the reference audio is less than a preset sampling rate threshold, the sampling rate of the reference audio is adjusted by upsampling; if the sampling rate of the reference audio is greater than the preset sampling rate threshold, the sampling rate of the reference audio is adjusted by downsampling.

[0016] Calculate the signal-to-noise ratio (SNR) of the reference audio. If the SNR of the reference audio is less than a preset SNR threshold, then add noise audio markers to the reference audio to obtain the first audio.

[0017] If the effective duration of the audio is not within the preset duration range, the reference audio is cropped to obtain the target audio within the preset duration range, and the sampling rate of the target audio is determined.

[0018] If the sampling rate of the target audio is less than a preset sampling rate threshold, the sampling rate of the target audio is adjusted by upsampling; if the sampling rate of the target audio is greater than the preset sampling rate threshold, the sampling rate of the target audio is adjusted by downsampling.

[0019] Calculate the signal-to-noise ratio (SNR) of the target audio. If the SNR of the target audio is less than a preset SNR threshold, then add noise audio markers to the reference audio to obtain the first audio.

[0020] In one possible implementation, the first audio is subjected to a speech enhancement operation to obtain a second audio, including:

[0021] With noise audio markers added to the first audio, the first audio is denoised, the denoised first audio is converted into a first complex time spectrum, and the first complex time spectrum is divided into multiple first sub-bands along the frequency axis. The high-dimensional context-aware representation of the first sub-band is determined, the complex mask of the high-dimensional context-aware representation of the first sub-band is predicted, the complex mask of the high-dimensional context-aware representation of the first sub-band is multiplied with the first complex time spectrum to obtain the enhanced first speech spectrum, and the first speech spectrum is reconstructed into the second audio.

[0022] Without adding noise audio markers to the first audio, the first audio is converted into a second complex time spectrum, and the second complex time spectrum is divided into multiple second sub-bands along the frequency axis. The high-dimensional context-aware representation of the second sub-band is determined, the complex mask of the high-dimensional context-aware representation of the second sub-band is predicted, the complex mask of the high-dimensional context-aware representation of the second sub-band is multiplied with the second complex time spectrum to obtain the enhanced second speech spectrum, and the second speech spectrum is reconstructed into the second audio.

[0023] In one possible implementation, speech activity detection and segmentation operations are performed on the second audio to obtain candidate speech segments, including:

[0024] Calculate the short-time energy and Mel-frequency cepstral coefficients of the second audio signal;

[0025] Obtain a predetermined dynamic threshold;

[0026] Using the short-time energy, the Mel-frequency cepstral coefficients, and the dynamic threshold, a speech activity detection operation is performed on the second audio. Based on the speech activity detection results, the second audio is segmented to obtain multiple candidate speech segments.

[0027] In one possible implementation, selecting target speech segments that meet the sound replication requirements from the candidate speech segments includes:

[0028] Calculate the signal-to-noise ratio, spectral continuity, and articulation of the candidate speech segments;

[0029] A weighted fusion operation is performed on the signal-to-noise ratio, the spectral continuity, and the pronunciation intelligibility to obtain the sound quality evaluation result of the candidate speech segment;

[0030] Based on the sound quality evaluation results of the candidate speech segments, target speech segments that meet the sound reproduction requirements are selected from the candidate speech segments.

[0031] In one possible implementation, the sound replication method further includes:

[0032] If the target speech segment is not selected from the candidate speech segments, the mean and standard deviation of the Euclidean distance between the Mel cepstral coefficients and the mean vector of the Mel cepstral coefficients of the specified audio frame in the second audio are calculated.

[0033] Based on the signal-to-noise ratio of the second audio, determine the adaptive adjustment coefficient;

[0034] Calculate the new dynamic threshold based on the mean, the standard deviation, and the adaptive adjustment coefficient;

[0035] Return to the steps of performing speech activity detection on the second audio using the short-time energy, the Mel cepstral coefficients, and the dynamic threshold, and execute them sequentially.

[0036] In one possible implementation, the sound replication method further includes:

[0037] Convert the target speech into target text;

[0038] Calculate the edit distance between the target text and the standardized text;

[0039] If the editing distance is greater than a preset distance threshold, the target speech is marked as having failed verification.

[0040] A second aspect of this application provides a sound replication device, comprising:

[0041] The enhancement module is used to perform audio verification on the reference audio to obtain the first audio, and to perform speech enhancement operation on the first audio to obtain the second audio;

[0042] The filtering module is used to perform speech activity detection and segmentation on the second audio to obtain candidate speech segments, and to filter out the target speech segments that meet the requirements for sound replication from the candidate speech segments.

[0043] The recognition module is used to perform speech recognition on the target speech segment to obtain transcribed text;

[0044] The standardization module is used to perform text standardization operations on the target text input by the user to obtain standardized text.

[0045] An extraction module is used to extract the timbre embedding of the target speech segment with reference to the transcribed text;

[0046] The generation module is used to generate the target speech corresponding to the standardized text based on the timbre embedding.

[0047] A third aspect of this application provides an electronic device, comprising at least one processor and a memory connected to the processor, wherein:

[0048] The memory is used to store computer programs;

[0049] The processor is used to execute the computer program so that the electronic device can implement the above-described sound replication method.

[0050] A fourth aspect of this application provides a computer storage medium carrying one or more computer programs, which, when executed by an electronic device, enable the electronic device to implement the aforementioned sound replication method.

[0051] By employing the above technical solution, this application provides a sound replication method and related apparatus. The method involves performing audio verification on a reference audio to obtain a first audio, performing speech enhancement operations on the first audio to achieve noise reduction, dereverberation, and improved audio signal-to-noise ratio, thereby obtaining a second audio, performing speech activity detection and segmentation operations on the second audio to obtain candidate speech segments, and selecting target speech segments from the candidate speech segments that meet the sound replication requirements. The optimal segment with clear timbre, high signal-to-noise ratio, and stable pronunciation is selected from the candidate speech segments for timbre embedding extraction and speech generation, reducing the probability of timbre deviation in the generated speech, improving the accuracy of speech generation, and thus improving the user experience. Attached Figure Description

[0052] The above and other features, advantages, and aspects of the embodiments of this disclosure will become more apparent from the accompanying drawings and the following detailed description. Throughout the drawings, the same or similar reference numerals denote the same or similar elements. It should be understood that the drawings are schematic, and the originals and elements are not necessarily drawn to scale.

[0053] Figure 1 A flowchart of a sound replication method provided in this application;

[0054] Figure 2 A flowchart of an audio processing method provided in this application;

[0055] Figure 3 A flowchart for audio segmentation provided in this application;

[0056] Figure 4 A flowchart for audio filtering provided in this application;

[0057] Figure 5 This application provides a schematic diagram of the structure of a sound replication device;

[0058] Figure 6 This is a schematic diagram of the structure of an electronic device provided in this application. Detailed Implementation

[0059] The embodiments of this application are described below with reference to the accompanying drawings. The terminology used in the implementation section of this application is for explaining specific embodiments only and is not intended to limit the scope of this application.

[0060] The embodiments of this application will now be described with reference to the accompanying drawings. Those skilled in the art will recognize that, with technological advancements and the emergence of new scenarios, the technical solutions provided in the embodiments of this application are equally applicable to similar technical problems.

[0061] The terms "first," "second," etc., used in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such terms are interchangeable where appropriate; this is merely a way of distinguishing objects with the same attributes in the embodiments of this application. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion, so that a process, method, system, product, or apparatus that comprises a series of elements is not necessarily limited to those elements, but may include other elements not explicitly listed or inherent to those processes, methods, products, or apparatuses.

[0062] To enable those skilled in the art to better understand this application, the technical terms used in this application are explained below.

[0063] Zero-shot Voice Cloning: A model can mimic and generate a person's voice without ever having seen or heard a sample of that person's voice (or with only a very short audio clip).

[0064] Reference Audio: A segment of the target speaker's speech, used as a "sound reference" during model generation to extract timbre features.

[0065] ASR (Automatic Speech Recognition): Based on deep learning technology, it converts speech in audio into text.

[0066] Vocal Separation: A technique for separating the "human voice" portion from background music or other sounds in mixed audio (such as songs or voice recordings).

[0067] Audio Denoising: Removes background noise (such as wind noise, air conditioner noise, electrical noise, etc.) from noisy audio signals, preserving clear speech.

[0068] SNR (Signal-to-Noise Ratio): Measures the ratio of signal intensity to noise intensity, measured in dB. The higher the SNR, the clearer the audio.

[0069] A speaker encoder is responsible for extracting features related to the speaker's identity from speech data.

[0070] Speaker embedding: A vector representation used to characterize the timbre features of a specific speaker (or target object).

[0071] Zero-shot voice replication is a cutting-edge field in human voice replication technology, aiming to achieve high-fidelity personalized speech generation using only a small amount (or even a single segment) of reference audio, without requiring historical training data from the target speaker. This technology has undergone a rapid evolution from data-dependent models to lightweight, ready-to-use systems: early methods (such as the SV2TTS framework) required a large number of speaker samples for fine-tuning, while in recent years, deep learning-based zero-shot models (such as YourTTS and improved variants of VITS) have significantly lowered the data threshold by extracting timbre embeddings through voiceprint encoders, driving their application in scenarios such as virtual assistants, content creation, and accessibility services.

[0072] Existing technologies have significant shortcomings in the reference audio preprocessing stage of voice reproduction. The core problem lies in the lack of intelligent intervention capabilities for the reference audio, making it unable to effectively meet the stringent input quality requirements of voiceprint encoders. Specifically, because existing solutions only use general speech activity detection (VAD) and traditional signal processing tools (such as spectral subtraction) for simple trimming or noise reduction, rather than designing customized preprocessing mechanisms for timbre embedding and extraction characteristics, the following problems exist:

[0073] Existing voice reproduction models are highly sensitive to the quality of the reference audio. When the reference audio contains background noise, low sampling rate, reverberation, or non-speech segments (such as breathing sounds, silence, and environmental noise), the general VAD only judges the speech region based on the energy threshold and cannot identify key timbre segments (such as vowel sustain segments). This results in the deletion of effective speech or the retention of noisy segments. As a result, when the voiceprint encoder processes audio segments containing interference, the extracted timbre embedding vector is mixed with noise features or lacks key spectral information. Ultimately, this causes the generated speech to have timbre deviation (such as the target speaker features drifting to a neutral timbre) and noise (such as the residual sound of keyboard typing).

[0074] Meanwhile, traditional noise reduction methods, which aim to improve speech intelligibility, are insufficient in suppressing reverberation and non-stationary noise, and excessively attenuate high-frequency formants (3-8kHz), destroying the individual characteristics of timbre, resulting in poor stability of generated speech (such as abrupt changes in timbre between sentences) or even synthesis failure.

[0075] Therefore, in this embodiment of the application, in order to solve the problem of voiceprint encoder input distortion caused by background noise, reverberation and non-speech segment interference in the reference audio during zero-sample sound replication, a multi-module collaborative preprocessing mechanism is used to accurately remove invalid segments and optimize audio quality, significantly reducing the timbre deviation rate and noise residue rate of the generated speech, while improving synthesis stability and success rate.

[0076] Specifically, a multi-module collaborative mechanism is introduced in the audio preprocessing stage. First, noise reduction and dereverberation are achieved through end-to-end speech enhancement networks based on deep learning (such as FRCRN (Frequency Recurrent Convolutional Recurrent Network, a single-channel speech enhancement model based on frequency recursion) and BS-Roformer (Band-Split RoFormer, the state-of-the-art attention model in the field of music source separation) to improve the audio signal-to-noise ratio. Second, speech activity detection (VAD) technology is used to automatically identify and remove silent and non-speech segments, retaining effective speech regions. Furthermore, a sound quality evaluation and key segment selection module is designed to select the optimal segment with clear timbre, high signal-to-noise ratio, and stable pronunciation from candidate speech segments for timbre embedding extraction.

[0077] Based on the above methods, this embodiment designs and implements a high-efficiency and robust voice reproduction system. The system adopts a three-level collaborative architecture of "enhancement-detection-optimization", and through the linkage of multiple modules, it realizes end-to-end processing from raw audio to high-quality speech segments, significantly improving the performance and robustness of the voice reproduction task.

[0078] Based on the above, one embodiment of this application provides a sound replication method, referring to... Figure 1 It can include:

[0079] S11. Perform audio verification on the reference audio to obtain the first audio, and perform speech enhancement operation on the first audio to obtain the second audio.

[0080] The reference audio is a voice clip from the user whose voice needs to be replicated. The user can upload the reference audio file through the system interface. The system receives the reference audio file and stores it in a temporary buffer as source data for subsequent processing. This step does not modify the audio content; it only completes data capture and format initialization.

[0081] When processing the reference audio, since users may upload invalid files in real-world scenarios (such as 5-second silence or 8kHz phone recordings), this can affect the subsequent speech synthesis. Therefore, in this embodiment, the reference audio can be transmitted to the audio verification module in binary stream form. The audio verification module needs to perform audio verification on the reference audio to adjust the reference audio that does not meet the sound replication requirements to the first audio that does meet the sound replication requirements.

[0082] In one implementation, refer to Figure 2 Step S11 may include:

[0083] S21. If the audio format of the reference audio is a specified format, calculate the effective duration of the reference audio.

[0084] In this embodiment, the reference audio should first be an audio format that can be processed by this application embodiment. The audio formats that can be processed by this application embodiment may include WAV (Waveform Audio File Format), MP3 (MPEG-1 Audio Layer 3), and other formats.

[0085] In practice, during format verification, the audio header information of the reference audio is parsed to verify whether its audio format is supported. If it is not a supported audio format, the user is prompted to re-upload a reference audio that meets the format requirements.

[0086] If the audio format is supported, subsequent duration verification is performed. Specifically, silent segments of the reference audio are removed, and then the effective duration of the remaining audio after removing silent segments is calculated. It should be noted that in this embodiment, only longer silent segments are initially removed; shorter silent segments can be retained. For example, if a user provides a 30-second reference audio, with the first 10 seconds and the 29th second being silent segments, only the first 10 seconds of silent segments can be removed. The 29th second of silent segments can then be processed during subsequent speech activity detection and segment segmentation.

[0087] S22. Determine whether the valid duration of the audio is not within the preset duration range; if yes, proceed to step S23; if no, proceed to step S26.

[0088] Specifically, the audio verification module has certain duration requirements for the valid audio duration, which is generally 1 to 30 seconds.

[0089] If the valid audio duration is within this timeframe, subsequent processing can proceed directly. If it is not within this timeframe, a trimming operation is required to adjust it to the required duration before further processing.

[0090] S23. Determine the sampling rate of the reference audio.

[0091] In this embodiment, if the sampling rate of the reference audio is too low, high-frequency information will be lost; if the sampling rate is too high, the subsequent model will not be able to understand the audio well. Therefore, in this embodiment, to ensure that high-frequency information is not lost and to facilitate the model's understanding of the audio, a more optimal sampling rate, such as 16kHz, is set. The sampling rate of the reference audio will then be verified to calculate its correct value and determine if it is indeed 16kHz.

[0092] S24. If the sampling rate of the reference audio is less than the preset sampling rate threshold, the sampling rate of the reference audio is adjusted by upsampling; if the sampling rate of the reference audio is greater than the preset sampling rate threshold, the sampling rate of the reference audio is adjusted by downsampling.

[0093] The preset sampling rate threshold is 16kHz as mentioned above. If the sampling rate of the reference audio is <16kHz, the resampling module (which can use the Lanczos interpolation algorithm) is activated to upsample to 16kHz to avoid loss of high-frequency information; if the sampling rate of the reference audio is >16kHz, it is downsampled to 16kHz to meet the processing standards of the subsequent model and to facilitate the model's understanding of the audio.

[0094] S25. Calculate the signal-to-noise ratio of the reference audio. If the signal-to-noise ratio of the reference audio is less than the preset signal-to-noise ratio threshold, then add noise audio markers to the reference audio to obtain the first audio.

[0095] After the sampling rate verification is completed, a preliminary signal-to-noise ratio (SNR) screening operation is performed. At this time, the SNR of the reference audio is estimated through short-time energy analysis. If the SNR is less than the preset SNR threshold (e.g., 5dB), it indicates that there is a lot of noise in the reference audio, and the reference audio needs to be marked as "high-noise audio" to obtain the first audio.

[0096] S26. Perform a cropping operation on the reference audio to obtain the target audio within a preset duration range, and determine the sampling rate of the target audio.

[0097] If the effective duration of the audio is not within the preset duration range, since only audio within the preset duration range can be processed, the target audio of 1 to 30 seconds can be cut out from the starting point where there is no silence in the reference audio.

[0098] The subsequent process for determining the sampling rate of the target audio is described in the corresponding section above.

[0099] S27. If the sampling rate of the target audio is less than the preset sampling rate threshold, the sampling rate of the target audio is adjusted by upsampling; if the sampling rate of the target audio is greater than the preset sampling rate threshold, the sampling rate of the target audio is adjusted by downsampling.

[0100] S28. Calculate the signal-to-noise ratio of the target audio. If the signal-to-noise ratio of the target audio is less than the preset signal-to-noise ratio threshold, then add noise audio markers to the reference audio to obtain the first audio.

[0101] For specific implementation details of steps S27 and S28, please refer to the corresponding descriptions above.

[0102] In this embodiment, through format verification, duration verification, sampling rate verification, and initial signal-to-noise ratio screening, a standardized audio data stream that has passed verification is finally output to the subsequent modules, thereby improving the accuracy of subsequent processing.

[0103] After obtaining the first audio, the preprocessing module can be used to perform timbre-aware speech enhancement on the first audio to obtain the second audio.

[0104] The preprocessing module employs a three-tiered cascaded architecture, comprising three sub-modules: speech enhancement processing, speech activity detection and segmentation, and audio quality assessment and key segment selection. A data feedback mechanism exists between these sub-modules to ensure the processing results meet the requirements of the voiceprint encoder. The three sub-modules process the audio data stream sequentially, outputting optimized audio segments.

[0105] In this embodiment, the speech enhancement processing submodule performs speech enhancement operations on the first audio to obtain the second audio.

[0106] In one implementation, the specific processing flow of the speech enhancement processing submodule is as follows:

[0107] If the first audio has been marked with noise (i.e., the "high noise audio" mentioned above), noise reduction processing is performed on the first audio. The specific noise reduction method can be a strong noise reduction method, such as FFmpeg noise reduction, to obtain the noise-reduced first audio.

[0108] Then, the BS-Roformer model can be used to perform end-to-end enhancement on the first audio after noise reduction, so as to separate the human voice from other sounds and obtain the human voice part.

[0109] In practice, the first audio signal after noise reduction is converted into a first complex time spectrum using STFT (Short-Time-Frequency Transform), and then the first complex time spectrum is divided into multiple first sub-bands along the frequency axis. A common implementation is to divide it into 8 sub-bands on a logarithmic scale, with special protection given to the 1–4 kHz frequency band, which is sensitive to human hearing.

[0110] Then, a high-dimensional context-aware representation for the first sub-band is determined. Specifically, each sub-band is input into a Rotary Position Embedding (RoPE) Transformer, and temporal dynamics and cross-band dependencies are jointly modeled to obtain the output. The Transformer can be a band-adaptive Transformer structure that dynamically suppresses background noise (such as keyboard sounds, wind noise) and reverberation using a band attention mechanism. The input to the RoPE Transformer model is the time-frequency features (such as complex spectra or amplitude spectra) of multiple sub-bands. The RoPE Transformer model is responsible for "understanding" the complex spatiotemporal structure of the speech in each sub-band, and its output is a high-dimensional context-aware embedding for each time-frequency point.

[0111] Then, the complex mask of the high-dimensional context-aware representation of the first sub-band is predicted, and the complex mask of the high-dimensional context-aware representation of the first sub-band is multiplied with the first complex temporal spectrum to obtain the enhanced first speech spectrum. The first speech spectrum is then reconstructed into the second audio.

[0112] Specifically, based on the high-dimensional context-aware representation of each time-frequency point output by the RoPE Transformer model, a complex mask (containing amplitude and phase, or real and imaginary parts) of the high-dimensional context-aware representation of the first sub-band is predicted through a lightweight output head (also known as a phase reconstruction network). The complex mask of the high-dimensional context-aware representation of the first sub-band is multiplied with the first complex time-frequency spectrum to obtain the enhanced first speech spectrum. Finally, the enhanced first speech spectrum is reconstructed into a high-quality noise-reduced frequency stream through iSTFT (Inverse Short-Time Fourier Transform) to obtain the second audio.

[0113] In this embodiment, a lightweight output head is used to correct phase distortion and ensure tonal continuity.

[0114] Without adding noise audio markers to the first audio, the first audio is converted into a second complex time spectrum, and the second complex time spectrum is divided into multiple second sub-bands along the frequency axis. The high-dimensional context-aware representation of the second sub-band is determined, the complex mask of the high-dimensional context-aware representation of the second sub-band is predicted, and the complex mask of the high-dimensional context-aware representation of the second sub-band is multiplied with the second complex time spectrum to obtain the enhanced second speech spectrum. The second speech spectrum is then reconstructed into the second audio.

[0115] Specifically, if no noise audio marker is added to the first audio, it means that the first audio is not high noise audio. In this case, the first audio is low noise audio. In this case, the fidelity of the timbre details is prioritized and no noise reduction processing is required. The RoPE Transformer model and the lightweight output head are used for sequential processing. For the specific implementation process, please refer to the corresponding description above.

[0116] In this embodiment, after processing the audio using the RoPE Transformer model and a lightweight output head, a high-quality noise-reduced frequency stream can be output, with a significantly improved signal-to-noise ratio and low distortion rate in the key frequency band.

[0117] S12. Perform speech activity detection and segmentation on the second audio to obtain candidate speech segments, and select the target speech segment that meets the requirements of sound reproduction from the candidate speech segments.

[0118] In this embodiment, although the signal-to-noise ratio of the second audio is significantly improved and the distortion rate of the key frequency band is low compared with the reference audio, the non-key speech segments in the second audio will still affect the sound reproduction effect. Therefore, in this embodiment, the second audio can be divided into multiple candidate speech segments by speech activity detection and segment segmentation operations, and then the key speech, i.e. the target speech segment, can be selected from the candidate speech segments.

[0119] S13. Perform speech recognition on the target speech segment to obtain the transcribed text.

[0120] In this embodiment, when performing sound replication, the text of the target speech segment is required. Therefore, a lightweight ASR engine (Conformer-Transducer architecture) can be called to identify the target speech segment and generate transcribed text.

[0121] Text transcription can provide intermediate hidden features (such as phoneme-level representations or speaker-independent acoustic features) for subsequent processes, which can assist in voiceprint modeling and improve the accuracy of human voice replication.

[0122] S14. Perform text standardization on the target text input by the user to obtain standardized text.

[0123] In this embodiment, the user is the user who needs to replicate the voice. The target text input by the user can be in UTF-8 encoded string format. The target text is subjected to text standardization operations such as removing special characters and word segmentation to obtain standardized text.

[0124] S15. Refer to the transcribed text and extract the timbre embedding of the target speech segment.

[0125] In this embodiment, the timbre embedding of the target speech segment is extracted by referencing the transcribed text using a voiceprint encoder (ECAPA-TDNN architecture).

[0126] S16. Generate the target speech corresponding to the standardized text based on the timbre embedding.

[0127] In this embodiment, a speech synthesis model (such as VITS, Variational Inference with Adversarial Learning for End-to-End Text-to-Speech) is used to fuse standardized text with timbre embeddings to generate the target speech corresponding to the standardized text.

[0128] In this embodiment, the reference audio is verified to obtain a first audio. Speech enhancement is performed on the first audio to achieve noise reduction, dereverberation, and improved audio signal-to-noise ratio, thereby obtaining a second audio. Speech activity detection and segmentation are performed on the second audio to obtain candidate speech segments. Target speech segments that meet the requirements for sound replication are selected from the candidate speech segments. The optimal segment with clear timbre, high signal-to-noise ratio, and stable pronunciation is selected from the candidate speech segments for timbre embedding extraction and speech generation, reducing the probability of timbre deviation in the generated speech, improving the accuracy of speech generation, and thus improving the user experience.

[0129] Based on any of the above embodiments, refer to Figure 3 The second audio file undergoes speech activity detection and segmentation to obtain candidate speech segments, which may include:

[0130] S31. Calculate the short-time energy and Mel-frequency cepstral coefficients of the second audio frequency.

[0131] In this embodiment, in the speech activity detection and segmentation submodule, the second audio is input into the improved VAD (based on Silero VAD fine-tuning), and the improved VAD (based on Silero VAD fine-tuning) is used to detect the speech activity of the second audio. The improved VAD (based on Silero VAD fine-tuning) calculates the short-time energy and Mel-frequency cepstrum coefficients of the second audio, where the Mel-frequency cepstrum coefficients can be 13-dimensional MFCC (Mel-Frequency Cipstal Coefficients) features.

[0132] S32. Obtain a predetermined dynamic threshold.

[0133] In this embodiment, the improved VAD employs a dynamic threshold strategy: it adaptively adjusts according to the noise level—the threshold is widened to -20 dB when there is high noise, and tightened to -30 dB otherwise, with an initial value of -25 dB. In this embodiment, a pre-determined dynamic threshold can be used.

[0134] S33. Using short-time energy, Mel-frequency cepstral coefficients, and dynamic thresholds, perform speech activity detection on the second audio. Based on the speech activity detection results, segment the second audio to obtain multiple candidate speech segments.

[0135] In this embodiment, the improved VAD uses short-time energy, Mel-frequency cepstral coefficients, and a dynamic threshold to perform speech activity detection on the second audio, outputting a timestamp sequence of speech activity, and thereby segmenting the second audio into at least one candidate speech segment. Each candidate speech segment includes active speech from the second audio, and each candidate speech segment is an independent audio stream.

[0136] Based on this embodiment, refer to Figure 4 The target speech segments that meet the requirements for voice reproduction are selected from the candidate speech segments, including:

[0137] S41. Calculate the signal-to-noise ratio, spectral continuity, and articulation of the candidate speech segments.

[0138] In this embodiment, the Mel spectrogram of the candidate speech segment is input into a lightweight CNN (Convolutional Neural Network), and the signal-to-noise ratio (SNR), spectral continuity (SC), and clarity index (CI) of the candidate speech segment are extracted by the lightweight CNN.

[0139] S42. Perform a weighted fusion operation on the signal-to-noise ratio, spectral continuity, and pronunciation intelligibility to obtain the sound quality evaluation results of the candidate speech segments.

[0140] In this embodiment, weight values ​​are configured for signal-to-noise ratio (SNR), spectral continuity, and articulation. These weight values ​​can be the same or different, depending on the actual configuration. Then, SNR, spectral continuity, and articulation are multiplied by their respective weight values ​​and summed to obtain the sound quality evaluation result of the candidate speech segment. The sound quality evaluation result includes a quality score, which is specifically a weighted sum of SNR, spectral continuity, and articulation.

[0141] S43. Based on the sound quality evaluation results of the candidate speech segments, select the target speech segments that meet the sound reproduction requirements from the candidate speech segments.

[0142] In this embodiment, the sound replication requirement includes a quality score greater than a quality threshold (e.g., 7.5). Specifically, high-quality candidate speech segments with a quality score ≥ 7.5 are selected. If multiple segments exist, the segment with a vowel duration > 0.8 seconds is chosen. If there are still multiple segments with a quality score ≥ 7.5 and a vowel duration > 0.8 seconds, the segment with the highest quality score or the longest vowel duration can be selected as the target speech segment.

[0143] If no high-quality candidate speech segment with a quality score ≥ 7.5 is found, the feedback mechanism is triggered to adjust the dynamic threshold used in the improved VAD and to re-execute step 32 and subsequent steps.

[0144] In one implementation, when the target speech segment is not selected from the candidate speech segments, the voice replication method further includes:

[0145] 1) Calculate the mean and standard deviation of the Euclidean distance between the Mel cepstral coefficients and the mean vector of the Mel cepstral coefficients of a specified audio frame in the second audio.

[0146] In this embodiment, the improved VAD includes a sliding noise model (or non-speech template), which contains a mean vector of Mel-frequency cepstral coefficients, specifically the mean vector of noise MFCCs. Based on a sliding window, the Euclidean distance between the MFCC of each current frame within the most recent N seconds (N is a positive integer) of the second audio and the mean vector of noise MFCCs in the sliding noise model can be calculated, and the mean 'a' and standard deviation 'b' of all Euclidean distances can be calculated.

[0147] 2) Determine the adaptive adjustment coefficients based on the signal-to-noise ratio of the second audio.

[0148] The adaptive adjustment coefficient is denoted by k, which is an adjustable hyperparameter that determines how much deviation from the mean is considered speech. The adaptive adjustment coefficient k can be estimated based on the SNR of the second audio. If the SNR is high, k needs to be decreased; if the SNR is low, k needs to be increased.

[0149] 3) Calculate the new dynamic threshold based on the mean, standard deviation, and adaptive adjustment coefficient.

[0150] In this embodiment, the formula for calculating the new dynamic threshold is: a + k × b.

[0151] Where a is the mean of all Euclidean distances, b is the standard deviation of all Euclidean distances, and k is the adaptive adjustment coefficient.

[0152] After calculating the new dynamic threshold, the process returns to the step of obtaining the predetermined dynamic threshold, using short-time energy, Mel-frequency cepstral coefficients, and the dynamic threshold to perform speech activity detection on the second audio. Based on the speech activity detection results, the second audio is segmented to obtain multiple candidate speech segments. Then, the target speech segment is selected. If the target speech segment still cannot be obtained, the dynamic threshold is adjusted again until the target speech segment is selected. If the target speech segment cannot be selected after multiple attempts, the user is prompted.

[0153] In this embodiment, the speech activity detection based on dynamic thresholds can adaptively adjust the dynamic thresholds according to the noise level during the enhancement phase of the input audio and perform corresponding target speech segment filtering operations.

[0154] Based on any of the above embodiments, in another implementation of this application, the sound replication method further includes:

[0155] 1) Convert the target speech into the target text.

[0156] In practical implementation, after synthesizing the speech data stream, i.e., the target speech mentioned above, a post-verification module can be set to verify whether the generated target speech is correct. Specifically, the verification can be performed by checking the degree of difference between the target text corresponding to the generated target speech and the standardized text input by the user after processing, to determine whether the generated target speech is correct.

[0157] First, the ASR engine can be used to identify the target text corresponding to the target speech.

[0158] 2) Calculate the edit distance between the target text and the standardized text. If the edit distance is greater than the preset distance threshold, mark the target speech as failing the verification.

[0159] In this embodiment, the edit distance (ED) between the target text and the standardized text is calculated. If ED > a preset distance threshold (e.g., 0.2), it indicates that the generated target text differs too much from the user-input text, meaning the generated text does not meet the user's requirements. In this case, the target text can be marked as "verification failed." The verification result (pass / fail) and the target speech are then output and returned to the user interface.

[0160] It should be noted that the post-verification module is executed after the synthesized speech. It is only used to calculate the editing distance and mark "verification failed", and does not modify the synthesis result.

[0161] In this embodiment, by setting a post-verification module, the fidelity and consistency of the speech content can be evaluated, so that users can understand whether the speech synthesis result is accurate.

[0162] In summary, because related technologies only use general speech activity detection and traditional noise reduction tools to process reference audio, they cannot identify key timbre segments and have insufficient noise suppression. This results in the voiceprint encoder receiving interference-laden input, and the extracted timbre embedding vector being mixed with noise features or lacking key spectral information, affecting the accuracy of subsequent speech synthesis. In this embodiment, the accuracy of speech synthesis is improved through a cascaded process of enhancement → detection → optimization. Specifically, the enhancement stage involves deep learning enhancement of the original reference audio based on timbre perception. The detection stage uses dynamic threshold-based speech activity detection (VAD), whose threshold must be adaptively adjusted according to the noise level in the enhancement stage. The optimization stage performs timbre perception quality assessment on candidate speech segments and forces the output of a single high-quality segment as input to the voiceprint encoder. After the above processing, in most cases, the audio segment received by the voiceprint encoder will meet the hard conditions of signal-to-noise ratio ≥15dB and pronunciation clarity ≥80% (if not met, the user is prompted to re-enter the reference audio), solving the problem of timbre embedding distortion caused by multi-segment splicing in existing technologies.

[0163] Through experimental analysis, in the presence of background noise in the reference audio, the embodiments of this application increase the success rate of the sound replication task from 65% to 85% in related technologies. At the same time, the timbre similarity of the synthesized speech (cosine similarity with the target speaker's reference audio) increases from 0.68 to 0.82, and the noise rate decreases from 38% to 9%, effectively reducing timbre drift and synthesis failure.

[0164] Based on the embodiments of the above-described sound replication method, another embodiment of this application provides a sound replication apparatus, referring to... Figure 5 ,include:

[0165] Enhancement module 11 is used to perform audio verification on the reference audio to obtain the first audio, and to perform speech enhancement operation on the first audio to obtain the second audio;

[0166] The filtering module 12 is used to perform speech activity detection and segmentation on the second audio to obtain candidate speech segments, and to filter out the target speech segments that meet the requirements of sound reproduction from the candidate speech segments.

[0167] The recognition module 13 is used to perform speech recognition on the target speech segment to obtain the transcribed text;

[0168] Standardization module 14 is used to perform text standardization operations on the target text input by the user to obtain standardized text;

[0169] Extraction module 15 is used to extract the timbre embedding of the target speech segment from the reference transcribed text;

[0170] The generation module 16 is used to generate the target speech corresponding to the standardized text based on the timbre embedding.

[0171] In one implementation, enhancement module 11 includes:

[0172] The duration calculation submodule is used to calculate the effective duration of the reference audio if the audio format of the reference audio is a specified format.

[0173] The sampling rate determination submodule is used to determine the sampling rate of the reference audio when the effective duration of the audio is within a preset duration range;

[0174] The sampling submodule is used to adjust the sampling rate of the reference audio by upsampling if the sampling rate of the reference audio is less than a preset sampling rate threshold, and by downsampling if the sampling rate of the reference audio is greater than the preset sampling rate threshold.

[0175] The signal-to-noise ratio (SNR) calculation submodule is used to calculate the SNR of the reference audio. If the SNR of the reference audio is less than the preset SNR threshold, a noise audio marker is added to the reference audio to obtain the first audio.

[0176] The trimming submodule is used to trim the reference audio when the effective duration of the audio is not within the preset duration range, so as to obtain the target audio within the preset duration range and determine the sampling rate of the target audio.

[0177] The sampling submodule is also used to adjust the sampling rate of the target audio by upsampling if the sampling rate of the target audio is less than a preset sampling rate threshold, and to adjust the sampling rate of the target audio by downsampling if the sampling rate of the target audio is greater than the preset sampling rate threshold.

[0178] The signal-to-noise ratio (SNR) calculation submodule is also used to calculate the SNR of the target audio. If the SNR of the target audio is less than the preset SNR threshold, a noise audio marker is added to the reference audio to obtain the first audio.

[0179] In one implementation, enhancement module 11 includes:

[0180] The first enhancement submodule is used to perform noise reduction processing on the first audio when the first audio has been marked with noise audio tags, convert the noise-reduced first audio into a first complex time spectrum, divide the first complex time spectrum into multiple first sub-bands along the frequency axis, determine the high-dimensional context-aware representation of the first sub-band, predict the complex mask of the high-dimensional context-aware representation of the first sub-band, multiply the complex mask of the high-dimensional context-aware representation of the first sub-band with the first complex time spectrum to obtain the enhanced first speech spectrum, and reconstruct the first speech spectrum into the second audio.

[0181] The second enhancement submodule is used to convert the first audio into a second complex time spectrum without adding noise audio markers, divide the second complex time spectrum into multiple second sub-bands along the frequency axis, determine the high-dimensional context-aware representation of the second sub-band, predict the complex mask of the high-dimensional context-aware representation of the second sub-band, multiply the complex mask of the high-dimensional context-aware representation of the second sub-band with the second complex time spectrum to obtain the enhanced second speech spectrum, and reconstruct the second speech spectrum into the second audio.

[0182] In one implementation, the filtering module 12 includes:

[0183] The coefficient calculation submodule is used to calculate the short-time energy and Mel cepstral coefficients of the second audio.

[0184] The threshold acquisition submodule is used to acquire a predetermined dynamic threshold.

[0185] The segmentation submodule is used to perform speech activity detection on the second audio using short-time energy, Mel-frequency cepstral coefficients, and dynamic thresholds. Based on the speech activity detection results, the second audio is segmented to obtain multiple candidate speech segments.

[0186] In one implementation, the filtering module 12 includes:

[0187] The data calculation submodule is used to calculate the signal-to-noise ratio, spectral continuity, and pronunciation intelligibility of candidate speech segments;

[0188] The weighted fusion submodule is used to perform weighted fusion operations on signal-to-noise ratio, spectral continuity and articulation to obtain the sound quality evaluation results of candidate speech segments;

[0189] The filtering submodule is used to select target speech segments that meet the requirements for sound reproduction from the candidate speech segments based on the sound quality evaluation results of the candidate speech segments.

[0190] One implementation also includes:

[0191] The statistics module is used to calculate the mean and standard deviation of the Euclidean distance between the Mel cepstral coefficients and the mean vector of the Mel cepstral coefficients of a specified audio frame in the second audio when the target audio segment has not been selected from the candidate audio segments.

[0192] The coefficient adjustment module is used to determine the adaptive adjustment coefficients based on the signal-to-noise ratio of the second audio.

[0193] The threshold calculation module is used to calculate a new dynamic threshold based on the mean, standard deviation, and adaptive adjustment coefficient.

[0194] The segmentation submodule is also used to perform speech activity detection on the second audio using short-time energy, Mel-frequency cepstral coefficients, and the dynamic threshold after calculating a new dynamic threshold.

[0195] One implementation also includes:

[0196] The conversion module is used to convert target speech into target text;

[0197] The distance calculation module is used to calculate the edit distance between the target text and the standardized text;

[0198] The marking module is used to mark the target speech as failing verification when the edit distance is greater than a preset distance threshold.

[0199] In this embodiment, the reference audio is verified to obtain a first audio. Speech enhancement is performed on the first audio to achieve noise reduction, dereverberation, and improved audio signal-to-noise ratio, thereby obtaining a second audio. Speech activity detection and segmentation are performed on the second audio to obtain candidate speech segments. Target speech segments that meet the requirements for sound replication are selected from the candidate speech segments. The optimal segment with clear timbre, high signal-to-noise ratio, and stable pronunciation is selected from the candidate speech segments for timbre embedding extraction and speech generation, reducing the probability of timbre deviation in the generated speech, improving the accuracy of speech generation, and thus improving the user experience.

[0200] It should be noted that the working process of each module and sub-module in this embodiment is described in the corresponding descriptions in the above embodiments, and will not be repeated here.

[0201] This application also provides an electronic device, including at least one processor and a memory connected to the processor, wherein:

[0202] Memory is used to store computer programs;

[0203] The processor is used to execute computer programs to enable electronic devices to implement the aforementioned sound replication method.

[0204] refer to Figure 6 The diagram illustrates a structural schematic suitable for implementing the electronic device in the embodiments of this application. The electronic device in the embodiments of this application may include, but is not limited to, fixed terminals such as mobile phones, laptops, PDAs (personal digital assistants), PADs (tablet computers), desktop computers, etc. Figure 6 The electronic device shown is merely an example and should not impose any limitation on the functionality and scope of use of the embodiments of this application.

[0205] like Figure 6As shown, the electronic device may include a processing unit (e.g., a central processing unit, a graphics processing unit, etc.) 601, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 602 or a program loaded from a storage device 608 into a random access memory (RAM) 603. When the electronic device is powered on, the RAM 603 also stores various programs and data required for the operation of the electronic device. The processing unit 601, ROM 602, and RAM 603 are interconnected via a bus 604. An input / output (I / O) interface 605 is also connected to the bus 604.

[0206] Typically, the following devices can be connected to I / O interface 605: input devices 606 including, for example, touchscreens, touchpads, keyboards, mice, cameras, microphones, accelerometers, gyroscopes, etc.; output devices 607 including, for example, liquid crystal displays (LCDs), speakers, vibrators, etc.; storage devices 608 including, for example, memory cards, hard drives, etc.; and communication devices 609. Communication device 609 allows electronic devices to communicate wirelessly or wiredly with other devices to exchange data. Although Figure 6 Electronic devices with various devices are shown, but it should be understood that it is not required to implement or have all of the devices shown. More or fewer devices may be implemented or have alternatively.

[0207] This application also provides a computer program product including computer-readable instructions, which, when executed on an electronic device, cause the electronic device to implement any of the sound replication methods provided in this application.

[0208] This application also provides a computer storage medium that carries one or more computer programs. When the one or more computer programs are executed by an electronic device, the electronic device can implement any of the sound replication methods provided in this application.

[0209] It should also be noted that the device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. In addition, in the device embodiment drawings provided in this application, the connection relationship between modules indicates that they have a communication connection, which can be implemented as one or more communication buses or signal lines.

[0210] Through the above description of the embodiments, those skilled in the art can clearly understand that this application can be implemented by means of software plus necessary general-purpose hardware, or it can be implemented by special-purpose hardware including application-specific integrated circuits, special-purpose CPUs, special-purpose memory, special-purpose components, etc. Generally, any function performed by a computer program can be easily implemented by corresponding hardware, and the specific hardware structure used to implement the same function can also be diverse, such as analog circuits, digital circuits, or special-purpose circuits. However, for this application, software program implementation is more often the preferred implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a readable storage medium, such as a computer floppy disk, USB flash drive, mobile hard disk, ROM, RAM, magnetic disk, or optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, training equipment, or network device, etc.) to execute the methods described in the various embodiments of this application.

[0211] In the above embodiments, implementation can be achieved, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented in software, it can be implemented, in whole or in part, as a computer program product.

[0212] The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions may be transmitted from one website, computer, training device, or data center to another website, computer, training device, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium may be any available medium that a computer can store or a data storage device such as a training device or data center that integrates one or more available media. The available media may be magnetic media (e.g., floppy disks, hard disks, magnetic tapes), optical media (e.g., DVDs), or semiconductor media (e.g., solid-state drives (SSDs)).

Claims

1. A method for sound replication, characterized in that, include: The reference audio is verified to obtain the first audio, and the first audio is enhanced to obtain the second audio. The second audio is subjected to speech activity detection and segmentation to obtain candidate speech segments. Target speech segments that meet the sound replication requirements are then selected from these candidate segments. This process includes: performing speech activity detection on the second audio using a dynamic threshold, the short-time energy of the second audio, and Mel-frequency cepstral coefficients; segmenting the second audio based on the speech activity detection results to obtain multiple candidate speech segments containing active speech from the second audio; and selecting target speech segments that meet the sound replication requirements from these candidate speech segments using the sound quality evaluation results and vowel durations. The evaluation result is obtained by weighted fusion of the signal-to-noise ratio, spectral continuity, and articulation of the candidate speech segments; if the target speech segment is not selected from the candidate speech segments, the mean and standard deviation of the Euclidean distance between the Mel-frequency cepstral coefficients and the mean vector of the Mel-frequency cepstral coefficients of a specified audio frame in the second audio are calculated; an adaptive adjustment coefficient is determined based on the signal-to-noise ratio of the second audio; a new dynamic threshold is calculated based on the mean, the standard deviation, and the adaptive adjustment coefficient; and the steps of performing speech activity detection on the second audio using the dynamic threshold, the short-time energy of the second audio, and the Mel-frequency cepstral coefficients are returned. The target speech segment is subjected to speech recognition to obtain transcribed text; Perform text standardization on the target text input by the user to obtain standardized text; Based on the transcribed text, the timbre embedding of the target speech segment is extracted; Based on the timbre embedding, the target speech corresponding to the standardized text is generated.

2. The sound replication method according to claim 1, characterized in that, The reference audio is verified to obtain the first audio, which includes: If the audio format of the reference audio is a specified format, calculate the effective duration of the reference audio. If the effective duration of the audio is within a preset duration range, the sampling rate of the reference audio is determined; If the sampling rate of the reference audio is less than a preset sampling rate threshold, the sampling rate of the reference audio is adjusted by upsampling; if the sampling rate of the reference audio is greater than the preset sampling rate threshold, the sampling rate of the reference audio is adjusted by downsampling. Calculate the signal-to-noise ratio (SNR) of the reference audio. If the SNR of the reference audio is less than a preset SNR threshold, then add noise audio markers to the reference audio to obtain the first audio. If the effective duration of the audio is not within the preset duration range, the reference audio is cropped to obtain the target audio within the preset duration range, and the sampling rate of the target audio is determined. If the sampling rate of the target audio is less than a preset sampling rate threshold, the sampling rate of the target audio is adjusted by upsampling; if the sampling rate of the target audio is greater than the preset sampling rate threshold, the sampling rate of the target audio is adjusted by downsampling. Calculate the signal-to-noise ratio (SNR) of the target audio. If the SNR of the target audio is less than a preset SNR threshold, then add noise audio markers to the reference audio to obtain the first audio.

3. The sound replication method according to claim 2, characterized in that, The first audio is subjected to speech enhancement to obtain the second audio, which includes: With noise audio markers added to the first audio, the first audio is denoised, the denoised first audio is converted into a first complex time spectrum, and the first complex time spectrum is divided into multiple first sub-bands along the frequency axis. The high-dimensional context-aware representation of the first sub-band is determined, the complex mask of the high-dimensional context-aware representation of the first sub-band is predicted, the complex mask of the high-dimensional context-aware representation of the first sub-band is multiplied with the first complex time spectrum to obtain the enhanced first speech spectrum, and the first speech spectrum is reconstructed into the second audio. Without adding noise audio markers to the first audio, the first audio is converted into a second complex time spectrum, and the second complex time spectrum is divided into multiple second sub-bands along the frequency axis. The high-dimensional context-aware representation of the second sub-band is determined, the complex mask of the high-dimensional context-aware representation of the second sub-band is predicted, the complex mask of the high-dimensional context-aware representation of the second sub-band is multiplied with the second complex time spectrum to obtain the enhanced second speech spectrum, and the second speech spectrum is reconstructed into the second audio.

4. The sound replication method according to claim 1, characterized in that, The second audio is subjected to speech activity detection and segmentation operations to obtain candidate speech segments, including: Calculate the short-time energy and Mel-frequency cepstral coefficients of the second audio signal; Obtain a predetermined dynamic threshold; Using the short-time energy, the Mel-frequency cepstral coefficients, and the dynamic threshold, a speech activity detection operation is performed on the second audio. Based on the speech activity detection results, the second audio is segmented to obtain multiple candidate speech segments.

5. The sound replication method according to claim 4, characterized in that, Selecting target speech segments that meet the requirements for voice reproduction from the candidate speech segments includes: Calculate the signal-to-noise ratio, spectral continuity, and articulation of the candidate speech segments; A weighted fusion operation is performed on the signal-to-noise ratio, the spectral continuity, and the pronunciation intelligibility to obtain the sound quality evaluation result of the candidate speech segment; Based on the sound quality evaluation results of the candidate speech segments, target speech segments that meet the sound reproduction requirements are selected from the candidate speech segments.

6. The sound replication method according to claim 5, characterized in that, The sound replication method also includes: If the target speech segment is not selected from the candidate speech segments, the mean and standard deviation of the Euclidean distance between the Mel cepstral coefficients and the mean vector of the Mel cepstral coefficients of the specified audio frame in the second audio are calculated. Based on the signal-to-noise ratio of the second audio, determine the adaptive adjustment coefficient; Calculate the new dynamic threshold based on the mean, the standard deviation, and the adaptive adjustment coefficient; Return to the steps of performing speech activity detection on the second audio using the short-time energy, the Mel cepstral coefficients, and the dynamic threshold, and execute them sequentially.

7. The sound replication method according to claim 1, characterized in that, The sound replication method also includes: Convert the target speech into target text; Calculate the edit distance between the target text and the standardized text; If the editing distance is greater than a preset distance threshold, the target speech is marked as having failed verification.

8. A sound replication device, characterized in that, include: The enhancement module is used to perform audio verification on the reference audio to obtain the first audio, and to perform speech enhancement operation on the first audio to obtain the second audio; A filtering module is used to perform speech activity detection and segmentation operations on the second audio to obtain candidate speech segments, and to filter out target speech segments that meet the sound replication requirements from the candidate speech segments. This includes: performing speech activity detection on the second audio using a dynamic threshold, the short-time energy of the second audio, and Mel-frequency cepstral coefficients; segmenting the second audio based on the speech activity detection results to obtain multiple candidate speech segments including active speech in the second audio; and using the sound quality evaluation results and vowel duration of the candidate speech segments to filter out target speech segments that meet the sound replication requirements. The sound quality evaluation result is obtained by weighted fusion of the signal-to-noise ratio, spectral continuity, and articulation of the candidate speech segments; if the target speech segment is not selected from the candidate speech segments, the mean and standard deviation of the Euclidean distance between the Mel-frequency cepstral coefficients and the mean vector of the Mel-frequency cepstral coefficients of a specified audio frame in the second audio are calculated; an adaptive adjustment coefficient is determined based on the signal-to-noise ratio of the second audio; a new dynamic threshold is calculated based on the mean, the standard deviation, and the adaptive adjustment coefficient; and the steps of performing speech activity detection on the second audio using the dynamic threshold, the short-time energy of the second audio, and the Mel-frequency cepstral coefficients are returned. The recognition module is used to perform speech recognition on the target speech segment to obtain transcribed text; The standardization module is used to perform text standardization operations on the target text input by the user to obtain standardized text. An extraction module is used to extract the timbre embedding of the target speech segment with reference to the transcribed text; The generation module is used to generate the target speech corresponding to the standardized text based on the timbre embedding.

9. An electronic device, characterized in that, It includes at least one processor and a memory connected to the processor, wherein: The memory is used to store computer programs; The processor is used to execute the computer program to enable the electronic device to implement the sound replication method as described in any one of claims 1 to 7.

10. A computer storage medium, characterized in that, The computer storage medium carries one or more computer programs that, when executed by an electronic device, enable the electronic device to implement the sound replication method as described in any one of claims 1 to 7.

Images