A method and system for high-sampling-rate Korean speech cloning based on zero samples

Abstract

This invention relates to the fields of speech signal processing and speech synthesis, and particularly to a method and system for cloning Korean speech using a high sampling rate based on zero samples. The method includes: extracting audio from Korean speech video data and performing speech enhancement to obtain a clean audio set; performing speaker segmentation and embedding similarity post-processing on the clean audio set to obtain a set of speaker-consistent speech segments; automatically transcribing the speech segment set and normalizing its glyphs to form a speech-text annotation set, which is then filtered by quality scoring and constrained by segment duration to generate a training dataset; constructing a Mel training feature set from the training dataset that maintains the length and dimension of the Mel frame sequence, and training a speech cloning model using a fully non-autoregressive diffusion converter with input stream matching; during inference, aligning the speech cloning model with the reference speech and target text, performing length alignment, denoising, stream space transformation, and ordinary differential equation sampling, and outputting a cloned speech waveform. This invention effectively improves speaker consistency and preserves speech details.

Description

Technical Field

[0001] This invention relates to the fields of speech signal processing and speech synthesis, and in particular to a method and system for cloning Korean speech at a high sampling rate based on zero samples. Background Technology

[0002] Traditional speech synthesis systems are usually based on large-scale speech data for modeling, and can only generate a limited number of fixed timbres, which makes it difficult to meet users' needs for personalized and diverse speech performance.

[0003] The earliest form of speech cloning involved fine-tuning a general text-to-speech (TTS) model using large-scale speech data to obtain a specialized model that closely matched the speaker's voice. This was followed by few-sample speech cloning, which, with only a small number of speech samples from the target speaker (a few seconds to a few minutes), extracts speaker embeddings as conditional input, enabling a general TTS model to quickly generate speech with the target voice without needing to train a separate full model. Since speaker embeddings are much smaller than a complete speech model, they can be extracted and encoded much faster. This allows users to customize their voice using only a small number of speech samples.

[0004] In recent years, due to the high demand for real-time interaction and personalized experience, zero-sample speech cloning has gradually become the mainstream in order to avoid the need to re-collect and train a large amount of speech data every time. It can generate speech that imitates the timbre of the target speaker in real time by using a unified multi-speaker speech model without having seen any training data of the target speaker, but only by using a reference speech provided in the inference stage.

[0005] Among them, Shanghai Jiao Tong University's F5-TTS model provides a highly efficient speech cloning scheme. This scheme sets up a text tokenizer and a flow-matching model in the speech cloning system. When the user provides a reference speaker's speech, reference text, and target text, the audio is extracted as a Mel spectrum. The reference and target texts are concatenated together, and the Mel spectrum is fed into the flow-matching model, which generates a Mel spectrum with the target timbre through audio restoration. A vocoder (such as HiFi-GAN) converts the Mel spectrum into the final audio waveform. With the introduction of large-scale models into the field of speech generation, Alibaba's CosyVoice2 model abandons the text tokenizer, uses a codec to extract semantic messages, obtains the target token through a large-scale model, and then uses the flow-matching model to generate audio with the target timbre. This method has a certain degree of duration control. In the technical solutions described by F5-TTS and CosyVoice2, the flow-matching predicts the flow (a quantity containing noise). Based on the manifold assumption, this model predicts a noisy quantity (high-dimensional space). The output quality of this training method is often positively correlated with the amount of data and the model parameters. To ensure good inference speed, most models currently on the market can only perform speech cloning at a sampling rate of 24000Hz. Furthermore, due to the scarcity of large-scale Korean speech data, these models only have good generalization ability with Chinese and English, and perform poorly with Korean cloning.

[0006] Furthermore, in the field of speech signal processing and speech synthesis, existing solutions for high-sampling-rate Korean speech cloning methods and systems based on zero-shot samples typically construct a corpus link around Korean speech and video data. A training dataset is formed through speech enhancement processing, speaker segmentation and embedding similarity post-processing, automatic transcription and glyph normalization processing, quality score filtering and segment duration constraint processing. The training dataset is then used to construct a Mel training feature set through Mel spectrum analysis. Finally, the cloned speech waveform is output based on generative acoustic modeling and waveform synthesis. However, this approach has limitations such as strong dependence on the consistency of Korean speech and video data sources and annotations, difficulty in maintaining speaker consistency under cross-scene noise and reverberation interference, and the inconsistency between word form and symbol mapping in the speech text annotation set after automatic transcription and glyph normalization processing. Existing methods often rely on segmented processing and multi-module serialization. When faced with the synchronization requirements of high-sampling-rate speech in terms of frequency domain details and time domain alignment, issues such as inconsistency between the Mel training feature set and the model input / output interface, and unstable organization of conditional information in the training and inference links can easily arise. This leads to problems such as duration alignment deviation and discontinuous denoising generation in the process of generating cloned speech waveforms driven by reference speech and target text. As a result, it is difficult to meet the requirements of stable calling and data closed-loop maintenance of speech clone models in engineering deployment. For the joint processing of original audio sets, clean audio sets, speaker-consistent speech segment sets, speech-text annotation sets, training datasets, and Mel training feature sets, existing technologies generally lack a unified mechanism in terms of cross-module input-output constraints, boundary handling of abnormal segments, and consistency management of model training and inference processes. It is difficult to form a continuous and consistent link process between data acquisition, speech enhancement processing, speaker segmentation and embedding similarity post-processing, automatic transcription and glyph normalization processing, quality score filtering and segment duration constraint processing, Mel spectrum construction, speech clone model training, and clone speech waveform generation. This leads to increased manual review and rework costs in application processes such as speech content production and dubbing. Summary of the Invention

[0007] This invention provides a method and system for cloning Korean speech based on zero samples and high sampling rate, in order to solve the problem of how to train a fully non-autoregressive diffusion converter that inputs the Mel training feature set into the stream matching and update the model parameters based on the stream matching loss to generate a speech cloning model.

[0008] To achieve the above objectives, the present invention provides a method for cloning Korean speech at a high sampling rate based on zero samples, comprising: S100: Acquire Korean voice and video data and extract 48 kHz audio to generate the original audio set; S200: Input the original audio set into speech enhancement processing to generate a clean audio set; S300: Input the clean audio set into speaker segmentation and embedding similarity post-processing to generate a speaker-consistent speech segment set; S400: Input the speaker-consistent speech segment set into automatic transcription and character normalization processing to generate a speech text annotation set; S500: Input the speech-text annotation set into quality score filtering and segment duration constraint processing to generate a training dataset; S600. Input the training dataset into the Mel spectrum construction, set the sampling rate to 48 kHz, the highest frequency to 16 kHz, and double the FFT window and frame shift parameters while keeping the length and dimension of the Mel frame sequence unchanged, and generate the Mel training feature set. S700: Train the fully non-autoregressive diffusion converter by inputting the Mel training feature set into the stream matching, perform real Mel spectrum prediction under masked speech conditions, update the model parameters according to the stream matching loss, and generate a speech clone model. S800: Based on the speech cloning model, perform alignment and denoising, stream space transformation and ordinary differential equation sampling processing to generate a 48 kHz cloned speech waveform.

[0009] Further, S100 includes: acquiring an open-source 48 kHz Korean speech dataset and publicly available genuine Korean drama audio and video materials; performing FFmpeg audio extraction and resampling to 48 kHz on the Korean drama audio and video materials to generate the original audio set.

[0010] Furthermore, the feature is that the speech enhancement processing in S200 uses MossFormer2_SE_48K to perform speech enhancement on the original audio set to generate the clean audio set.

[0011] Furthermore, in S300, speaker segmentation uses PyAnnote-audio to output initial speaker segments from the clean audio set; speaker embeddings are extracted from the initial speaker segments, and speaker clusters with a cosine similarity greater than 0.6 are clustered and merged to generate merged speaker clusters.

[0012] Further, the embedding similarity post-processing in S300 includes: dividing the clean audio set into 1.5-second basic segments and extracting segment embeddings for each basic segment; marking adjacent basic segments whose segment embedding cosine similarity is less than 0.5 as boundaries of different speakers; assigning each basic segment to the speaker cluster whose segment embedding has the highest cosine similarity to the speaker embedding of the merged speaker cluster, thereby generating a redistributed segment set.

[0013] Furthermore, the embedding similarity post-processing in S300 also includes: merging adjacent basic segments of the redistributed segment set according to the speaker cluster identifier to obtain candidate speech segments; terminating the merging when the cumulative duration of the candidate speech segments exceeds 27 seconds or the silence interval between adjacent basic segments exceeds 2 seconds, and outputting the speaker-consistent speech segment set.

[0014] Furthermore, in S400, the automatic transcription uses FunASR's Paraformer-KO to generate transcribed text from the speaker-consistent speech fragment set; in S400, the glyph normalization uses N2gk+ to convert numbers, English words and symbols in the transcribed text into Korean glyphs, generating normalized text, and combining the normalized text with the corresponding speech fragments to form the speech text annotation set.

[0015] Furthermore, the quality score filtering in S500 uses UTMOS to calculate the quality score of the speech segments in the speech text annotation set; speech segments with a quality score less than 2.2 are deleted; the segment duration constraint in S500 deletes speech segments with a duration less than 3 seconds or a duration greater than 30 seconds, thereby generating the training dataset.

[0016] Furthermore, in S600, the Mel spectrum construction sets the Mel filter frequency range to 0 to 16000 Hz, and doubles the FFT points, window length parameters, and frame shift parameters used for Mel spectrum calculation relative to the 24 kHz configuration, while keeping the number of Mel spectrum frames and Mel frequency bands unchanged, to generate the Mel training feature set; The fully non-autoregressive diffusion converter in S700 includes a ConvNeXt text feature extraction network, a diffusion converter backbone with adaLN-0, sinusoidal position embedding, and convolutional position embedding; the true Mel spectrum prediction in S700 is performed on concatenated inputs of character sequences, noisy speech, and masked speech; the ordinary differential equation sampling in S800 includes mapping the denoised generation result to the stream space and performing ordinary differential equation solving to output the 48 kHz cloned speech waveform.

[0017] Furthermore, a high-sampling-rate Korean speech cloning system based on zero samples includes: a data acquisition module, a speech enhancement module, a speaker segmentation and embedding similarity post-processing module, an automatic transcription and glyph normalization processing module, a quality score filtering and segment duration constraint processing module, a Mel spectrum construction module, a stream-matched fully non-autoregressive diffusion converter training module, a length alignment and denoising generation module, and a stream space transformation and ordinary differential equation sampling processing module; the modules are connected in sequence to implement the method described in any of the above-mentioned steps.

[0018] The following are its main beneficial effects: (1) Compared with the existing training path that mainly relies on stepwise denoising fitting or frame-by-frame recursion, the model parameters are updated based on flow matching loss and real Mel spectrum prediction is performed under the masked speech condition. This enables the speech clone model to form a consistent conditional modeling and full sequence prediction link under the cascaded input constraints of character sequence, noisy speech and masked speech, thereby reducing the impact of modeling deviation caused by inconsistent condition organization during training on the stable generation of the speech clone model.

[0019] (2) Compared with the existing practice of changing the number of Mel spectrum frames or Mel frequency bands to complete feature adaptation when the sampling rate changes, by doubling the FFT window and frame shift parameter in the Mel spectrum construction and keeping the length and dimension of the Mel frame sequence unchanged, the Mel training feature set and the training of the stream-matched non-autoregressive diffusion converter are kept consistent in terms of input and output form, reducing the impact of training link reconfiguration and inconsistency of historical data playback caused by feature interface changes on training continuity.

[0020] (3) Compared with the existing inference process that separates length alignment, denoising generation and sampling process, by performing length alignment and denoising generation, stream space transformation and ordinary differential equation sampling processing on the speech cloning model, the reference speech and target text, the cloned speech waveform completes state transfer and sampling calculation in the same inference link, reducing the impact of length alignment fluctuation and output discontinuity caused by the inconsistency between stages on the generation consistency.

[0021] In summary, this invention aims to provide a complete high-quality, high-sampling-rate (48000 Hz) multi-speaker Korean zero-shot speech cloning technology solution. Its innovations mainly lie in the refined processing of the data pipeline and the optimization of the TTS model architecture and training strategy. First, this invention constructs a powerful Korean speech data collection and preprocessing pipeline. By integrating FFmpeg, the MossFormer2_SE_48K audio enhancement model, and incorporating a speaker segmentation and clustering post-processing workflow optimized by the "dark side of the moon" approach, it effectively extracts and filters approximately 1500 hours of high-quality, high-definition speech data from about 300 Korean dramas. Data processing also includes using Paraformer-KO and N2gk+ specifications to ensure text-acoustic consistency and employing UTMOS thresholding for rigorous quality filtering, guaranteeing the large-scale, high-fidelity dataset required for subsequent model training. Second, regarding the speech cloning method, this invention adopts a fully non-autoregressive architecture based on Flow Matching and Diffusion Transformer (DiT), achieving fast inference and powerful zero-shot cloning capabilities. The core technological innovation lies in its adaptation to high sampling rates: by adjusting the Mel spectrum parameters (doubling the sampling rate and highest frequency) without increasing the number of model parameters, the target frequency of the model was successfully extended from 8kHz to 16kHz, directly learning the high-frequency information of 48kHz audio. Furthermore, the innovative training strategy instructs the DiT model to directly predict the real Mel spectrum and combines it with stream matching loss for efficient training. Finally, sampling is performed by solving ordinary differential equations (ODEs), ensuring the high fidelity of the generated audio. Attached Figure Description

[0022] The accompanying drawings, which are included to provide a further understanding of the invention and form part of this invention, illustrate exemplary embodiments of the invention and are used to explain the invention, but do not constitute an undue limitation of the invention. In the drawings: Figure 1 A flowchart illustrating the high-sampling-rate Korean speech cloning method based on zero samples provided in this application embodiment; Figure 2 A structural block diagram of a high-sampling-rate Korean speech cloning system based on zero samples provided in this application embodiment. Figure 3 A flowchart of a Korean TV drama audio data processing method provided in this application embodiment; Figure 4 This is a model reasoning flowchart provided for an embodiment of this application. Detailed Implementation

[0023] To facilitate a clear description of the technical solutions in the embodiments of the present invention, the terms "first" and "second" are used to distinguish identical or similar items with essentially the same function and effect. For example, the first threshold and the second threshold are merely used to distinguish different thresholds and do not limit their order. Those skilled in the art will understand that the terms "first" and "second" do not limit the quantity or execution order, and that the terms "first" and "second" are not necessarily different.

[0024] It should be noted that in this invention, the terms "exemplary" or "for example" are used to indicate examples, illustrations, or descriptions. Any embodiment or design described as "exemplary" or "for example" in this invention should not be construed as being more preferred or advantageous than other embodiments or designs. Specifically, the use of terms such as "exemplary" or "for example" is intended to present the relevant concepts in a concrete manner.

[0025] In this invention, "at least one" means one or more, and "more than one" means two or more. "And / or" describes the relationship between the associated objects, indicating that three relationships can exist.

[0026] like Figure 1 The diagram shown is a flowchart of a high-sampling-rate Korean speech cloning method based on zero samples provided in an embodiment of the present invention. This process may include at least steps S100-S800: S100: Acquire Korean voice and video data and extract 48 kHz audio to generate the original audio set; S200: Input the original audio set into speech enhancement processing to generate a clean audio set; S300: Input the clean audio set into speaker segmentation and embedding similarity post-processing to generate a speaker-consistent speech segment set; S400: Input the speaker-consistent speech segment set into automatic transcription and character normalization processing to generate a speech text annotation set; S500: Input the speech-text annotation set into quality score filtering and segment duration constraint processing to generate a training dataset; S600. Input the training dataset into the Mel spectrum construction, set the sampling rate to 48 kHz, the highest frequency to 16 kHz, and double the FFT window and frame shift parameters while keeping the length and dimension of the Mel frame sequence unchanged, and generate the Mel training feature set. S700: Train the fully non-autoregressive diffusion converter by inputting the Mel training feature set into the stream matching, perform real Mel spectrum prediction under masked speech conditions, update the model parameters according to the stream matching loss, and generate a speech clone model. S800: Based on the speech cloning model, perform alignment and denoising, stream space transformation and ordinary differential equation sampling processing to generate a 48 kHz cloned speech waveform.

[0027] S100: Acquire Korean voice and video data and extract 48 kHz audio to generate the original audio set; Specifically, this step is executed by the data acquisition scheduling unit of the data access server after receiving the acquisition task triggering conditions. These triggering conditions include reaching a preset acquisition time window, detecting a new Korean audio / video data source entry, and receiving a manually submitted acquisition task order. The Korean audio / video data consists of an open-source 48kHz Korean audio dataset and publicly available, licensed Korean drama audio / video materials. The open-source 48kHz Korean audio dataset is accessed in audio file format and carries dataset identifiers, speaker identifiers, sampling rate identifiers, and language identifiers. The publicly available, licensed Korean drama audio / video materials are accessed in audio / video file format and carry episode identifiers, number identifiers, source identifiers, and copyright identifiers. This step performs integrity verification and metadata registration on the Korean audio and video data during the access phase. The integrity verification includes file hash verification, container parsing verification, and duration readability verification. The metadata registration includes writing the drama identifier, episode identifier, dataset identifier, source identifier, sampling rate identifier, channel number identifier, bit depth identifier, duration identifier, file hash, and access timestamp into the acquisition index table, and assigning a unique audio and video resource identifier to each access file to form a pending list. The pending list is written into the acquisition task status record in this step for auditing and version management. The version management uses a two-field method of "data source list version number + acquisition task batch number" for fixation. The data source list version number is generated and written by the acquisition scheduling unit based on the data source entry set each time the task is started.

[0028] Further, this step performs audio extraction processing on the publicly available, officially licensed Korean drama audio and video materials in the pending list. This audio extraction processing is performed by the audio extraction unit calling FFmpeg (Fast Forward MPEG, an audio and video processing tool). The audio extraction unit reads the corresponding audio and video stream for each audio and video resource identifier and parses the audio track information within it. When multiple audio tracks are detected, the target audio track is selected according to the track language identifier and channel layout identifier. If the track language identifier cannot be read, a default audio track is selected according to a preset priority rule. Subsequently, the target audio track is decoded into Pulse Code Modulation (PCM) format, and a unified sampling rate processing is performed on the decoded audio. This unified sampling rate processing fixes the output sampling rate to 48 kHz, resamples audio with an input sampling rate other than 48 kHz, and performs a sampling rate consistency check on audio with an input sampling rate of 48 kHz. The consistency check includes reading the sampling rate field in the output audio header information and comparing it with 48 kHz, reading the actual sampling point interval, and performing a consistency determination. For channel number processing, this step records the output channel number as a channel number identifier. When a multi-channel track is detected, channel mixing is performed to obtain a single-channel waveform. The channel mixing performs weighted summation and normalization on each channel sample according to the channel layout identifier to obtain a single-channel sample sequence; single-channel tracks are directly retained. For amplitude range processing, this step performs amplitude normalization on the output waveform and records the amplitude normalization parameters. The amplitude normalization includes DC component removal and peak clipping threshold processing. The peak clipping threshold is written as a configurable parameter in this step into the acquisition task configuration table and stored in association with the audio and video resource identifier. For the open-source 48kHz Korean speech dataset, this step performs sampling rate consistency verification and metadata mapping processing. The metadata mapping processing maps the speaker identifiers in the dataset to unified speaker key values ​​and writes them into the index table so that subsequent steps can reference the same namespace. For any file that fails to extract or resample audio, this step records the failure reason code, failure stage identifier, and number of retries. After the number of retries reaches the preset limit, the file is written to the exception queue and its task status is frozen. The exception queue entries also carry audio and video resource identifiers for subsequent manual review and batch repair.

[0029] Furthermore, this step encapsulates the output audio after audio extraction and unified sampling rate processing into a raw audio set. This raw audio set is a collection of audio samples organized according to audio / video resource identifiers and associated with corresponding metadata. Each member of the raw audio set contains, at the storage level, an audio file path, audio / video resource identifier, source identifier, episode identifier or dataset identifier, sampling rate field, channel number field, bit depth field, duration field, file hash field, acquisition batch number field, and processing status field. During operation, the acquisition scheduling unit synchronously updates the acquisition index table after writing the raw audio set, updating the processing status field from "pending processing" to "raw audio generated," and registers the raw audio set as the input object for subsequent S200 input into speech enhancement processing, thus forming a reference relationship from the output of this step to the input of the next step in cross-step connections. In one engineering embodiment, the data access server accesses publicly available, genuine Korean drama audio and video materials according to the acquisition task order and completes audio extraction and 48 kHz resampling. The generated original audio set is stored in the object storage and written to the index table. When the processing status field in the index table is updated to "original audio has been generated", the acquisition scheduling unit sends a task message containing audio and video resource identifiers and audio file paths to the downstream queue. The task message is read by the speech enhancement processing of S200 and loaded as input.

[0030] In summary, the technical effects of this step are as follows: This step establishes a traceable data access and audio extraction link around Korean voice and video data, unifying publicly available, genuine Korean drama audio and video materials with open-source 48kHz Korean voice datasets into a 48kHz audio representation. The processing procedures and status records are solidified through consistency verification, metadata registration, and anomaly queue mechanisms. The original audio set contains index information at the field level that can be directly referenced by subsequent steps, supporting batch processing calls of the original audio set by the S200. The versioned acquisition records and batch output organization provide an auditable input foundation for the construction of subsequent training data.

[0031] This invention further optimizes the pipeline for collecting and preprocessing Korean speech data to support the construction of high-quality training data. Specifically, the data sources include two parts: (1) Open source high sampling rate speech data: The Genshin-Voice and StarRail-Voice datasets are introduced as supplements. These datasets provide approximately 30 hours of Korean speech with a sampling rate of 48kHz, covering multiple speakers and multiple style scenarios, to enhance data diversity.

[0032] (2) Korean TV drama audio data processing flow: From approximately 300 publicly available, officially licensed Korean TV dramas, 48kHz audio is extracted using FFmpeg and then subjected to hierarchical processing: Audio enhancement processing: The MossFormer2_SE_48K pre-trained model from Alibaba Tongyi Labs is used to suppress noise, background noise, and reverberation, thereby improving signal clarity.

[0033] Speaker segmentation and post-processing optimization: Based on the initial segmentation of PyAnnote-audio, the post-processing workflow provided by Dark Side of the Moon is introduced. First, clusters with speaker embedding cosine similarity greater than 0.6 are merged to reduce fragmentation of the same speaker. Second, the audio is divided into 1.5-second basic segments, and speaker boundaries are marked based on segment embedding cosine similarity (threshold 0.5), and segments are reassigned to the most similar speaker clusters. Finally, adjacent segments are merged, terminating when the cumulative duration exceeds 27 seconds or the silence interval is greater than 2 seconds, generating a more consistent set of speech segments.

[0034] Text extraction and normalization: FunASR's Paraformer-KO model is used for automatic transcription, and N2gk+ is used to convert numbers, English words and symbols into Korean characters to ensure text normalization.

[0035] Data quality screening: Low-quality samples are filtered using UTMOS quality score (threshold 2.2), and the reasonableness of the segments is controlled by time constraints (3-30 seconds).

[0036] Through the aforementioned channels, this invention generates approximately 1500 hours of high-quality Korean speech data from TV drama data, providing large-scale input for subsequent training.

[0037] S200: Input the original audio set into speech enhancement processing to generate a clean audio set; Specifically, this step is triggered by the enhancement orchestration unit of the speech enhancement server when it detects that the processing status field of the original audio set is "Original audio has been generated". The enhancement orchestration unit reads the audio file path, sampling rate field, channel digital segment, bit depth field, duration field, and file hash field corresponding to the audio and video resource identifier from the acquisition index table, forms the enhancement task input, and writes it into the enhancement task queue. The speech enhancement processing is a signal processing link that performs non-speech component suppression and speech component preservation on the audio samples in the original audio set. In this embodiment, the signal processing link adopts the MossFormer2_SE_48K model. MossFormer2_SE_48K is a speech enhancement model for a 48 kHz sampling rate. Its structure includes a front-end transformation unit for time-frequency representation, a sequence modeling unit for long-term dependency modeling, and a back-end reconstruction unit for waveform reconstruction. The sequence modeling unit uses a combination of self-attention and gated convolution to perform context aggregation on the time-frequency unit. The back-end reconstruction unit maps the enhanced time-frequency representation back to the time-domain waveform. This step performs input consistency verification on the original audio set during the task enqueue phase. The input consistency verification includes consistency checks on the sampling rate field and 48 kHz, consistency checks on the channel digital segment and single-channel constraints, and consistency checks on the duration field and minimum duration threshold. If the consistency checks are not met, the corresponding audio / video resource identifier is written to the exception queue and the failure reason code is recorded. The minimum duration threshold, as one of the minimum sets of core parameters in this step, is fixed in the enhancement task configuration table and stored versioned with the task batch number. The versioned storage writes the enhancement model version identifier, inference framework version identifier, and input consistency verification parameters together into the audit record.

[0038] Furthermore, the enhancement orchestration unit loads the audio samples that have passed consistency verification into a fixed-length or variable-length streaming buffer, and divides the audio samples into enhancement processing segments according to a preset segment length. The segment length is written as a configurable parameter into the enhancement task configuration table for organizing batch processing under inference memory constraints and throughput constraints. For adjacent enhancement processing segments, this step adds an overlapping window at the segment boundary and performs overlapping addition in the output stage to reduce waveform abrupt changes introduced by boundary discontinuities. Subsequently, this step performs amplitude normalization and silence segment labeling on each enhancement processing segment. The amplitude normalization maps the segment sample amplitude to a preset amplitude range and records the normalization factor. The silence segment labeling generates a silence mask based on a joint determination of short-time energy and zero-crossing rate. The silence mask serves as a bypass control signal in subsequent inference. The interval indicated by the silence mask bypasses model inference and is directly written into the silence sample sequence, thereby reducing invalid inference. For non-silent regions, this step inputs the audio segment into MossFormer2_SE_48K for enhanced inference. This enhanced inference includes mapping the time-domain waveform to a time-frequency representation, estimating the speech mask and suppressing non-speech components in the time-frequency domain, and reconstructing the enhanced time-frequency representation back into a time-domain waveform. The speech mask is a weight matrix with the same resolution as the time-frequency representation. The value of the speech mask is determined by the confidence and amplitude channels output by the model. During inference, the post-processing unit performs threshold pruning and smoothing. The threshold and smoothing window length are among the smallest sets of core parameters for this step and are bound to the model version identifier for storage. For inference anomalies, if a non-numerical sample, overflow sample, or length mismatch sample is detected in the output waveform, this step marks the enhanced segment as a failed segment and reverts to the bypass output strategy. The bypass output strategy writes the corresponding segment waveform from the original audio set into the output and records the failed segment position index. This failed segment position index is used for subsequent manual review and model version rollback decisions. After completing the segment-level inference, this step performs splicing and overlapping addition on each enhancement processing segment in chronological order, and performs DC component removal and peak normalization to obtain the enhanced waveform. The file hash field, sampling rate field, channel digital segment, duration field and processing status field of the enhanced waveform are written into the enhancement index table.

[0039] Further, this step encapsulates the enhanced waveform into a clean audio set based on audio / video resource identifiers. The clean audio set is a collection of audio samples with the same index structure as the original audio set. The audio file path, audio / video resource identifier, sampling rate field, channel number field, bit depth field, duration field, file hash field, enhancement model version identifier, inference batch number field, and failed segment location index field of the clean audio set members are written into the data directory service, thereby forming a traceable enhancement product at the data level. During operation, the clean audio set is registered as the input object for subsequent speaker segmentation and embedding similarity post-processing in S300. Specifically, after the enhancement orchestration unit updates the processing status field to "clean audio generated" in the enhancement index table, it sends a task message containing the audio / video resource identifier and the clean audio path to the downstream queue. The task message is subscribed to and the clean audio set is loaded by the speaker segmentation server. In one engineering embodiment, the voice enhancement server is deployed on a computing node equipped with a graphics processing unit. The enhancement orchestration unit performs concurrency adjustment based on the task queue length and the graphics processing unit load. When the graphics processing unit load exceeds the load threshold, the enhancement orchestration unit reduces the batch size and extends the task scheduling interval. The load threshold and scheduling interval are written into the enhancement task configuration table and recorded along with the inference batch number, thereby enabling the clean audio set generation process and resource scheduling process to have an auditable running trajectory.

[0040] Summary of the technical effects of this step: This step completes the engineering implementation of the speech enhancement processing link around the 48 kHz sampling rate scenario. The original audio set is organized by adding and summing the inference of MossFormer2_SE_48K and the segment-level overlap, and outputs a clean audio set consistent with the index. This step solidifies the anomaly handling path at the level of input consistency verification, bypass output and failure segment position index record, so that the clean audio set has stable batch processing output and traceable version information. The clean audio set, as the input object of S300, completes the cross-step connection under the queue delivery and index status update mechanism.

[0041] S300: Input the clean audio set into speaker segmentation and embedding similarity post-processing to generate a speaker-consistent speech segment set; Specifically, this step is triggered by the segmentation scheduling unit of the speaker segmentation server after receiving a task message from S200. The task message carries audio and video resource identifiers, clean audio paths, sampling rate fields, channel digital segments, duration fields, and enhancement model version identifiers. The segmentation scheduling unit performs an input consistency check on the clean audio set. The consistency check includes determining that the sampling rate field is 48 kHz, determining that the channel digital segment is a single channel, and determining that the file hash field is consistent with the registered value. The check results are then written into the segmentation audit record. The speaker segmentation involves performing speaker activity detection and speaker change point detection on each clean audio segment in the clean audio set, outputting initial speaker segments with timestamp boundaries and speaker identifiers. In this embodiment, the speaker segmentation adopts the PyAnnote-audio model, which is a speaker segmentation model consisting of a feature extraction front-end, a temporal modeling backbone, and a segmentation decoding head. The feature extraction front-end performs short-time framing on the clean audio and calculates acoustic representations. The temporal modeling backbone performs context aggregation on the frame-level representations and outputs a speaker activity probability sequence. The segmentation decoding head decodes the probability sequence into segment boundaries and assigns initial speaker identifiers to the segments. During operation, the segmentation scheduling unit organizes the clean audio set into segmentation batches according to the duration field. When the graphics processing unit load exceeds the load threshold, it reduces the segmentation batch concurrency and records the scheduling parameters. The load threshold and concurrency are stored as the minimum set of core parameters in the segmentation task configuration table in a versioned manner, thereby enabling the segmentation process to have an auditable running trajectory.

[0042] Further, after obtaining the initial speaker segment, this step enters the embedding similarity post-processing stage. This post-processing is a chain of steps that performs consistency correction on segment-level speaker identifiers, and its core includes three interconnected processes: clustering and merging, block-level reallocation, and segment merging. The embedding similarity is the similarity value obtained by calculating the cosine similarity of the speaker embeddings of two speech segments in the same embedding space. The speaker embedding is a fixed-dimensional vector extracted from the speech segment by a speaker representation network. This speaker representation network is deployed in the embedding extraction unit. After performing energy gating and silence removal on the initial speaker segment, the embedding extraction unit inputs non-silent speech frames into the speaker representation network and outputs the speaker embedding. The speaker embedding, along with the segment timestamp boundary and the initial speaker identifier, is then registered in the embedding index table. The clustering merging process involves establishing speaker clusters along the initial speaker identifier dimension and calculating a representative speaker embedding for each cluster. This representative speaker embedding is obtained by weighting and normalizing the speaker embeddings of segments within the cluster according to the duration field. Subsequently, the embedding extraction unit calculates the cosine similarity of the representative speaker embeddings of any two speaker clusters. When the cosine similarity is greater than 0.6, the corresponding two speaker clusters are merged into a single speaker cluster, generating a merged speaker cluster. The 0.6 threshold is fixed in the post-processing configuration table as one of the minimum sets of core parameters for this step and is recorded with the inference batch number. During the clustering merging process, conflict resolution rules are set. These rules include sorting multiple merging requests for the same speaker cluster from highest to lowest similarity and recalculating the representative speaker embedding after merging. When a circular merging occurs, causing cluster identifier rearrangement, the post-processing unit updates the mapping table for the cluster identifiers and writes the mapping table into the audit record.

[0043] Further, after generating the merged speaker cluster, this step performs block-level reallocation, which is the process of segmenting the clean audio set into basic segments and reallocating them to the merged speaker cluster. Specifically, the post-processing unit segments each clean audio segment into basic segments of 1.5 seconds in length along the time axis and extracts segment embeddings for each basic segment. The segment embeddings are output and registered by the embedding extraction unit on the same speaker representation network. Subsequently, the post-processing unit calculates the cosine similarity of the segment embeddings of adjacent basic segments, and marks adjacent basic segments as different speaker boundaries when the cosine similarity is less than 0.5. The 1.5-second interval and the 0.5 threshold are the minimum set of core parameters for this step and are written into the post-processing configuration table. For each basic segment, the post-processing unit calculates the cosine similarity between the segment embedding of the basic segment and the representative speaker embedding of the merged speaker cluster, and assigns the basic segment to the speaker cluster with the highest cosine similarity, generating a redistributed segment set. When the highest cosine similarity corresponds to multiple speaker clusters, the post-processing unit makes a decision based on the priority of the cumulative duration field of the speaker clusters and records the decision reason code. The block-level redistribution process includes boundary consistency verification. The boundary consistency verification performs a secondary review on the basic segments near the boundaries of different speakers. The secondary review determines whether the boundary falls within the silence interval based on the energy curve of the basic segment and the silence mask. When the boundary falls within the silence interval, the boundary position is aligned to the center of the silence interval and the corresponding timestamp boundary field is updated. The silence mask comes from the silence segment annotation field of S200 and is inherited by this step as input.

[0044] Further, after obtaining the redistributed segment set, this step performs segment merging, which is an iterative merging process of basic segments with the same speaker cluster identifier and adjacent time. Specifically, the post-processing unit traverses the redistributed segment set in chronological order, splices adjacent basic segments with the same speaker cluster identifier to form candidate speech segments, and updates the cumulative duration of the candidate speech segments. When the cumulative duration of the candidate speech segments exceeds 27 seconds, the merging is terminated and the current candidate speech segment is output. When the silence interval between adjacent basic segments exceeds 2 seconds, the merging is terminated and the current candidate speech segment is output. The 27-second and 2-second thresholds are the minimum set of core parameters for this step and are fixed in the post-processing configuration table. The silence interval is calculated by the post-processing unit based on the silence mask, and the silence interval is the time length corresponding to the continuous silence sample sequence between adjacent basic segments. When the silence mask is missing or the silence mask cannot cover the boundary, the post-processing unit switches to determining the silence interval based on the energy threshold and writes the determination source field into the audit record. For the output candidate speech segments, this step encapsulates them as members of the speaker-consistent speech segment set, and writes the audio / video resource identifier, speaker cluster identifier, segment start and end timestamp field, segment duration field, audio segment path field, segment embedding summary field, and post-processing parameter version field into the speaker-consistent speech segment set index table; the post-processing parameter version field is associated with the configuration versions of the 0.6 threshold, 0.5 threshold, 1.5 second, 27 second, and 2 second threshold, so that the output product and the running process have a consistent version traceability relationship.

[0045] Further, in the output stage, this step performs segment validity checks on the speaker-consistent speech segment set. The segment validity check includes checking for non-zero segment duration, non-overlapping segment start and end timestamp fields, and readability of the audio segment path field. If the checks are not met, the corresponding speaker cluster identifier and segment start and end timestamp fields are written to an exception queue. After the validity check is completed, this step registers the speaker-consistent speech segment set as the input object for automatic transcription and glyph normalization processing by S400. Specifically, the processing status field in the segment set index table is updated to "Speaker-consistent speech segment generated," and a task message containing audio / video resource identifiers, audio segment path fields, and speaker cluster identifiers is sent to the downstream transcription queue, enabling S400 to subscribe to the task message and load the speaker-consistent speech segment set. In one engineering embodiment, the speaker segmentation server and the embedding extraction unit run on the same computing node. The segmentation scheduling unit uses a pipelined approach to first segment the same audio and video resource identifier, then perform embedding extraction and post-processing, and generates a unique processing batch number for each audio and video resource identifier. The processing batch number and the version field of the post-processing parameter are used together for cross-batch duplicate processing detection. When the file hash field of the same clean audio path is detected to be consistent with the historical record, the segmentation scheduling unit skips the duplicate processing and directly reuses the historical speaker consistent speech segment set index, thereby maintaining a stable automated processing rhythm in the engineering operation.

[0046] In summary, the technical effects of this step are as follows: This step forms a closed-loop link between speaker segmentation and embedding similarity post-processing on the clean audio set. Through continuous operation of cluster merging, block-level reallocation, and segment merging, it outputs a set of speaker-consistent speech segments. This step versionifies and solidifies the cosine similarity threshold, basic segment length, upper limit of candidate speech segment merging, and silence interval threshold, and synchronously records the post-processing parameter version field in the index table, so that subsequent steps can run consecutively on the same terminology and the same data structure. The set of speaker-consistent speech segments is registered as the input object of S400 and cross-step connection is achieved through task message delivery.

[0047] S400: Input the speaker-consistent speech segment set into automatic transcription and character normalization processing to generate a speech text annotation set; The speaker-consistent speech segment set is output by S300 and includes multiple speech segments and their associated metadata. The speech segments are single-channel speech data after consistency constraints are completed according to speaker cluster identifiers. The associated metadata includes a segment identifier field, a speaker cluster identifier field, a segment start and end time field, and a sampling rate field. Specifically, in this step, after receiving the speaker-consistent speech segment set, the transcription pipeline scheduling unit in the data processing server establishes a queue to be processed according to the segment identifier field, and writes a task status field and a retry count field when enqueuing the segment. When the task status field meets the requirements for processing and the corresponding speech segment file passes verification, the automatic transcription sub-process is triggered. The automatic transcription belongs to Automatic Speech Recognition (ASR) processing. The Paraformer-KO of FunASR is used to perform end-to-end transcription inference on the speech segments to obtain the transcribed text field and simultaneously obtain the transcription confidence field. The transcribed text field is a Korean character sequence, and the transcription confidence field is a segment-level confidence representation of the transcribed character sequence. To address the discontinuity of segment boundaries after long audio segmentation, this step performs segment-level input normalization on the speech segment before transcription inference. This normalization includes silence trimming and amplitude normalization. Silence trimming determines the silence interval based on short-time energy and zero-crossing rate, and trims it accordingly. Amplitude normalization calculates a normalization coefficient based on peak amplitude and performs linear scaling. When the usable speech interval length obtained from silence determination is lower than a preset lower limit, the task status field is set to abnormal and an abnormal reason field is written. The abnormal reason field indicates that the silence percentage exceeds the threshold. The speech segment is then temporarily stored in an isolated directory and exits subsequent processing in this step. Furthermore, to ensure consistency between the transcription output and the upstream speaker cluster identifier, this step binds and stores the segment identifier field and the speaker cluster identifier field together with the transcribed text field when writing to the transcribed text field, forming a segment-level transcription record. This segment-level transcription record serves as the input carrier for subsequent glyph normalization processing.

[0048] The glyph normalization process converts numbers, English words, and symbols in the transcribed text field into Korean glyphs. It employs N2gk+ to perform rule-driven text normalization and glyph mapping, generating a normalized text field. Specifically, after receiving the fragment-level transcription record, the glyph normalization module performs character category labeling on the transcribed text field. This labeling classifies characters into Korean characters, Arabic numerals, Latin letters, and symbols, and writes these categories into a category sequence field in the fragment-level transcription record. Subsequently, the category sequence field is segmented and merged to obtain a mixed fragment unit sequence. This sequence is then aggregated adjacently according to the same category to obtain number units, English units, and symbol units. For numeric units, this step converts the integer, separator, and decimal parts into corresponding Korean glyphs based on the Korean pronunciation rules table, and performs unit entry mapping when unit symbols are present. For English units, this step performs entry-level mapping based on a pre-set foreign language transliteration dictionary; when an entry is not matched, letter-level transliteration rules are executed and a Korean glyph sequence is output. For symbol units, this step performs retention or replacement processing based on a symbol whitelist; symbols outside the whitelist are deleted from the normalized text field and a deletion mark is written in the normalized difference record field. To avoid whitespace, repeated punctuation, or invisible control characters in the transliterated output affecting subsequent quality scoring filtering, this step performs whitespace compression and punctuation normalization on the normalized text field. Whitespace compression merges consecutive whitespace into a single whitespace, and punctuation normalization merges repeated punctuation and replaces them according to Korean writing conventions. The normalized difference record field is used to record the replacement position and replacement type between the transliterated text field and the normalized text field. The replacement type includes at least numeric replacement, English replacement, symbol replacement, and deletion mark, thereby making the normalization process auditable.

[0049] In the engineering embodiment, after the data processing server completes steps S100 to S300 on publicly available genuine Korean drama audio and video materials, it writes the obtained speaker-consistent speech segment set into distributed storage and generates a segment list file. The segment list file includes a segment identifier field, a speaker cluster identifier field, a file path field, and a segment start and end time field. In step S400, the pipeline scheduling unit periodically scans the newly added records in the segment list file and triggers a transcription task. The transcription task loads the Paraformer-KO model parameters by the inference computing node and infers the speech segments one by one. The inference output is written into the segment-level transcription record and handed over to the glyph normalization module for N2gk+ processing, finally forming a speech text annotation set and writing it into the dataset list. To support the stable operation of the pipeline, this step performs version management on the model and rule set. The model version field records the Paraformer-KO model parameter package identifier, and the normalization version field records the N2gk+ rule table and dictionary package identifier. Each record in the speech-text annotation set is simultaneously written to both the model version field and the normalization version field. When either the model version field or the normalization version field is updated, the pipeline scheduling unit generates an incremental rerun task and writes the rerun reason field into the task status field. The rerun reason field indicates the version change, thus enabling traceability of the output of the same speech segment under different versions. The speech-text annotation set consists of multiple annotation records, which include a segment identifier field, a speaker cluster identifier field, a transcribed text field, a normalized text field, a transcribed confidence field, a model version field, and a normalization version field. The speech-text annotation set is used as input to S500, which performs quality scoring filtering and segment duration constraint processing on the speech segments and the normalized text fields.

[0050] In summary, the technical effects of this step are as follows: By performing automatic speech recognition inference based on Paraformer-KO on the set of consistent speaker speech segments and generating a transcribed text field, segment-level speech and text binding annotations are formed. N2gk+ is used to perform Korean glyph conversion on the transcribed text field for numbers, English words, and symbols, generating a standardized text field, ensuring the annotated text has a uniform writing style while preserving differences. By writing model version and standardized version fields into the speech-text annotation set and linking them with the task status field, a traceable closed-loop operation of the transcription and standardization process is achieved, providing stable input to the S500.

[0051] S500: Input the speech-text annotation set into quality score filtering and segment duration constraint processing to generate a training dataset; The speech-text annotation set is output by S400 and serves as the direct input for this step. The speech-text annotation set contains multiple annotation records, each carrying at least a segment identifier field, a speaker cluster identifier field, an audio segment path field, a segment start and end time field, a transcribed text field, and a normalized text field. It may further include a transcribed confidence field, a model version field, and a normalized version field. Specifically, this step is triggered by the filtering and scheduling unit in the dataset construction server upon receiving an entry event for the speech-text annotation set. The entry event carries the dataset batch number and upstream version identifier. The filtering and scheduling unit first performs integrity checks on the annotation records. These integrity checks include accessibility checks for the audio segment path field, sequence checks for the segment start and end time fields, and non-empty checks for the normalized text field. The check results are written to the filtering audit record. For annotation records that fail the checks, the filtering and scheduling unit writes them to an isolation list and generates an exception reason field. This exception reason field indicates missing fields, unreadable paths, or abnormal time boundaries. The isolation list is then sent back to the runtime log corresponding to the dataset batch number, thus providing a traceable runtime trajectory for exception handling.

[0052] Further, this step proceeds to the quality scoring filtering stage. Quality scoring filtering is a processing chain that performs a computable representation of the subjective perceived quality of speech segments in the speech-text annotation set and deletes them accordingly. Specifically, the quality scoring filtering uses the UTMOS (Unsupervised Textless Mean Opinion Score) model to calculate a quality score field for each annotation record's corresponding speech segment. This quality score field is a scalar representation of the overall perceived listening quality of the speech segment and corresponds to the commonly used subjective average opinion score (MOS) metric. In this embodiment, the UTMOS model is deployed as a quality scoring engine, which consists of an audio loading unit, a feature calculation unit, and a scoring inference unit. The audio loading unit reads the speech waveform based on the audio segment path field and verifies that the sampling rate field is 48 kHz. The feature calculation unit performs frame conversion, energy statistics, and spectral characterization calculations on the speech waveform and forms the model input tensor. When the graphics processing unit is available, the scoring inference unit outputs the quality score field using batch inference. When the graphics processing unit is unavailable or the load exceeds the concurrency threshold, the scoring inference unit switches to central processing unit inference and reduces the batch size. At the same time, the inference device field and the batch size field are written to the screening audit record. After the quality scoring engine outputs a quality score field for each labeled record, it binds the quality score field with the segment identifier field and the speaker cluster identifier field and writes it into the scoring result table. Subsequently, the filtering and scheduling unit performs filtering judgment based on the scoring threshold field. In this embodiment, the scoring threshold field is set to 2.2. When the quality score field is less than 2.2, the corresponding labeled record is marked as removed and the removal reason field is written, indicating that the quality score is insufficient. When the quality score field cannot be calculated or the inference fails, the corresponding labeled record is marked as pending retry and the retry count field is incremented. When the retry count field exceeds a preset number of times, it is transferred to the isolation list. The scoring threshold field and the UTMOS model version field constitute the minimum set of core parameters for implementing quality scoring filtering in this step, and are fixed to the batch configuration snapshot during each batch processing to ensure version consistency in the quality scoring filtering process.

[0053] Further, after completing the quality scoring and filtering, this step enters the segment duration constraint processing stage. The segment duration constraint is a processing link that applies upper and lower bound constraints to the duration of speech segments and deletes them accordingly. Specifically, the screening scheduling unit calculates the segment duration field based on the segment start and end time field. When the segment start and end time field is missing or unreliable, it switches to counting the sampling points of the speech waveform corresponding to the audio segment path field and converting it into the segment duration field by combining it with the sampling rate field. At the same time, the duration source field is written into the screening audit record. The segment duration constraint is determined using a lower duration threshold field and a higher duration threshold field. In this embodiment, the lower duration threshold field is three seconds, and the higher duration threshold field is thirty seconds. When the segment duration field is less than three seconds or greater than thirty seconds, the corresponding annotation record is marked as removed and written into the removal reason field, which indicates that the duration exceeds the limit. The lower duration threshold field and the higher duration threshold field constitute the minimum set of core parameters for implementing segment duration constraints in this step, and together with the scoring threshold field, they form a batch configuration snapshot of "joint quality and duration screening". Understandably, to avoid the same audio segment being counted repeatedly in different screening rounds, this step establishes a primary key association between the segment identifier field and the duration result table, and generates a deduplication marker field during the removal process, so that the subsequent summary process only retains the records that have not been removed.

[0054] In the output phase, this step generates a training dataset by summarizing the labeled records filtered by quality scores and constrained by segment duration. This training dataset is a structured data set input to the S600 and is stored on disk or in a database as a training dataset list. Specifically, each data record in the training dataset contains at least a segment identifier field, a speaker cluster identifier field, an audio segment path field, a normalized text field, and a segment duration field, and further includes a quality score field, a model version field, a normalized version field, and a batch number field. The normalized text field and the audio segment path field serve as the direct input carriers for the S600 to construct the Mel spectrum from the training dataset, while the batch number field and each version field are used for cross-batch traceability. In one engineering embodiment, the dataset construction server processes the speech-text annotation set from publicly available, licensed Korean drama audio and video materials in batches according to dataset batch number. First, the quality scoring engine performs batch inference on the speech segments pointed to by the audio segment path field during off-peak hours and writes them into the scoring result table. Then, the screening and scheduling unit serially executes segment duration constraints and outputs a training dataset list. When the screening and scheduling unit detects a change in the upstream model version field or normalization version field, it triggers an incremental rerun based on the batch configuration snapshot and writes a rerun identifier field into the training dataset list, thereby maintaining an auditable evolution record for this step during continuous pipeline operation. Further, in addition to the scoring threshold field, duration lower limit threshold field, and duration upper limit threshold field, this step may optionally introduce transcription confidence field consistency verification and text anomaly regularization verification as extended functions. The extended functions are recorded as switch fields in the batch configuration snapshot, and their activation or deactivation does not change the minimum set of core parameters on which this step forms the training dataset.

[0055] In summary, this step achieves the following technical results: It establishes a continuous filtering chain of quality score filtering and segment duration constraints on the stated speech-text annotation set, using the quality score field and segment duration field as traceable processing records. By solidifying the score threshold field, lower duration threshold field, and upper duration threshold field into batch configuration snapshots and recording the model version field and normalized version field, a stable version association is formed between the training dataset and the runtime process. The training dataset, as the input object of S600, is output in a structured manner, completing the cross-step connection.

[0056] S600. Input the training dataset into the Mel spectrum construction, set the sampling rate to 48 kHz, the highest frequency to 16 kHz, and double the FFT window and frame shift parameters while keeping the length and dimension of the Mel frame sequence unchanged, and generate the Mel training feature set. The training dataset is output by the S500 and serves as the direct input for this step. The training dataset contains multiple training sample records, each carrying at least a segment identifier field, a speaker cluster identifier field, an audio segment path field, a normalized text field, a segment start and end time field, and a segment duration field. It may further include a quality score field, a batch number field, and an upstream version field. Specifically, this step is triggered by the feature generation scheduling unit in the feature construction server upon detecting an event where the training dataset list is written to disk or stored in the database. This disk or database writing event carries a batch number field and a sample quantity field. The feature generation scheduling unit first performs readability and sampling rate consistency checks on the speech segments pointed to by the audio segment path field and writes the check results into the feature construction audit record. For training sample records with a sampling rate other than 48 kHz or unreadable audio segments, the feature generation scheduling unit marks them as abnormal samples and writes them into the abnormal reason field. At the same time, the abnormal samples are written into the isolation list for traceability during subsequent batch reruns. For readable training sample records, the feature generation scheduling unit combines the segment identifier field and the audio segment path field into a feature task list and puts it into the parallel feature calculation queue. The parallel feature calculation queue uses batch scheduling when the graphics processing unit is available, and switches to the central processing unit and reduces the concurrency when the graphics processing unit is unavailable. The inference device field and the concurrency field are synchronously written into the feature construction audit record, thereby making the feature construction process traceable.

[0057] Further, this step enters the Mel spectrum construction processing chain. Mel spectrum construction is a process of converting speech waveforms into time-frequency representations for model training. The time-frequency representation uses a Mel-scale filter bank to aggregate spectral energy and form a Mel frame sequence. Specifically, the feature calculation unit loads the speech waveform according to the audio segment path field, and performs time clipping on the speech waveform when the segment start and end time fields exist. When the segment start and end time fields are missing or unreliable, boundary alignment is performed using the segment duration field and the audio timecode information. Subsequently, the feature calculation unit performs amplitude range normalization and DC component suppression on the clipped speech waveform, and records the peak marker field when abnormal peaks are detected. The amplitude range normalization and DC component suppression are optional extended functions controlled by the switch field in the batch configuration snapshot. Whether they are enabled or not does not change the minimum set of core parameters for this step. Subsequently, the feature calculation unit performs frame processing and frequency domain analysis on the speech waveform. This frequency domain analysis uses Fast Fourier Transform (FFT) to calculate the spectrum of each speech frame. The FFT window indicates the time window span for each frame participating in the spectrum calculation, and the frame shift parameter indicates the time advance step size between adjacent frames. In this embodiment, the FFT window and frame shift parameter are doubled relative to the 24 kHz configuration and are linked to the 48 kHz sampling rate. This ensures that even with an increased sampling rate, the frame advance rhythm and the coverage duration of each frame maintain the same time scale as the 24 kHz configuration, thus matching the constraint of "keeping the length and dimension of the Mel frame sequence unchanged." During this linkage process, the feature generation scheduling unit permanently writes the FFT window parameter field, frame shift parameter field, sampling rate field, and configuration version field into the batch configuration snapshot for subsequent experimental reproduction and auditing.

[0058] Furthermore, after obtaining the spectrum of each frame, the feature calculation unit sets a frequency upper limit based on the highest frequency of 16,000 Hz and configures the frequency range of the Mel filter bank to 0 to 16,000 Hz, so that the Mel spectrum construction only aggregates the spectral energy within the frequency range. Subsequently, the feature calculation unit performs power spectrum calculation on each frame spectrum and sums the filter bank energy through the Mel filter bank to obtain the Mel energy vector of each frame, and performs logarithmic compression and numerical stabilization processing on the Mel energy vector to form a Mel frame sequence. Understandably, the Mel frame sequence length is used to represent the number of time frames, and the dimension is used to represent the number of Mel frequency bands. In this embodiment, the number of Mel frequency bands is fixed by the Mel frequency band number parameter field in the batch configuration snapshot. The Mel frequency band number parameter field, as one of the core parameters for dimension preservation, together with the sampling rate field, the highest frequency field, the FFT window parameter field, and the frame shift parameter field, constitutes the necessary configuration for achieving "consistent feature morphology under high sampling rate input" in this step. To maintain consistency across batches, the feature generation scheduling unit outputs a Mel feature record for each training sample record, and writes the following fields into the Mel feature record: segment identifier field, speaker cluster identifier field, normalized text field, Mel frame sequence field, Mel frame digital segment, Mel frequency band digital segment, sampling rate field, highest frequency field, FFT window parameter field, frame shift parameter field, and feature version field. The normalized text field and Mel frame sequence field serve as the paired input carriers for the S700 to train the fully non-autoregressive diffusion converter that matches the Mel training feature set input stream. The segment identifier field is used for sample alignment and deduplication during the training phase, and the feature version field forms an end-to-end traceability link with the upstream version field. When the feature calculation unit detects that a speech segment is too short, causing frame conversion failure, or detects numerical anomalies causing the Mel energy vector to be unavailable, it marks the corresponding training sample record as a feature failure and writes it into the failure reason field, while simultaneously incrementing the retry count field. When the retry count field exceeds a preset number of times, it outputs an isolation list and writes the isolation list back to the running log corresponding to the batch number field, thus preventing abnormal samples from entering the Mel training feature set.

[0059] In the output phase, this step aggregates all Mel feature records constructed through feature building to generate a Mel training feature set. This Mel training feature set is a structured feature set for input to the S700 and is output as a two-level object: a feature list and a feature storage. Specifically, the feature list includes at least a fragment identifier field, a feature storage location field, a Mel frame digital segment, a Mel frequency band digital segment, a sampling rate field, a highest frequency field, a feature version field, and a batch number field. The feature storage includes at least a Mel frame sequence field corresponding one-to-one with the fragment identifier field and its metadata. After generating the Mel training feature set, the feature generation scheduling unit registers it as a "Mel training feature set batch object" and associates it with the batch number field of the training dataset. Simultaneously, it writes the hash identifier field of the batch configuration snapshot into the feature construction audit record, allowing the subsequent S700 training process to directly reference the same batch configuration snapshot. In one engineering embodiment, the feature construction pipeline operates automatically with the batch number field as the boundary: when the training dataset list is added to the database, a feature task order is automatically created and batch calculation is completed; when a change in the batch configuration snapshot or an upgrade of the feature version field is detected, an incremental rerun is automatically triggered and the existing feature storage location field is recorded as a version fork, while retaining the old version features to meet the requirements of auditable evolution, thereby enabling the Mel training feature set to maintain a stable traceability relationship during continuous construction and to complete the cross-step connection with S700.

[0060] In summary, this step achieves the following technical results: It establishes a Mel spectrum construction link for 48kHz speech on the training dataset and solidifies the sampling rate field, highest frequency field, FFT window parameter field, and frame shift parameter field into batch configuration snapshots. By doubling the FFT window and frame shift parameters relative to 24kHz while maintaining the length and dimension of the Mel frame sequence, a consistent organization of feature morphology is achieved under high sampling rate conditions. The Mel training feature set is provided as a traceable, field-based output for direct use by the S700, thereby completing input alignment and version association during the training phase.

[0061] S700: Train the fully non-autoregressive diffusion converter by inputting the Mel training feature set into the stream matching, perform real Mel spectrum prediction under masked speech conditions, update the model parameters according to the stream matching loss, and generate a speech clone model. The Mel training feature set is output by S600 and serves as the direct input for this step. The Mel training feature set includes Mel frame sequence fields corresponding one-to-one with the segment identifier field, Mel frame digital segments, Mel frequency band digital segments, normalized text fields, speaker cluster identifier fields, sampling rate fields, highest frequency fields, FFT window parameter fields, frame shift parameter fields, feature version fields, and batch number fields. Specifically, this step is triggered by the training job orchestration unit in the training orchestration server after detecting the registration completion event of the Mel training feature set batch object. The registration completion event carries the batch number field, feature version field, and batch configuration snapshot hash identifier field. Based on this, the training job orchestration unit generates a training job description and distributes it to the training nodes. The training job description includes at least the data loading configuration field, model structure configuration field, optimizer configuration field, random seed field, checkpoint saving strategy field, and runtime audit field. Furthermore, the training job orchestration unit performs consistency checks on the Mel training feature set. The consistency checks include Mel frequency band digital segment consistency checks, Mel frame digital segment value range checks, normalized text field non-empty checks, and fragment identifier field deduplication checks. When a sample record that does not meet the consistency check conditions is found, the sample record is written into the training isolation list and the isolation reason is recorded in the operation audit field. At the same time, the training batch continues to be constructed for the remaining sample records, so that the training process has a traceable anomaly handling path.

[0062] Furthermore, the flow matching fully non-autoregressive diffusion converter is a neural network training framework that performs conditional generation in the time-frequency feature domain. Here, "fully non-autoregressive" means that the target prediction of the Mel frame sequence does not depend on frame-by-frame recursive output but adopts a parallel output form. "Diffusion converter" means that the backbone network adopts a converter structure and performs denoising mapping on the noise-driven generation trajectory. "Flow matching" means that during the training phase, the continuous time generation flow is fitted and the flow matching loss is constructed by using Flow Matching. Specifically, the data loading unit on the training node reads the Mel frame sequence field and the normalized text field from the Mel training feature set. First, it performs character serialization processing on the normalized text field to obtain the character sequence field. The character sequence field is the result of mapping Korean characters to discrete symbol sequences according to a preset character table. The character table and word segmentation rules are fixed by the character table configuration field and archived along with the batch configuration snapshot hash identifier field. Subsequently, the data loading unit performs length alignment sampling on the Mel frame sequence field. The length alignment sampling is to trim or fill the time axis without changing the Mel frequency band digital segment and generate a length mask field. The length mask field records the effective frame interval and allows the subsequent loss aggregation stage to skip invalid frames. Furthermore, within each training iteration, the data loading unit samples from the Mel frame sequence field to generate a noisy speech field and a masked speech field. The noisy speech field is a noisy Mel sequence obtained by superimposing random noise within the effective frame interval of the Mel frame sequence. The masked speech field is a masked Mel sequence obtained by masking part of the time frame or part of the frequency band position of the noisy Mel sequence according to the mask ratio configuration field. The masking can be done by replacing the corresponding position with a preset padding value or by replacing it with the same distributed noise. The mask position set field is written into the runtime audit field during generation to facilitate the reuse of the same random seed field to generate a consistent masking trajectory during training and reproduction experiments.

[0063] Furthermore, the model training unit on the training node loads the fully non-autoregressive diffusion transformer and performs forward computation. The fully non-autoregressive diffusion transformer includes the ConvNeXt text feature extraction network (Convolutional Next, ConvNeXt), the diffusion transformer backbone with adaLN-0 (Adaptive Layer Normalization Zero-init, adaLN-0), Sinusoidal Positional Embedding, and Convolutional Positional Embedding. The ConvNeXt text feature extraction network receives character sequence fields and outputs text feature sequence fields, which are high-dimensional representations aligned to character positions. Sine position embedding and convolutional position embedding inject position-related information into the character sequence fields and masked speech fields, respectively. The former provides global position signals, and the latter provides local neighborhood position signals. These are written into the position fusion feature fields during the feature fusion stage. The diffusion converter backbone receives the concatenated input of the text feature sequence fields, noisy speech fields, and masked speech fields corresponding to the character sequence fields and outputs a predicted Mel field. The concatenated input is an input tensor formed by channel-level concatenation of text conditional features and noisy masked Mel features at the same time axis alignment scale. The predicted Mel field is an estimated sequence of the true Mel spectrum. Understandably, "performing true Mel spectrum prediction under masked speech conditions" corresponds to the model training unit calculating the difference between the predicted Mel field and the Mel frame sequence field under the masking constraint given by the masked speech field, and obtaining the prediction error field. The difference calculation is performed within the valid frame interval indicated by the length mask field to avoid invalid frames interfering with training updates. When the prediction error field is detected to have numerical overflow or be incalculable, the model training unit marks the current iteration as an abnormal iteration and writes it into the abnormal iteration identifier field. At the same time, it triggers the gradient skip strategy field, which means that parameter updates are not performed in this iteration and the process continues to the next iteration, thereby maintaining the continuous operation of the training process.

[0064] Furthermore, the model training unit constructs a flow matching loss based on the predicted Mel field, the noisy speech field, the masked speech field, and the Mel frame sequence field. This flow matching loss is a loss term that aggregates the deviations between the generated flow field and the target flow field. Its calculation process includes generating a sampling time identifier field, a flow space mapping field, and a loss aggregation field. Specifically, within each iteration, the model training unit generates a sampling time identifier field and maps the noisy Mel sequence to a flow space mapping field. This flow space mapping field converts the noise-driven state representation into a representation suitable for flow field matching. Subsequently, the model training unit calculates the deviation between the predicted Mel field's representation in the flow space mapping field and the target representation to obtain a flow matching error field. The flow matching error field is then aggregated within the effective frame interval to obtain a flow matching loss scalar field. The "update model parameters based on flow matching loss" refers to the optimizer unit reading the flow matching loss scalar field and performing backpropagation to obtain the gradient field. Subsequently, gradient clipping is performed on the gradient field and written into the clipping marker field. Then, the model parameters are updated based on the learning rate configuration field and the optimizer configuration field. The learning rate configuration field can be a fixed learning rate policy field or a segmented decay policy field. The optimizer configuration field includes at least a momentum configuration field and a weight decay configuration field. During the training process, the batch number field, feature version field, random seed field, current iteration field, flow matching loss scalar field, checkpoint number field, and model structure configuration field are continuously recorded in the runtime audit field, thereby making the model update trajectory auditable. When the early stopping condition field is met, the training job orchestration unit triggers a training stop and enters the model encapsulation stage. The early stopping condition field is a set of rules based on the stability determination of the loss window based on the validation subset or based on the maximum number of iterations. The validation subset is obtained by hashing and bucketing the Mel training feature set according to the segment identifier field using the batch partitioning strategy field and is archived along with the batch number field.

[0065] During the output phase, the model encapsulation unit encapsulates the trained network weights, word list configuration fields, batch configuration snapshot hash identifier fields, model structure configuration fields, and inference preprocessing configuration fields to generate a speech clone model. The speech clone model serves as the input object of the S800 and includes model checkpoint path fields, model version fields, feature version fields, word list version fields, and training batch number fields. Among them, the model checkpoint path field points to the weight file in persistent storage, and the model version field is generated by the version management unit based on the timestamp and commit record and written to the version index library. The version management unit retains at least two historical versions for rollback and records the parent version field and difference summary field in the version index library, thereby supporting stable evolution during training strategy iteration. In one engineering embodiment, after the training orchestration server receives the registration completion event of a new batch of Mel training feature sets, it automatically creates a training job and automatically triggers the encapsulation, indexing, and publishing of the speech clone model after training is completed. When the publishing of the model fails or the checkpoint verification fails, the system marks the speech clone model as pending review and prohibits it from entering the S800 call path. At the same time, it records the reason for failure in the running audit field and generates a rerun task sheet, thereby completing the cross-step connection and maintaining the traceable operation of the entire link.

[0066] This invention employs a fully non-autoregressive diffusion converter based on flow matching, with an architecture similar to F5-TTS, but innovatively introduces a real target prediction strategy. Specifically, the model directly predicts the real Mel spectrum using a concatenated input of character sequences, noisy speech, and masked speech. and through flow matching loss Update parameters.

[0067]

[0068] in, Represents the mathematical expectation; Represents the square of the L2 norm; Indicates the flow matching model in time ,state The predicted "flow"; Represents the learnable parameters of the model; Indicates time Data status; Indicates the standardized time step; Represents the actual target "flow"; Indicates the model in time ,state The actual Mel spectrum directly predicted below; This represents a true, noise-free Mel spectrum target.

[0069] This loss function ensures efficient training while adapting to high sampling rates (48kHz): the highest frequency of the Mel spectrum is increased to 16kHz, and the FFT parameters are doubled, but the frame sequence length and dimension remain unchanged to avoid increasing the number of model parameters. During inference, the prediction results are sampled as waveforms through stream space transformation and ordinary differential equations.

[0070] In summary, this step establishes a flow-matching, fully non-autoregressive diffusion converter training pipeline on the Mel training feature set, and incorporates real Mel spectrum predictions under masked speech conditions into the training iterations. Through flow-matching loss-driven parameter updates and checkpoint version management, the speech clone model forms a traceable training trajectory and reproducible experimental configuration. The speech clone model encapsulates its output and aligns it with the S800 input object fields, thus completing the transition from training to generation.

[0071] S800: Align the speech cloning model with the input length of the reference speech and target text, generate noise reduction, perform stream space transformation and ordinary differential equation sampling processing to generate a 48 kHz cloned speech waveform. Specifically, the speech cloning model originates from the flow matching training output of the S700. The initial flow matching is called FlowMatching, which internally includes a character feature extraction network, a diffusion converter backbone, and a position embedding module. On the inference side, it executes an integrated chain of "text length alignment—denoising generation—flow space transformation—ordinary differential equation sampling." The reference speech is the speech waveform data of the speaker to be imitated, and the target text is the Korean character sequence data to be synthesized. Both are submitted to the inference service by the business caller in a single cloning request. The inference service assigns a request identifier to this request and reads the model version identifier, model weight verification information, and inference configuration parameters corresponding to the speech cloning model from the model registry. The inference configuration parameters include at least a sampling rate of 48 kHz, a maximum frequency of 16 kHz, a Mel filter frequency range, and ordinary differential equation sampling step parameters. These inference configuration parameters are consistent with the Mel spectrum construction of the S600, ensuring consistent use for subsequent Mel feature encoding and waveform reconstruction.

[0072] Further, the inference service performs input normalization on the reference speech, which includes unifying the number of channels, normalizing waveform amplitude, coarsely screening silence segments, and correcting the sampling rate. When the sampling rate of the reference speech is not 48 kHz, the resampling unit is invoked to complete the sampling rate conversion, and the conversion record is written to the request audit record. Subsequently, the inference service performs Mel spectrum construction on the reference speech. The Mel spectrum construction reuses the frequency upper limit and window step size relationship determined by S600, outputs a reference Mel spectrum sequence, and records this sequence as the reference speech Mel feature field. In parallel, the inference service performs glyph normalization processing on the target text. The glyph normalization processing reuses the normalization rules of S400, mapping numbers, convertible English words and symbols to Korean glyphs, and outputs a normalized text field. After the normalized text field is generated, the inference service invokes the text segmentation unit to perform word segmentation and tag mapping on the Korean character sequence, generates a character token sequence, and sends the character token sequence into the character feature extraction network of the speech cloning model, outputting a text feature sequence field.

[0073] During the length alignment phase, the inference service calls the length alignment unit to establish a length correspondence between the text feature sequence field and the reference speech Mel feature field. Specifically, the length alignment unit generates alignment length parameters based on the number of tokens in the character token sequence, the number of frames in the reference speech Mel feature field, and a preset alignment strategy, and pads the character token sequence into an aligned character sequence. The padding uses a padding token insertion method to extend the length of the aligned character sequence to an alignment length that matches the length of the input speech. The padding tokens are mapped as non-speech placeholder features on the model side, which are used to drive the diffusion converter backbone to perform conditional generation on a fixed frame grid. Subsequently, the inference service constructs a denoising generation input, which includes at least the aligned character sequence, a noisy speech feature field, and a masked speech feature field. The noisy speech feature field is generated by the noise initialization unit on the alignment length, and the masked speech feature field is obtained by the mask generation unit writing a mask region onto the reference speech Mel feature field. The mask region is used to represent the target speech segment that needs to be completed by the model. The mask generation unit determines the start and end frames of the mask based on the target text length, the available frame range of the reference speech, and the mask ratio parameters, and writes boundary markers at the mask boundaries so that the diffusion converter backbone can distinguish between the reserved segments and the segments to be generated during the inference process.

[0074] In the denoising generation stage, the inference service feeds the denoising generation input into the diffusion converter backbone of the speech cloning model. Under the flow step size condition and position embedding constraint, the diffusion converter backbone outputs the true Mel prediction field and writes it into the generation buffer. Subsequently, in the flow space transformation stage, the inference service calls the flow space transformation unit to perform a representation transformation on the true Mel prediction field, generating a flow space state field. This flow space state field is a state variable representation of the Ordinary Differential Equation (ODE), containing the current flow state value, sampling step number, and step size scheduling parameters. During the ODE sampling stage, the ODE solver is invoked. The ODE solver iteratively updates the flow space state field in multiple steps according to the step size scheduling parameters, writing the intermediate state into a sampling trajectory record at each step. This sampling trajectory record includes the step number, state summary, and numerical stability flag. When the numerical stability flag triggers an anomaly threshold, the ODE solver reverts to the previous stable state and reduces the step size to continue iterating until the sampling termination condition is met, at which point it outputs the generated Mel spectrum field.

[0075] During the waveform synthesis stage, the inference service calls the vocoder unit to convert the generated Mel spectrum field into a 48 kHz cloned speech waveform. The vocoder unit is a Mel spectrum-to-waveform generation network, taking a Mel spectrum frame sequence as input and a time-domain waveform sampling point sequence as output, and recording the output as a 48 kHz cloned speech waveform field. Further, the inference service performs loudness normalization and endpoint trimming on the 48 kHz cloned speech waveform field, writes it to an audio container, and generates an output file path field. Simultaneously, it associates and stores the output file path field, model version identifier, and request audit record. The 48 kHz cloned speech waveform field is returned to the business caller as the output product of S800 and simultaneously sent to an offline evaluation queue for subsequent sampling and regression verification. The offline evaluation queue records the mapping relationship between the target text, the reference speech, and the 48 kHz cloned speech waveform.

[0076] In the engineering implementation, the speech cloning service is deployed on the inference server cluster. After the client uploads the reference speech and inputs the target text in the interactive interface, a cloning request is triggered. After receiving the request, the inference service automatically completes the reference speech normalization, target text glyph normalization, length alignment, denoising generation, stream space transformation and ordinary differential equation sampling, and generates a 48 kHz cloned speech waveform for playback. Each time a new model version is released, the inference service writes a new version identifier and rollback pointer to the model registry. Online requests select the model version identifier according to the version routing strategy. Audit records are retained for a long time and support the reproduction of the inference link according to the request identifier, thus forming a traceable version management strategy.

[0077] In summary, the technical effects of this step are as follows: By aligning the target text with the input speech frame grid on the inference side, and concatenating real Mel prediction, stream spatial transformation, and ordinary differential equation sampling into a single generation chain, a 48kHz cloned speech waveform is output. Through a combination of alignment strategies, mask region definitions, and sampling trajectory recording, the input, state, and output relationships of the inference process are verifiable and recordable. Through the linkage of model version identification, request audit logs, and offline evaluation queues, the inference chain exhibits traceability and replayability throughout its evolution.

[0078] Example 2: Figure 2 A block diagram of a high-sampling-rate Korean speech cloning system based on zero samples, according to an embodiment of the present invention, is shown. Figure 2 As shown, the structure may include: The data acquisition module 01 is used to acquire Korean voice and video data and extract 48 kHz audio to generate an original audio set. The output of the data acquisition module is connected to the input of the speech enhancement module. Specifically, the data acquisition module receives Korean voice and video data as input, performs audio-video stream separation processing on the Korean voice and video data and extracts the audio stream, verifies the sampling rate of the extracted audio stream, adjusts the sampling rate to 48 kHz, and writes it into the original audio set. The original audio set contains audio segments organized by segment granularity and their correspondence information with the Korean voice and video data. The correspondence information is output along with the original audio set. When the Korean voice and video data is missing, damaged, or the audio stream cannot be extracted, the data acquisition module discards the input object and writes an exception record. The data acquisition module provides the original audio set to the input of the speech enhancement module through its output, so that the speech enhancement module can call it and enter the speech enhancement processing flow.

[0079] The speech enhancement module 02 is used to input the original audio set into the speech enhancement processing to generate a clean audio set. The output of the speech enhancement module is connected to the input of the speaker segmentation and embedding similarity post-processing module. Specifically, the speech enhancement module reads audio segments from the original audio set output by the data acquisition module as input objects, performs framing and windowing processing on each audio segment to form a time-frequency representation for speech enhancement processing, and performs noise suppression, reverberation suppression, and energy normalization processing on this time-frequency representation before reconstruction and synthesis to generate clean audio segments that correspond one-to-one with the input audio segments and collect them into the clean audio set. The speech enhancement module verifies the sampling rate, amplitude saturation, and silence ratio of the input audio segment during processing. When the sampling rate is inconsistent, the amplitude is abnormal, or the silence ratio exceeds the preset conditions, the speech enhancement module marks the audio segment and writes an anomaly record. At the same time, it still incorporates the processed product into the clean audio set and retains the marking information. The speech enhancement module outputs the clean audio set to the input of the speaker segmentation and embedding similarity post-processing module through its output terminal, so that the speaker segmentation and embedding similarity post-processing module reads and performs speaker segmentation and embedding similarity post-processing at the audio segment granularity.

[0080] The speaker segmentation and embedding similarity post-processing module 03 is used to input the clean audio set into speaker segmentation and perform embedding similarity post-processing to generate a speaker-consistent speech segment set. The output of the speaker segmentation and embedding similarity post-processing module is connected to the input of the automatic transcription and character normalization processing module. Specifically, the speaker segmentation and embedding similarity post-processing module receives the clean audio set from the speech enhancement module as input, performs speaker segmentation processing on the clean audio set and outputs initial speaker segments. Simultaneously, it extracts speaker embeddings from the initial speaker segments and calculates the speaker embedding cosine similarity. Speaker clusters with a cosine similarity greater than 0.6 are clustered and merged to form merged speaker clusters. The embedding similarity post-processing includes segmenting the clean audio set into 1.5-second basic segments and extracting segment embeddings for each basic segment. Adjacent basic segments with a segment embedding cosine similarity less than 0.5 are marked as different. The speaker boundary is defined, and each basic segment is assigned to the speaker cluster corresponding to the one with the highest cosine similarity between the segment embedding and the speaker embedding of the merged speaker cluster, generating a redistributed segment set. The redistributed segment set is then merged with adjacent basic segments according to the speaker cluster identifier to obtain candidate speech segments. When the cumulative duration of the candidate speech segments exceeds 27 seconds or the silence interval between adjacent basic segments exceeds 2 seconds, the merging is terminated and the speaker-consistent speech segment set is output. When the clean audio set has speaker segmentation failure, missing speaker embedding, or abnormal cosine similarity calculation, the speaker segmentation and embedding similarity post-processing module performs discarding or rollback to the adjacent basic segment merging boundary processing on the relevant segments and writes the abnormal record. The speaker segmentation and embedding similarity post-processing module provides the speaker-consistent speech segment set to the input of the automatic transcription and glyph normalization processing module through its output end, so that the automatic transcription and glyph normalization processing module can call it to generate a speech text annotation set.

[0081] The automatic transcription and glyph standardization processing module 04 is used to automatically transcribe the speaker-consistent speech segment set and perform glyph standardization processing on the transcribed text to generate a speech-text annotation set. The output of the automatic transcription and glyph standardization processing module is connected to the input of the quality scoring filtering and segment duration constraint processing module. Specifically, the automatic transcription and glyph standardization processing module reads speech segments from the speaker-consistent speech segment set output by the speaker segmentation and embedding similarity post-processing module as input objects, performs automatic transcription processing on each speech segment to generate transcribed text, and simultaneously performs glyph standardization processing on the transcribed text, converting numbers, English words, and symbols into Korean glyphs and generating standardized text. The automatic transcription and glyph standardization processing module then... The transcribed text is bound to the corresponding speech segment to form the speech text annotation set, which contains the correspondence information between the speech segment, the transcribed text, and the normalized text. When the automatic transcription processing output is empty, the output does not match the duration of the speech segment, or there are characters that cannot be mapped in the character normalization processing, the automatic transcription and character normalization processing module marks the speech segment and writes an exception record, and at the same time, the marking information is incorporated into the speech text annotation set. The automatic transcription and character normalization processing module outputs the speech text annotation set to the input of the quality scoring filtering and segment duration constraint processing module through its output terminal, so that the quality scoring filtering and segment duration constraint processing module can perform quality scoring filtering and segment duration constraint processing based on the speech text annotation set.

[0082] The quality score filtering and segment duration constraint processing module 05 is used to input the speech text annotation set into the quality score filtering and perform segment duration constraint processing to generate a training dataset. The output of the quality score filtering and segment duration constraint processing module is connected to the input of the Mel spectrum construction module. Specifically, the quality score filtering and segment duration constraint processing module receives the speech text annotation set output from the automatic transcription and character normalization processing module as input, calculates the quality score of the speech segments in the speech text annotation set, and binds the quality score with the corresponding normalized text. The quality score filtering includes deleting speech segments with a quality score less than 2.2 and simultaneously deleting their corresponding normalized text bindings. The segment duration constraint processing includes deleting speech segments with a duration of less than three seconds or a duration of more than thirty seconds and simultaneously deleting their corresponding normalized text binding relationships. When the speech text annotation set has missing speech segments, missing normalized text, or unavailable duration information, the quality score filtering and segment duration constraint processing module deletes the relevant entries and writes them into an anomaly record. The speech segments retained after the quality score filtering and segment duration constraint processing, together with their corresponding normalized text, form the training dataset. The training dataset is transmitted to the input of the Mel spectrum construction module through the output of the quality score filtering and segment duration constraint processing module, so that the Mel spectrum construction module can read it and generate a Mel training feature set.

[0083] The Mel spectrum construction module 06 is used to input the training dataset into the Mel spectrum construction module and set the sampling rate to 48 kHz and the highest frequency to 16 kHz. It doubles the Fast Fourier Transform window parameters and frame shift parameters while maintaining the Mel frame sequence length and dimension, generating a Mel training feature set. The output of the Mel spectrum construction module is connected to the input of the stream-matched fully non-autoregressive diffusion converter training module. Specifically, the Mel spectrum construction module reads speech segments and their corresponding normalized text from the training dataset output by the quality scoring filtering and segment duration constraint processing module as input objects. It performs sampling rate verification on the speech segments while maintaining the sampling rate at 48 kHz, performs frame processing on the speech segments, and performs spectrum calculation according to the configuration of doubling the Fast Fourier Transform window parameters and frame shift parameters, limiting the spectrum calculation results to the highest frequency. The high-frequency range of 16,000 Hz is mapped to the Mel spectrum. The Mel spectrum construction module, while maintaining the Mel frame sequence length and dimension, aligns the Mel spectra of different speech segments, forming a unified Mel training feature set at the speech segment granularity. This Mel training feature set is then bound to the corresponding normalized text. When the training dataset exhibits abnormal speech segment sampling rates, Mel spectrum construction failures, or Mel frame sequence length and dimension not meeting constraints, the Mel spectrum construction module deletes the relevant entries and writes an error record. The Mel training feature set is transmitted from the output of the Mel spectrum construction module to the input of the flow-matching fully non-autoregressive diffusion converter training module, which then calls and generates a speech clone model.

[0084] The stream-matching fully non-autoregressive diffusion converter training module 07 is used to input the Mel training feature set into the stream-matching fully non-autoregressive diffusion converter for training. Under masked speech conditions, it performs real Mel spectrum prediction and updates model parameters based on the stream matching loss to generate a speech clone model. The output of the stream-matching fully non-autoregressive diffusion converter training module is connected to the input of the length alignment and denoising generation, stream space transformation, and ordinary differential equation sampling processing module. Specifically, the stream-matching fully non-autoregressive diffusion converter training module receives the Mel training feature set output from the Mel spectrum construction module as input, reads the normalized text binding relationship corresponding to the speech segments in the Mel training feature set as training conditions, performs masked speech condition construction on the Mel training feature set, constructs the Mel training feature set into input samples containing masked speech, and performs real Mel spectrum prediction on these input samples to obtain the predicted Mel spectrum. The stream-matching fully non-autoregressive diffusion converter training module... The diffusion converter training module performs difference calculations on the predicted Mel spectrum and the real Mel spectrum of the Mel training feature set to generate a flow matching loss. Based on this flow matching loss, it updates the model parameters. The update process iteratively reads the Mel training feature set in batches until a preset termination condition is met. When the Mel training feature set has missing binding relationships, failed mask speech condition construction, or an abnormal flow matching loss, the flow matching fully non-autoregressive diffusion converter training module skips the relevant samples and writes an error record, while maintaining the continuous operation of the training chain for the remaining samples. After the model parameters are updated, a speech clone model is output. This speech clone model is transmitted through the output of the flow matching fully non-autoregressive diffusion converter training module to the input of the length alignment and denoising generation, flow space transformation, and ordinary differential equation sampling processing module. This module then calls upon the length alignment and denoising generation, flow space transformation, and ordinary differential equation sampling processing module to generate a 48 kHz cloned speech waveform.

[0085] The length alignment and denoising generation, stream space transformation, and ordinary differential equation sampling processing module 08 is used to perform length alignment and denoising generation, stream space transformation, and ordinary differential equation sampling processing on the input of the speech cloning model, reference speech, and target text to generate a 48 kHz cloned speech waveform. Specifically, the length alignment and denoising generation, stream space transformation, and ordinary differential equation sampling processing module receives the speech cloning model output from the stream-matched fully non-autoregressive diffusion converter training module as input, and simultaneously receives the reference speech and target text as inference input. It performs representation construction on the reference speech consistent with the input constraints of the speech cloning model, and establishes a length alignment relationship between the target text and the representation of the reference speech to form the alignment input required for denoising generation. The denoising generation performs progressive denoising under the alignment input constraints and generates intermediate generation products, which then enter the stream space transformation. The process generates a stream space representation, which is solved in the ordinary differential equation sampling processing link according to a preset sampling step sequence and outputs a 48 kHz cloned speech waveform. When the reference speech is missing, the target text is empty, or length alignment fails, the length alignment and denoising generation, stream space transformation, and ordinary differential equation sampling processing module discards the inference input and writes it into an exception record. The 48 kHz cloned speech waveform is output as the final output product by the length alignment and denoising generation, stream space transformation, and ordinary differential equation sampling processing module, and is bound to the reference speech and target text for subsequent retrieval and tracing.

[0086] Figure 3 This application provides a flowchart of a method for processing Korean TV drama audio data, detailing the preprocessing pipeline from raw video to high-quality training segments. Figure 3 As shown, the process begins in step S310, downloading publicly available, officially licensed Korean drama video resources. In step S320, FFmpeg is used to extract audio signals with a sampling rate of 48kHz from the video. Next, in step S330, a pre-trained speech enhancement model (such as MossFormer2_SE_48K) is used to denoise and sharpen the audio.

[0087] The process makes a judgment in step S340 to determine whether to perform speaker segmentation optimization. If optimization is selected, steps S341, S342 and S343 are executed sequentially: Step S341 (speaker cluster merging) calculates the embeddings of the initial speaker clusters and merges clusters with a cosine similarity exceeding a preset threshold (e.g., 0.6); Step S342 (block-based reallocation) divides the audio into basic segments of fixed duration (e.g., 1.5 seconds) and reallocates speaker labels based on the cosine similarity of speaker embeddings between segments (e.g., below 0.5 as the limit); Step S343 (segment merging strategy) iteratively merges adjacent segments belonging to the same speaker and terminates merging when the total duration of the merged segments exceeds the upper limit (e.g., 27 seconds) or the silence interval between adjacent segments exceeds the threshold (e.g., 2 seconds), thereby generating longer segments with higher speaker consistency.

[0088] Regardless of whether steps S341 to S343 have been optimized, the process proceeds to step S350, where text is extracted using an automatic speech recognition model (such as Paraformer-KO), and normalization tools (such as N2gk+) are used to convert numbers, English letters, etc., in the text into Korean characters. Subsequently, in step S360, the processed speech segments are assessed for quality (e.g., using the UTMOS model with a threshold of 2.2). If the score does not meet the requirements, step S361 is executed to remove the low-quality sample; if the score meets the requirements, the process ends, and the segment is included in the final training dataset. This flowchart clearly illustrates the key steps, judgment logic, and optimization path of data preprocessing, and is the core process for constructing a high-quality, high-sampling-rate Korean speech clone dataset.

[0089] Figure 4 This application provides a model inference flowchart that clearly illustrates the data flow and core processing steps of the inference stage of a high-sampling-rate Korean speech cloning system. As shown in the figure, the inference process begins with input preparation on the left: the reference text and the target text are first concatenated, then deep features are extracted by the Conv2NeXT module and upsampled by 2x to adapt to the dimensionality requirements of subsequent modules; simultaneously, the reference audio is converted into high-frequency resolution Mel-spectral features by the 48k Mel-spectral extraction module. The processed text features, Mel-spectral features, and the input noise and target length noise to be predicted are fused in the concatenation module to form the complete input of the model.

[0090] The concatenated data is then input into the core DiT (Diffusion Transformer) model. The DiT model performs denoising and feature transformation on the input based on a flow matching strategy, and its output is subsequently fed into the ODE (Ordinary Differential Equation) solver. The ODE solver samples the probability flow through numerical integration, transforming the model prediction into the data space. Finally, the sampling result is reconstructed by the vocoder decoding module, outputting the final 48kHz cloned speech waveform. This flowchart fully presents the end-to-end inference chain from multimodal input to high-quality speech waveform generation, and concretizes the "length alignment, denoising generation, flow space transformation, and ordinary differential equation sampling" processes described in the claims and S800 steps.

[0091] Although the invention has been described herein in conjunction with various embodiments, those skilled in the art will understand and implement other variations of the disclosed embodiments by reviewing the accompanying drawings, the disclosure, and the appended claims in carrying out the claimed invention. In the claims, the word "comprising" does not exclude other components or steps, and "a" or "an" does not exclude a plurality. A single processor or other unit can implement several functions listed in the claims. While different dependent claims may recite certain measures, this does not mean that these measures cannot be combined to produce good results.

[0092] Although the invention has been described in conjunction with specific features and embodiments, it is obvious that various modifications and combinations can be made therein without departing from the spirit and scope of the invention. Accordingly, this specification and drawings are merely exemplary descriptions of the invention as defined by the appended claims, and are considered to cover any and all modifications, variations, combinations, or equivalents within the scope of the invention. Clearly, those skilled in the art can make various alterations and modifications to the invention without departing from its spirit and scope. Thus, if such modifications and modifications of the invention fall within the scope of the claims and their equivalents, the invention is also intended to include such modifications and modifications.

Claims

1. A method for cloning Korean speech at a high sampling rate based on zero samples, characterized in that, include: S100: Acquire Korean voice and video data and extract 48 kHz audio to generate the original audio set; S200: Input the original audio set into speech enhancement processing to generate a clean audio set; S300: Input the clean audio set into speaker segmentation and embedding similarity post-processing to generate a speaker-consistent speech segment set; S400: Input the speaker-consistent speech segment set into automatic transcription and character normalization processing to generate a speech text annotation set; S500: Input the speech-text annotation set into quality score filtering and segment duration constraint processing to generate a training dataset; S600. Input the training dataset into the Mel spectrum construction, set the sampling rate to 48 kHz, the highest frequency to 16 kHz, and double the FFT window and frame shift parameters while keeping the length and dimension of the Mel frame sequence unchanged, and generate the Mel training feature set. S700: Train the fully non-autoregressive diffusion converter by inputting the Mel training feature set into the stream matching, perform real Mel spectrum prediction under masked speech conditions, update the model parameters according to the stream matching loss, and generate a speech clone model. S800: Based on the speech cloning model, perform alignment and denoising, stream space transformation and ordinary differential equation sampling processing to generate a 48 kHz cloned speech waveform.

2. The method according to claim 1, characterized in that, S100 includes: acquiring an open-source 48kHz Korean speech dataset and publicly available, genuine Korean drama audio and video materials; performing FFmpeg audio extraction and resampling to 48kHz on the Korean drama audio and video materials to generate the original audio set.

3. The method according to claim 1, characterized in that, The speech enhancement processing in S200 uses MossFormer2_SE_48K to perform speech enhancement on the original audio set to generate the clean audio set.

4. The method according to claim 1, characterized in that, Speaker segmentation in S300 uses PyAnnote-audio to output initial speaker segments from the clean audio set; speaker embeddings are extracted from the initial speaker segments, and speaker clusters with a cosine similarity greater than 0.6 are clustered and merged to generate merged speaker clusters.

5. The method according to claim 4, characterized in that, The embedding similarity post-processing in S300 includes: dividing the clean audio set into 1.5-second basic segments and extracting segment embeddings for each basic segment; marking adjacent basic segments with a segment embedding cosine similarity of less than 0.5 as boundaries of different speakers; assigning each basic segment to the speaker cluster with the largest segment embedding cosine similarity to the speaker embedding of the merged speaker cluster, thereby generating a redistributed segment set.

6. The method according to claim 5, characterized in that, The embedding similarity post-processing in S300 further includes: merging adjacent basic segments of the redistributed segment set according to the speaker cluster identifier to obtain candidate speech segments; terminating the merging when the cumulative duration of the candidate speech segments exceeds 27 seconds or the silence interval between adjacent basic segments exceeds 2 seconds, and outputting the speaker-consistent speech segment set.

7. The method according to claim 1, characterized in that, The automatic transcription in S400 uses FunASR's Paraformer-KO to generate transcribed text from the speaker-consistent speech fragment set; the glyph normalization in S400 uses N2gk+ to convert numbers, English words and symbols in the transcribed text into Korean glyphs, generating normalized text, and combining the normalized text with the corresponding speech fragments to form the speech text annotation set.

8. The method according to claim 1, characterized in that, The quality score filtering in S500 uses UTMOS to calculate the quality score of speech segments in the speech-text annotation set; speech segments with a quality score less than 2.2 are deleted; the segment duration constraint in S500 deletes speech segments with a duration less than 3 seconds or a duration greater than 30 seconds, and generates the training dataset.

9. The method according to claim 1, characterized in that, In S600, the Mel spectrum construction setting is configured with the Mel filter frequency range from 0 to 16000 Hz, and the FFT points, window length parameters and frame shift parameters used for Mel spectrum calculation are doubled relative to the 24 kHz configuration, while keeping the number of Mel spectrum frames and Mel frequency bands unchanged, to generate the Mel training feature set; The fully non-autoregressive diffusion converter in S700 includes a ConvNeXt text feature extraction network, a diffusion converter backbone with adaLN-0, sinusoidal position embedding, and convolutional position embedding; the true Mel spectrum prediction in S700 is performed on concatenated inputs of character sequences, noisy speech, and masked speech; the ordinary differential equation sampling in S800 includes mapping the denoised generation result to the stream space and performing ordinary differential equation solving to output the 48 kHz cloned speech waveform.

10. A high-sampling-rate Korean speech cloning system based on zero samples, characterized in that, include: The system comprises a data acquisition module, a speech enhancement module, a speaker segmentation and embedding similarity post-processing module, an automatic transcription and glyph normalization module, a quality score filtering and segment duration constraint processing module, a Mel spectrum construction module, a stream matching fully non-autoregressive diffusion converter training module, a length alignment and denoising generation module, and a stream space transformation and ordinary differential equation sampling processing module; these modules are connected in sequence to implement the method described in any one of claims 1-9.

Images