Power dispatch voice processing method and system based on voiceprint clustering and voice cloning

Abstract

The application discloses a power dispatch voice processing method and system based on voiceprint clustering and voice cloning. The method comprises the following steps: obtaining original voice data of a power dispatch scene, performing voice activity detection and segmentation on the original voice data to obtain a plurality of audio segments, and performing initial voice recognition on the audio segments to obtain corresponding initial recognition text; performing voiceprint feature extraction on the audio segments, and clustering the extracted voiceprint features based on an unsupervised clustering algorithm to distinguish different speakers and generate speaker labels; and generating structured dialogue text based on a knowledge base in the power dispatch field and a preset dialogue template. Through the implementation of the application, the problem of high artificial annotation cost is solved, the training corpus scale is expanded, the training efficiency and recognition accuracy of the voice recognition model in the power dispatch scene are significantly improved, and the dependence on real annotation data is reduced.

Description

Technical Field

[0001] This invention relates to the field of speech processing technology, and in particular to a power dispatch speech processing method and system based on voiceprint clustering and speech cloning. Background Technology

[0002] With the continuous advancement of smart grid construction, the requirements for automation and intelligence in the power dispatching field are increasing. As a crucial foundation for automated recording of dispatching instructions, monitoring of violations, and decision support, automatic speech recognition (ASR) technology faces significant challenges in its application within power dispatching scenarios.

[0003] Currently, most mainstream speech recognition models employ deep learning algorithms, and their performance is highly dependent on large-scale, high-quality labeled data. However, in the field of power dispatching, obtaining such data faces the following insurmountable obstacles:

[0004] Data annotation is extremely costly: Power dispatch recordings usually involve multi-party conversations, complex background noise, and require precise speaker diarization and text proofreading. Traditional processing methods rely on a large amount of manual listening and annotation, which is not only time-consuming and labor-intensive, but also limited by human fatigue, making it difficult to guarantee the consistency and accuracy of the data.

[0005] Technical terminology recognition is difficult: dispatch voice contains a large number of specific equipment names (such as "Baoding Transformer", "Liangmin Line"), operational terms and number pronunciations (such as "yao", "liang", "guai"). General speech recognition models lack training data for these technical terms, resulting in a high error rate and easy misrecognition of homophones.

[0006] Data scarcity and the long tail problem: Building high-performance ASR models requires massive amounts of training data, especially coverage of different speakers and accents. In the power industry, due to privacy concerns and difficulties in data acquisition, real-world recording data often falls short of the required scale for model training, resulting in poor model generalization ability.

[0007] To address the aforementioned issues, existing technologies typically employ simple data augmentation or general text correction methods. However, these methods fail to integrate with the operational logic of power dispatching, and cannot achieve end-to-end automated data cleaning, speaker separation, and the generation of high-quality synthetic data. Therefore, there is an urgent need for a power dispatching speech processing method and system based on speakerprint clustering and speech cloning to overcome the shortcomings of existing technologies. Summary of the Invention

[0008] To address the shortcomings of existing technologies, this invention provides a power dispatching voice processing method and system based on voiceprint clustering and voice cloning, aiming to solve the aforementioned problems.

[0009] On the one hand, this application provides a power dispatching voice processing method based on voiceprint clustering and voice cloning, including the following steps:

[0010] The raw voice data of the power dispatch scenario is acquired, and voice activity detection and segmentation are performed on the raw voice data to obtain multiple audio segments. Initial speech recognition is then performed on the audio segments to obtain the corresponding initial recognition text.

[0011] Voiceprint features are extracted from the audio segments, and the extracted voiceprint features are clustered based on an unsupervised clustering algorithm to distinguish different speakers and generate speaker labels.

[0012] Based on a knowledge base in the field of power dispatching and a preset dialogue template, a structured dialogue text is generated, and a corresponding speaker identity is assigned to the structured dialogue text according to the speaker tag.

[0013] At least one audio segment that meets the preset quality conditions for each speaker is selected as a reference audio. Using a zero-shot speech cloning model, the structured dialogue text is synthesized into synthetic speech data with the speaker's timbre characteristics.

[0014] The synthesized speech data, or a combination of the synthesized speech data and real labeled speech data, are used to train or fine-tune the basic speech recognition model.

[0015] The system automatically evaluates trained or fine-tuned speech recognition models, calculates the recognition error rate, and generates evaluation reports. Speaker separation and data cleaning are achieved through speaker clustering, solving the problem of high manual annotation costs. High-quality synthetic data is generated using zero-shot speech cloning, expanding the training corpus. Combined with automated evaluation, a closed loop is formed, significantly improving the training efficiency and recognition accuracy of speech recognition models in power dispatching scenarios, and reducing reliance on real-world labeled data.

[0016] Furthermore, the step of extracting voiceprint features from the audio segment and clustering the extracted voiceprint features based on an unsupervised clustering algorithm specifically includes:

[0017] The CAM++ model is used to extract the speaker embedding vector of the audio segment;

[0018] The audio segments are screened for quality based on at least one of the following indicators: speech duration, text length, speech rate, and signal-to-noise ratio. Segments that meet preset quality conditions are then used for clustering.

[0019] The voiceprint embedding vectors of the selected segments are clustered using the DBSCAN algorithm based on cosine distance, and the clustering radius parameter of the DBSCAN algorithm is determined by an adaptive parameter selection method.

[0020] Furthermore, the adaptive parameter selection method includes:

[0021] Randomly sample a portion of the voiceprint embedding vector and calculate the cosine distance distribution between each pair of vectors.

[0022] Calculate the 25th quantile and median of the cosine distance distribution;

[0023] The cluster radius parameter is set to the average of the 25th quantile and the median.

[0024] Furthermore, before generating the structured dialogue text, the initial identified text is further corrected for phonetic similarity based on a lexicon of professional terms in the field of power dispatching. This phonetic similarity correction includes:

[0025] Convert the text to be corrected and the candidate words into pinyin sequences respectively;

[0026] Calculate the Levenshtein edit distance between the pinyin sequences and perform normalization processing, wherein the normalization processing is the edit distance divided by the maximum value of the length of the text to be corrected and the length of the candidate word pinyin sequence;

[0027] A dynamic threshold is calculated based on the length of the candidate word, where the dynamic threshold = 0.35 - 0.02 × L, and L is the length of the candidate word;

[0028] When the normalized edit distance is less than the dynamic threshold, the text to be corrected is replaced with the corresponding candidate word.

[0029] Furthermore, the pinyin similarity correction also includes: employing a greedy matching strategy to scan the text to be corrected from left to right, prioritizing the matching of the longest candidate word.

[0030] Furthermore, the step of using a zero-shot speech cloning model to synthesize the structured dialogue text into synthetic speech data with the speaker's timbre features specifically includes:

[0031] The MaskGCT model is used as a zero-shot speech cloning model, and the model includes at least a semantic encoder, an acoustic encoder-decoder, a text-to-semantic model, and a semantic-to-acoustic model.

[0032] The reference audio, the reference text corresponding to the reference audio, and the target text to be synthesized are input into the MaskGCT model to generate synthesized audio with the timbre features of the reference audio. The target text is the structured dialogue text.

[0033] The generated synthesized audio is post-processed, including resampling and format conversion, to adapt it to the input requirements of the underlying speech recognition model.

[0034] Furthermore, the automated evaluation of the trained or fine-tuned speech recognition model, calculation of the recognition error rate, and generation of an evaluation report specifically include:

[0035] The test set is input into the baseline speech recognition model and the trained or fine-tuned speech recognition model respectively for parallel inference to obtain the baseline recognition result and the fine-tuned recognition result.

[0036] The baseline recognition results and the fine-tuned recognition results are subjected to text standardization, which includes at least converting the scheduling and uppercase readings of numbers into a standard reading.

[0037] The edit distance between the standardized recognition result and the reference text is calculated based on the dynamic programming algorithm, and the number of replacement, deletion and insertion errors is counted to calculate the character error rate;

[0038] Based on the character error rate, an evaluation report is generated that includes a performance comparison between the baseline model and the fine-tuned model, as well as the absolute improvement and relative improvement.

[0039] Furthermore, the preset dialogue templates include multiple power dispatch scenario templates, and the scenario templates include at least one or more of the following: opening greeting, operation instructions, abnormal reports, repetition confirmation, and execution reports; the generation process of the structured dialogue text includes: randomly selecting variable values ​​from the knowledge base to fill the dialogue templates, and performing a rationality constraint check, while automatically extracting hot words from the structured dialogue text.

[0040] On the other hand, this application provides a power dispatching voice processing system based on voiceprint clustering and voice cloning, comprising:

[0041] The preprocessing module is used to acquire the raw voice data of the power dispatching scenario and perform preprocessing to obtain multiple audio segments and corresponding initial recognition text;

[0042] The voiceprint clustering module is used to extract voiceprint features from the audio segments and cluster the extracted voiceprint features based on an unsupervised clustering algorithm to generate speaker tags.

[0043] The dialogue text generation module is used to generate structured dialogue text based on a knowledge base in the field of power dispatching and a preset dialogue template, and to assign corresponding speaker identities to the structured dialogue text according to the speaker tags.

[0044] The speech cloning module is used to select at least one audio segment that meets the preset quality conditions for each speaker as a reference audio, and use a zero-sample speech cloning model to synthesize the structured dialogue text into synthetic speech data with the speaker's timbre characteristics.

[0045] The model training and fine-tuning module is used to train or fine-tune the basic speech recognition model using the synthesized speech data, or a combination of the synthesized speech data and real labeled speech data.

[0046] The automated evaluation module is used to automatically evaluate the trained or fine-tuned speech recognition model, calculate the recognition error rate, and generate an evaluation report.

[0047] Furthermore, the voiceprint clustering module specifically includes a CAM++ voiceprint extraction unit, a multi-level quality control unit, and a DBSCAN clustering unit based on adaptive parameters; the speech cloning module specifically includes a MaskGCT model inference unit and an audio post-processing unit; and the automated evaluation module specifically includes a dual-model parallel inference unit, a text normalization unit, a dynamic programming character error rate calculation unit, and a report generation unit.

[0048] The substantial effects of this invention:

[0049] 1. In this invention, by introducing a voiceprint clustering module based on the CAM++ model and the adaptive DBSCAN algorithm, fully automatic speaker separation of power dispatch recordings is achieved. This technical solution does not rely on manually labeled speaker tags, can extract high-dimensional voiceprint features from unlabeled raw audio, and automatically determine clustering parameters using distance distribution analysis. It effectively solves the problems of high cost and low efficiency of traditional manual labeling, and realizes the association of the same speaker across audio files.

[0050] 2. In this invention, a text correction mechanism for technical terms is constructed by combining a knowledge base in the field of power dispatching with a pinyin similarity correction algorithm. This scheme utilizes pinyin edit distance and dynamic threshold matching to automatically identify and correct homophones of equipment names and operation instructions in the initial ASR recognition results, significantly improving the text accuracy of training data and reducing model training noise caused by misidentification of technical terms.

[0051] 3. In this invention, by applying MaskGCT zero-sample speech cloning technology and intelligent dialogue template engine, a high-quality synthetic data generation pipeline is established. It can use a small amount of real audio as a reference to clone a large number of scheduled dialogues with specific speaker timbre characteristics, which greatly expands the scale and diversity of training data, effectively alleviates the problem of scarcity of real data in the power field, and completely eliminates the need for manual recording and annotation.

[0052] 4. In this invention, by designing an automated evaluation module that includes dual-model parallel inference and text standardization, rapid closed-loop feedback of model performance is achieved. It can automatically compare the character error rate of the baseline model and the fine-tuned model and generate a detailed analysis report, so that the model iteration process can complete the effect verification without manual intervention, which greatly improves the efficiency of algorithm development. Attached Figure Description

[0053] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0054] Figure 1 This is a schematic diagram of the method flow in Example 1.

[0055] Figure 2 This is a schematic diagram of the method flow in Example 2.

[0056] Figure 3 This is a system principle block diagram of Example 3. Detailed Implementation

[0057] To more clearly illustrate the technical solutions in the embodiments of the invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0058] Example 1:

[0059] Reference Figure 1 As shown, a power dispatching voice processing method based on voiceprint clustering and speech cloning includes the following steps:

[0060] The raw speech data of the power dispatch scenario is acquired, speech activity detection and segmentation are performed on the raw speech data to obtain multiple audio segments, and initial speech recognition is performed on the audio segments to obtain the corresponding initial recognition text.

[0061] Voiceprint features are extracted from audio segments, and the extracted voiceprint features are clustered based on an unsupervised clustering algorithm to distinguish different speakers and generate speaker labels.

[0062] Based on a knowledge base in the field of power dispatching and preset dialogue templates, structured dialogue text is generated, and corresponding speaker identities are assigned to the structured dialogue text according to speaker tags.

[0063] At least one audio segment that meets the preset quality conditions for each speaker is selected as the reference audio. Using the zero-shot speech cloning model, the structured dialogue text is synthesized into synthetic speech data with the speaker's timbre characteristics.

[0064] Use synthetic speech data, or a combination of synthetic speech data and real labeled speech data, to train or fine-tune the basic speech recognition model;

[0065] The system automatically evaluates the trained or fine-tuned speech recognition model, calculates the recognition error rate, and generates an evaluation report.

[0066] As one implementation method, voiceprint features are extracted from audio segments, and the extracted voiceprint features are clustered based on an unsupervised clustering algorithm, specifically including:

[0067] The CAM++ model is used to extract the speaker embedding vector of audio segments;

[0068] Audio segments are screened for quality based on at least one of the following indicators: speech duration, text length, speech rate, and signal-to-noise ratio. Segments that meet preset quality conditions are selected for clustering. The preset quality conditions include: speech duration between 120 seconds, text length between 25 characters, speech rate between 4 words / second, and signal-to-noise ratio greater than 6dB.

[0069] The voiceprint embedding vectors of the selected segments are clustered using the DBSCAN algorithm based on cosine distance, and the clustering radius parameter of the DBSCAN algorithm is determined by an adaptive parameter selection method.

[0070] As one implementation method, the adaptive parameter selection method includes:

[0071] Randomly sample a portion of the voiceprint embedding vector and calculate the cosine distance distribution between each pair of vectors.

[0072] Calculate the 25th quantile and median of the cosine distance distribution;

[0073] Set the cluster radius parameter to the average of the 25th quantile and the median.

[0074] As one implementation method, before generating structured dialogue text, the initial identified text is further corrected for phonetic similarity based on a lexicon of professional terms in the field of power dispatching. The phonetic similarity correction includes:

[0075] Convert the text to be corrected and the candidate words into pinyin sequences respectively;

[0076] Calculate the Levenshtein edit distance between the pinyin sequences and normalize it by dividing the edit distance by the maximum value of the length of the text to be corrected and the length of the candidate word pinyin sequence.

[0077] The dynamic threshold is calculated based on the length of the candidate word. The dynamic threshold is calculated as 0.35 - 0.02 × L, where L is the length of the candidate word.

[0078] When the normalized edit distance is less than the dynamic threshold, the text to be corrected is replaced with the corresponding candidate word.

[0079] As one implementation method, pinyin similarity correction also includes: using a greedy matching strategy to scan the text to be corrected from left to right and prioritizing the matching of the longest candidate word.

[0080] As one implementation method, a zero-shot speech cloning model is used to synthesize structured dialogue text into synthetic speech data with the speaker's vocal timbre characteristics, specifically including:

[0081] The MaskGCT model is used as a zero-shot speech cloning model. The model includes at least a semantic encoder, an acoustic encoder-decoder, a text-to-semantic model, and a semantic-to-acoustic model.

[0082] The reference audio, the corresponding reference text, and the target text to be synthesized are input into the MaskGCT model to generate synthesized audio with the timbre features of the reference audio, and the target text is the structured dialogue text.

[0083] The generated synthesized audio is post-processed, including resampling and format conversion, to adapt it to the input requirements of the basic speech recognition model.

[0084] As one implementation method, the trained or fine-tuned speech recognition model is automatically evaluated, the recognition error rate is calculated, and an evaluation report is generated, specifically including:

[0085] The test set is input into the baseline speech recognition model and the trained or fine-tuned speech recognition model respectively for parallel inference to obtain the baseline recognition result and the fine-tuned recognition result.

[0086] Text standardization is performed on the baseline recognition results and the fine-tuned recognition results. Text standardization includes at least converting the scheduling and uppercase readings of numbers into the standard reading.

[0087] The edit distance between the standardized recognition result and the reference text is calculated based on the dynamic programming algorithm, and the number of replacement, deletion and insertion errors is counted to calculate the character error rate;

[0088] Based on the character error rate, an evaluation report is generated that includes a performance comparison between the baseline model and the fine-tuned model, as well as the absolute and relative improvement rates.

[0089] As an implementation manner, the preset dialogue template includes multiple power dispatching scenario templates, and the scenario template includes at least one or more of opening greetings, operation instructions, abnormal reports, repeat confirmations, and execution reports; the generation process of the structured dialogue text includes: randomly extracting variable values from the knowledge base to fill the dialogue template, performing a rationality constraint check, and automatically extracting hot words in the structured dialogue text.

[0090] Example 2:

[0091] Refer to Figure 2 As shown, this embodiment is basically the same as Embodiment 1, except that a power dispatching speech recognition method based on voice print clustering, voice cloning, and automated evaluation is provided. Through five core technologies of automated voice print extraction, zero-shot voice cloning, pinyin similarity correction, intelligent template generation, and full-automatic evaluation, the problem that traditional ASR systems rely on a large number of manual annotations is completely solved. The method includes the following steps:

[0092] Step 1, domain data cleaning and hot word extraction:

[0093] Extract professional terms such as device names and protection signals from the business database, and perform the following processing:

[0094] Device name parsing: Different parsing rules are adopted according to the device type (transformer, disconnecting switch, circuit breaker, load, earthing switch, bus, line end point).

[0095] Voltage level conversion: Convert Arabic numeral voltages to Chinese readings, such as "110kV" → "One hundred and ten kilovolts", "Two hundred and twenty kilovolts".

[0096] Normalize the pronunciation of numbers to generate two variants of dispatching readings and ordinary readings:

[0097] Dispatching reading: 1 → yao, 2 → liang, 7 → guai, 0 → dong.

[0098] Ordinary reading: 1 → yi, 2 → er, 7 → qi, 0 → ling.

[0099] Roman numeral conversion: Convert Roman numerals (Ⅰ, Ⅱ, Ⅲ) in the bus number to Arabic numerals and then to Chinese.

[0100] Hot word extraction: Extract key fragments from the device name as ASR hot words, such as "Baoding Substation", "Liang Min", "Switch", etc.

[0101] Step 2, Initial ASR Inference and Speaker Separation (used to obtain timestamped audio segments and preliminary speaker labels, providing input data for voiceprint clustering in Step 3. The voiceprint module integrated in Step 2 has limited accuracy, and the resulting temporary speaker IDs are only used to assist segment segmentation and preliminary verification; they will be discarded in Step 3, and the voiceprints will be re-extracted based on the superior CAM++ model for high-precision cross-file clustering):

[0102] The FunASR framework was used for preliminary speech recognition to provide basic data for subsequent speaker extraction.

[0103] Multi-process parallel processing: ProcessPoolExecutor is used to implement multi-process parallel inference, with each process independently loading the model onto the GPU.

[0104] Integrated VAD + punctuation + voiceprint: Simultaneously enables the Voice Activity Detection (VAD), punctuation recovery, and speaker recognition modules.

[0105] Timestamp Alignment: Outputs recognition results containing precise timestamps and temporary speaker IDs.

[0106] Segment segmentation: Automatically divides long recordings into short segments of 4-20 seconds to meet voiceprint extraction requirements.

[0107] Step 3, cross-file voiceprint extraction and speaker clustering (traditional methods require manual annotation of the speaker's identity for each recording; 50 hours of data requires annotation of approximately 3000 segments, taking about 150 person-hours):

[0108] Fully automatic speaker clustering based on CAM++ voiceprint model and DBSCAN algorithm:

[0109] High-quality voiceprint embedding extraction: 192-dimensional voiceprint embedding vectors are extracted using the CAM++ model;

[0110] Multi-level quality control strategy:

[0111] Voice duration threshold: 4-20 seconds (filtering out segments that are too short or too long).

[0112] Text length threshold: 6-50 characters (to ensure sufficient semantic information);

[0113] Speech rate threshold: 3-8.5 words / second (excluding abnormal speech rates);

[0114] Signal-to-noise ratio check: >6dB (to ensure audio quality);

[0115] Only high-quality speech segments that have passed multi-level quality control are retained for subsequent processing, with a success rate of approximately 65-75%.

[0116] Distance distribution analysis and parameter adaptation:

[0117] Calculate the cosine distance distribution by randomly sampling 1000 embedding vectors;

[0118] Statistical analysis: minimum distance, maximum distance, mean distance, median, standard deviation, 25th / 75th percentile;

[0119] Adaptive eps parameter selection: eps = (25th percentile + 50th percentile) / 2, typical value 0.125.

[0120] Physical meaning: The cosine distance ranges from [0,2]. The 25th percentile approximately represents the lower bound of the distance between closely connected segments within a class, while the median reflects the central tendency of the overall distance distribution. Taking the average of the two yields a threshold biased towards smaller distances, which ensures that segments from the same speaker are effectively aggregated while avoiding the introduction of different speakers due to an excessively large threshold.

[0121] Experimental basis: On the power dispatch voice dataset, multiple eps selection strategies (K-distance graph inflection point method, mean method, median method, etc.) were compared. The adaptive formula obtained the best clustering purity and FMI index, with a typical eps value of about 0.125. It can stably adapt to the distribution differences of different batches of data without manual intervention.

[0122] Cosine distance is used as the metric without normalization; the original vector distance is used directly for calculation to ensure voiceprint distinguishability.

[0123] DBSCAN cross-file clustering:

[0124] Unsupervised clustering is performed using the cosine distance metric.

[0125] min_samples=3 ensures that each cluster has at least 3 samples;

[0126] Automatically identify noise points (isolated segments) and mark them as SPK_NOISE;

[0127] Linking the same speaker across files requires no manual intervention.

[0128] Voiceprint library construction:

[0129] Generate a unique ID (SPK0001,SPK0002,...) for each speaker.

[0130] The average embedding vector is calculated as the speaker's voiceprint template;

[0131] Save speaker_library.json, which contains:

[0132] embedding_file: Voiceprint template path;

[0133] segment_count: Number of segments;

[0134] segments: Detailed information for all segments (audio_id, start, end, text)).

[0135] By pioneering an adaptive clustering parameter selection method based on distance distribution analysis, no manual parameter tuning is required.

[0136] Multi-level quality control strategies ensure the quality of voiceprint extraction and filter low-quality segments; a breakpoint resume mechanism supports large-scale data processing and improves engineering reliability; full automation eliminates the need for manual speaker identification and saves time.

[0137] By automating voiceprint extraction and clustering, this labor cost is completely eliminated; after processing 50 hours of recordings (approximately 3000 segments), 8-12 speakers are automatically identified; the clustering accuracy rate is over 90% (verified by manual sampling); the processing time is approximately 2-3 hours (GPU accelerated), saving 99% of the time compared to manual annotation.

[0138] Step 4, Pinyin Similarity Text Correction (General ASR models have a high error rate in recognizing electrical terminology, such as misidentifying "Baoding Bian" as "Baoding de" and "Liangmin Line" as "Liangmin County". Traditional methods require manual verification and correction of each item, which takes about 100 person-hours for 50 hours of data (approximately 3000 segments).

[0139] Automatically correct device names that are incorrectly identified by ASR using a Pinyin similarity matcher:

[0140] 1) Constructing a candidate word library:

[0141] Build a candidate thesaurus (approximately 5,000-10,000 entries) from the list of device names generated in step 1.

[0142] Includes substation name, equipment name, line name, and operating terminology;

[0143] Stored categorized by device type for easy and quick retrieval;

[0144] 2). Pinyin encoding and normalization:

[0145] Use the pypinyin library to convert the recognized text and candidate words into pinyin sequences;

[0146] Handling polyphonic characters: Select the most likely pronunciation based on the context;

[0147] Remove tones and retain only the syllable sequence (reduce matching difficulty);

[0148] Example: "Baoding Substation" → ["bao", "ding", "bian"];

[0149] 3). Edit distance calculation:

[0150] Use the dynamic programming algorithm to calculate the Levenshtein distance of the pinyin sequence;

[0151] Count the number of substitution (S), insertion (I), and deletion (D) operations respectively;

[0152] Normalization: normalized_distance = distance / max(len(text_pinyin), len(candidate_pinyin));

[0153] Distance range: from 0 (perfect match) to 1 (completely different);

[0154] 4). Dynamic threshold matching:

[0155] Base threshold: 0.35 (empirical value);

[0156] Dynamic adjustment: threshold = base_threshold + 0.02 × len(candidate);

[0157] Relax the threshold for short words (tolerate more errors), and tighten the threshold for long words (improve accuracy);

[0158] Minimum number of matching syllables: 2 (to avoid mis-matching of single syllables);

[0159] 5). Selection and correction of the best candidate:

[0160] Calculate the normalized distance for all candidate words;

[0161] Select the candidate word with the minimum distance and lower than the threshold;

[0162] Record the matching position (start, end) and confidence;

[0163] Output the correction result: original text → corrected text + confidence;

[0164] 6). Greedy matching strategy:

[0165] Scan the text from left to right and preferentially match the longest candidate word;

[0166] Avoid overlapping matches to ensure correction consistency;

[0167] Applicable to scenarios where a long text contains multiple device names;

[0168] The device name correction algorithm based on pinyin edit distance effectively solves the problem of homophones; the dynamic threshold mechanism adaptively adjusts according to the length of candidate words to balance recall and accuracy; the greedy matching strategy ensures unified correction of multiple errors in long texts; and the system is fully automated, eliminating the need for manual proofreading and saving approximately 100 person-hours.

[0169] The accuracy rate of professional terminology correction is over 85%;

[0170] Processing 3000 segments took approximately 30 minutes (single CPU core).

[0171] Typical error corrections:

[0172] *"Pull Baoding apart" → "Make Baoding separate"✓;

[0173] *"Good Citizens of 511 Counties" → "Good Citizens of 511 Line" ✓;

[0174] *"Pull the faucet" → "Pull the switch"✓.

[0175] Step 5, scheduling dialogue template generation and intelligent filling (training the ASR model requires a large amount of diverse dialogue data. Traditional methods require tens of thousands of dialogues to be manually written or recorded, which is costly, time-consuming, and lacks diversity. The cost of manually writing 5,000 dialogues is about 200 person-hours, and the cost of recording is even higher, exceeding 500 person-hours):

[0176] Batch generation of dialogue text conforming to power dispatching specifications based on the Jinja2 template engine and domain knowledge base:

[0177] 1) Template Library Design:

[0178] More than 10 dialogue scenario templates have been designed, covering the entire scheduling process:

[0179] *Opening greeting: "{{tone_start}}{{station name}}, {{dispatcher}}.";

[0180] *Preliminary question: "What is the current status of {{site name}} and {{device}}?"

[0181] *Operation command: "Please switch {{device}}{{operation}} to {{status}}.";

[0182] *Abnormality Report: "Report, {{device}} is experiencing {{phenomenon}}, please provide instructions.";

[0183] *Repeat for confirmation: "Received, {{operation}}{{device}}, please confirm.";

[0184] *Execution report: "{{Device}} has been {{operated}}, current status {{status}}.";

[0185] *Shift handover: "{{time}} shift handover, current operating mode...";

[0186] *Emergency Command: "Immediately operate the equipment and execute!"

[0187] Each template contains variable placeholders that can be flexibly filled.

[0188] 2) Domain knowledge base construction:

[0189] Substation list: Extract all substation names (approximately 100-200) from the equipment list;

[0190] Equipment list: parses the names of equipment such as transformers, disconnectors, and circuit breakers (approximately 5000-10000 items);

[0191] Operational terms: opening, closing, tripping, switching to maintenance, switching to operation, etc. (approximately 20-30);

[0192] Status terms: running, hot standby, cold standby, under maintenance, out of service, etc. (approximately 10-15);

[0193] Phenomenon terminology: tripping, abnormality, overload, voltage instability, protection action, etc. (approximately 10-20);

[0194] Personnel Names: List of common dispatcher names (approximately 100);

[0195] Interjections: Hello, Ah, um, Okay, Received, etc. (about 10-15);

[0196] 3) Smart variable filling:

[0197] Random sampling: Randomly select variable values ​​from the knowledge base to populate the template;

[0198] Constraint checks: Ensure that the equipment is compatible with the substation and that its operation and status are reasonable;

[0199] Diversity guarantee: Use shuffle and random.sample to avoid duplication;

[0200] Time generation: Randomly generate Chinese time expressions that conform to scheduling conventions;

[0201] *Hours: 0-23, converted to "midnight" to "11 PM";

[0202] *Minutes: 0-59, converted to "0 minutes" to "fifty-nine minutes";

[0203] *Examples: "13:25", "08:05";

[0204] 4) Automatic hot keyword annotation:

[0205] Analyze the variables in the template and automatically extract the corresponding hot words;

[0206] Example: Template "Please operate {{device}}" → Hot words = [device name, operation terminology].

[0207] Supports hot word enhancement during ASR inference to improve recognition accuracy.

[0208] 5) Batch generation and deduplication:

[0209] Set the target number of records to be generated (e.g., 1000-5000 records for each scenario);

[0210] Use a set to remove duplicates and avoid having the same conversations.

[0211] Generate JSON format: {"text":"dialogue content","hot_word":"hot_word1,hot_word2"};

[0212] The total number of diverse dialogues generated can reach 50,000 to 100,000.

[0213] A dialogue template library covering the entire power dispatch process has been built, which can generate tens of thousands of training corpora that conform to industry standards; the intelligent variable filling mechanism ensures the rationality and diversity of the generated content; automatic hot word labeling provides convenience for subsequent ASR enhancement; the generation is fully automated, eliminating the need for manual writing and recording, saving about 200-500 person-hours.

[0214] It generates 76,110 high-quality scheduling dialogues, covering more than 10 scenarios; the generation time is about 10-20 minutes (single CPU core); the dialogues are highly diverse and have no obvious repetitive patterns; they can be directly used in the subsequent voice cloning process.

[0215] Step 6, Zero-shot speech cloning and data augmentation (Traditional ASR training requires a large amount of real labeled audio, which is costly and time-consuming. 50 hours of labeled data requires several months and hundreds of thousands of yuan in cost. This invention uses zero-shot speech cloning technology to generate a large amount of synthetic data from a small amount of reference audio, completely solving the problem of data scarcity):

[0216] Zero-shot speech cloning was performed using the MaskGCT model to generate synthetic training data. Voiceprint clustering results were used as conditional input to ensure that the synthesized speech retained the timbre characteristics of specific speakers. Specifically, the voiceprint database constructed in step 3 assigned a unique ID (SPK0001, SPK0002, ...) to each speaker and saved all their high-quality audio clips. The dialogue text generated in step 5 was bound to the speaker IDs in the voiceprint database according to the task role. The following detailed process is based on this to achieve accurate reproduction of the speaker's timbre.

[0217] Reference audio filtering:

[0218] From the voiceprint database generated in step 3, select audio segments for each speaker according to speaker ID;

[0219] Quality filtration:

[0220] File size < 160KB (ensure the duration is moderate, approximately 3-8 seconds);

[0221] Exclude noisy speakers (SPK_NOISE);

[0222] Exclude specific speakers (e.g., those with poor voice quality);

[0223] Each speaker will randomly select up to 3000 reference audio clips;

[0224] Ultimately, a dedicated reference audio pool is established for each valid speaker, totaling approximately 5,000-10,000 high-quality reference audio files. These audio files directly correspond to the speaker IDs in the voiceprint database, ensuring that the source of the voice timbre is strictly bound to the speaker's identity.

[0225] Text preprocessing:

[0226] Remove all punctuation marks: , . ! ? 、 ; : ''() 《

[0227] Remove spaces and special characters;

[0228] Retain only Chinese text and numbers;

[0229] Example: "Please disconnect the 110kV Liangmin 511 line of Baoding Substation." → "Please disconnect the 110kV Liangmin 511 line of Baoding Substation."

[0230] MaskGCT inference process:

[0231] Model architecture:

[0232] Semantic Model: Semantic encoder (extracts semantic features from text);

[0233] Semantic Codec: Semantic codec (192-dimensional semantic token);

[0234] Acoustic Codec: an acoustic codec that extracts acoustic features from audio.

[0235] T2S Model: Text-to-Semantic Model (Generates Sequences of Semantic Tokens);

[0236] S2A Model: Semantic to Acoustic Model (Generates Acoustic Token Sequences).

[0237] Audio post-processing:

[0238] Resampling: Resamples 24kHz audio to 16kHz (to meet FunASR input requirements).

[0239] Python

[0240] resampled_data = resample(recovered_audio,int(len(recovered_audio) *16000 / 24000));

[0241] Save format: 16-bit PCM WAV file;

[0242] File naming: syn_XXXXXX.wav (in ascending order of serial number);

[0243] Directory organization: Store 5000 files in a subdirectory to avoid too many files in a single directory.

[0244] Dataset Construction:

[0245] Generate JSONL format entries:

[0246] json

[0247] { "source": " / abs / path / to / syn_000001.wav",

[0248] "target": "Please disconnect the 110kV Liangmin 511 line in Baoding".

[0249] "hotword": ["Baoding Change", "Liangmin", "Lakai"],

[0250] "hotword_lengths": [3, 2, 2],

[0251] "speaker_id": "SPK0001"}

[0252] Resuming interrupted downloads:

[0253] Save wav.scp, text, and mapping.json once every 100 records are generated;

[0254] Save a temporary JSONL file (temp_all_data.json);

[0255] Upon startup, check for existing progress and skip files that have already been generated.

[0256] Loop generation:

[0257] Target quantity: 76,110 (can be adjusted according to needs);

[0258] Iterate through all the dialogue texts generated in step 5 (each sentence has been assigned and bound to a specific speaker ID in the voiceprint library through scheduling roles).

[0259] For each text, based on its bound speaker ID, an audio file is randomly selected from the speaker's reference audio pool as prompt_wav_path, and input into MaskGCT for cloning, ensuring that the generated synthesized speech accurately preserves the timbre features of the target speaker.

[0260] Repeat the process multiple times until the target quantity is reached.

[0261] Dataset partitioning:

[0262] Randomly allocated according to a ratio of 90% / 5% / 5%:

[0263] train.json: 68499 records (training set);

[0264] dev.jsonl: 3805 records (validation set);

[0265] test.json: 3806 records (test set);

[0266] Randomly shuffle the mixture to ensure even distribution.

[0267] It also generates Kaldi format wav.scp and text files (compatible with older versions of FunASR).

[0268] The innovative application of MaskGCT zero-shot speech cloning technology to ASR training data augmentation solves the problem of domain data scarcity; it supports the generation of a large amount of high-quality synthetic data (169 hours) from a small amount of reference audio (50 hours), with a data expansion ratio of 3.4:1; the breakpoint resume mechanism supports large-scale generation tasks and improves engineering reliability; it is fully automated, eliminating the need for manual recording and saving approximately 500 person-hours and hundreds of thousands of yuan in costs.

[0269] 76,110 synthesized audio tracks were generated, with a total duration of approximately 169 hours;

[0270] The generation process takes approximately 48-72 hours (for a single A100 GPU card).

[0271] The speech quality is high, the tone is natural, and the similarity with the reference audio is over 85%.

[0272] The similarity calculation uses an objective indicator: the CAM++ model in step 3 is used to extract the 192-dimensional voiceprint embedding vectors of the synthesized audio and the corresponding reference audio, and the cosine similarity is calculated. The average cosine similarity measured on 500 randomly selected audio pairs is 0.85, which indicates that the synthesized speech can highly preserve the timbre features of the target speaker.

[0273] Compared to traditional methods:

[0274] Cost reduction: from hundreds of thousands of yuan to almost zero (GPU electricity costs only);

[0275] Time reduced: from months to 3 days;

[0276] Increased data volume: from 50 hours to 219 hours (real + synthetic).

[0277] Step 7, DeepSpeed ​​distributed fine-tuning:

[0278] Distributed fine-tuning of the Paraformer model using the DeepSpeed ​​framework:

[0279] Data merging: Real labeled data (50 hours) and synthetic data generated by speech clones (169 hours) were merged. A direct mixing training strategy was adopted, with the ratio of real data to synthetic data being approximately 1:3, totaling approximately 219 hours.

[0280] DeepSpeed ​​configuration: Employs the ZeRO Stage 1 optimization strategy and supports large batch size training.

[0281] 1. Learning rate scheduling: Use the warmuplr scheduler and set the warmup steps to 2000.

[0282] 2. Multi-metric monitoring: Simultaneously monitors CER (character error rate) and ACC (accuracy rate).

[0283] 3. Checkpoint saving: Save checkpoints every 10,000 steps, and retain the 5 best models.

[0284] Training effect verification: On the power dispatch domain-specific test set, the baseline model trained with only real data had a CER of 9.8%. After adding synthetic data for mixed training, the CER dropped to 4.2%, a relative reduction of 57.1%, and no performance degradation was observed. This confirms that synthetic data effectively improves the domain recognition accuracy without introducing additional noise.

[0285] Step 8, Model Export and Deployment:

[0286] Convert the DeepSpeed-trained model to standard PyTorch format:

[0287] 1. Checkpoint analysis: Extract model weights from DeepSpeed's mp_rank_00_model_states.pt.

[0288] 2. Weight Restructuring: Remove the "module" prefix and restructure into the standard state_dict format.

[0289] 3. Model Validation: Validate the completeness and usability of the exported model.

[0290] Note: This step is a standard model export process and is not the core innovation of this invention.

[0291] Step 9, Automated Evaluation and Comparative Analysis (Traditional model evaluation requires manual sampling to identify results and calculate error rates, which is time-consuming and labor-intensive. Manual evaluation of a 50-hour test set requires approximately 50 person-hours and is difficult to comprehensively analyze error patterns. This invention implements a fully automated, multi-dimensional evaluation system, completely eliminating the need for manual evaluation):

[0292] A fully automated model performance evaluation system that supports comparative analysis with baseline models:

[0293] 1) Dual-model parallel inference:

[0294] Fine-tuned model inference:

[0295] Load the finely tuned Paraformer model;

[0296] Read the test audio path from test.json;

[0297] Batch inference (batch_size=1, num_workers=4);

[0298] The result is saved as test.json.

[0299] Baseline model inference:

[0300] Load the untuned baseline model (speech_seaco_paraformer_large_asr_nat-zh-cn-16k-common-vocab8404-pytorch).

[0301] Inference was performed using the same test set;

[0302] Save the results as baseline_results / test.json;

[0303] Resume interrupted downloads: Supports skipping completed inference tasks.

[0304] 2). Manual CER calculation (based on dynamic programming):

[0305] Text normalization:

[0306] Lowercase conversion: Convert all English letters to lowercase.

[0307] Numeric normalization:

[0308] Dispatch reading → Standard reading: Yao → One, Liang → Two, Guai → Seven, Dong → Zero;

[0309] Chinese uppercase numerals → Standard reading: Yi → One, Er → Two, Qi → Seven, Ling → Zero;

[0310] Combined numbers: Shi → Ten, Bai → Hundred, Qian → Thousand, Wan → Ten thousand;

[0311] Example: "One hundred and ten thousand volts, Yao Liang Guai Dong" → "One hundred and ten thousand volts, One Two Seven Zero".

[0312] Sample-level detailed records:

[0313] Each sample record: key, ref_original, hyp_original, ref_normalized, hyp_normalized;

[0314] Error statistics: errors, chars, cer;

[0315] Facilitate subsequent error analysis.

[0316] 3). Overall metric calculation:

[0317] Overall CER: total_errors / total_chars;

[0318] Best CER: min(sample_cer for sample in results);

[0319] Worst CER: max(sample_cer for sample in results);

[0320] Average CER: sum(sample_cer) / len(results);

[0321] WER calculation (optional): Word error rate based on word segmentation.

[0322] 4). Analysis of the top 5 worst samples:

[0323] Sort in descending order of CER to identify the 5 samples with the highest error rate.

[0324] Record for each sample:

[0325] Reference text (original + standardized);

[0326] Identified text (original + standardized);

[0327] Number of errors, number of characters, CER;

[0328] Error type distribution (S / I / D).

[0329] Manual analysis of typical error patterns:

[0330] Homophonic characters: "of Baoding" vs "Baoding change";

[0331] Misrecognition of numbers: "May 11" vs "511";

[0332] Proper nouns: "Good People County" vs "Liangmin Line".

[0333] 5). Comparative analysis:

[0334] Absolute improvement: CER_baseline - CER_finetuned;

[0335] Relative improvement: (Absolute improvement / CER_baseline) × 100%;

[0336] Performance improvement: (1 - CER_finetuned / CER_baseline) × 100%.

[0337] Item-by-item comparison:

[0338] Compare the performance of the two models on the same samples;

[0339] Identify samples with significant improvement and degraded samples;

[0340] Analyze the reasons for improvement (hot word enhancement? Voice cloning? ).

[0341] Markdown format comparison report (evaluation_comparison.md):

[0342] Evaluation overview (time, number of samples, number of characters);

[0343] Table for comparing core performance indicators;

[0344] Statistics on performance improvement;

[0345] Comparison of the top 5 worst samples (baseline vs finetuned);

[0346] Conclusions and suggestions (give targeted suggestions based on the improvement rate);

[0347] Output file description.

[0348] Detailed sample analysis (cer_detailed.json):

[0349] Detailed CER calculation results for all samples;

[0350] It can be used for further analysis and visualization.

[0351] Top 5 worst samples (cer_top_5_worst.json):

[0352] Save the 5 samples with the highest error rates separately;

[0353] It facilitates quick problem location.

[0354] Example 3:

[0355] Reference Figure 3 As shown, this embodiment is basically the same as embodiment 1, except that it provides a power dispatching voice processing system based on voiceprint clustering and voice cloning, including:

[0356] The preprocessing module is used to acquire the raw voice data of the power dispatching scenario, and to perform voice activity detection, segmentation and initial speech recognition to obtain multiple audio segments and corresponding initial recognition text;

[0357] The voiceprint clustering module is used to extract voiceprint features from audio segments and cluster the extracted voiceprint features based on an unsupervised clustering algorithm to generate speaker labels. The audio segments used for clustering are segments that meet preset quality conditions, including: speech duration between 4 and 20 seconds, text length between 6 and 50 characters, speech rate between 3 and 8.5 words per second, and signal-to-noise ratio greater than 6 dB.

[0358] The dialogue text generation module is used to generate structured dialogue text based on a knowledge base in the field of power dispatch and preset dialogue templates, and to assign corresponding speaker identities to the structured dialogue text according to speaker tags.

[0359] The speech cloning module is used to select at least one audio segment that meets the preset quality conditions for each speaker as a reference audio, and use the zero-sample speech cloning model to synthesize the structured dialogue text into synthetic speech data with the speaker's timbre characteristics.

[0360] The model training and fine-tuning module is used to train or fine-tune the basic speech recognition model using synthetic speech data, or a combination of synthetic speech data and real labeled speech data.

[0361] The automated evaluation module is used to automatically evaluate the trained or fine-tuned speech recognition model, calculate the recognition error rate, and generate an evaluation report.

[0362] As one implementation method, the voiceprint clustering module specifically includes a CAM++ voiceprint extraction unit, a multi-level quality control unit, and a DBSCAN clustering unit based on adaptive parameters; the adaptive parameters are the average of the 25th quantile and the median of the cosine distance between the voiceprint embedding vectors.

[0363] As one implementation method, the voice cloning module specifically includes a MaskGCT model inference unit and an audio post-processing unit;

[0364] As one implementation method, the automated evaluation module specifically includes a dual-model parallel inference unit, a text standardization unit, a dynamic programming character error rate calculation unit, and a report generation unit.

[0365] It should be noted that while the preferred embodiments of the present invention are provided in the specification and accompanying drawings, the present invention can be implemented in many different forms and is not limited to the embodiments described herein. These embodiments are not intended to impose additional limitations on the content of the present invention; their purpose is to provide a more thorough and comprehensive understanding of the disclosure of the present invention. Furthermore, the above-described technical features can be combined with each other to form various embodiments not listed above, all of which are considered to be within the scope of the present invention specification. Moreover, those skilled in the art can make improvements or modifications based on the above description, and all such improvements and modifications should fall within the protection scope of the appended claims.

Claims

1. A power dispatching voice processing method based on voiceprint clustering and speech cloning, characterized in that, Includes the following steps: The raw voice data of the power dispatch scenario is acquired, and voice activity detection and segmentation are performed on the raw voice data to obtain multiple audio segments. Initial speech recognition is then performed on the audio segments to obtain the corresponding initial recognition text. Voiceprint features are extracted from the audio segments, and the extracted voiceprint features are clustered based on an unsupervised clustering algorithm to distinguish different speakers and generate speaker labels. Based on a knowledge base in the field of power dispatching and a preset dialogue template, a structured dialogue text is generated, and a corresponding speaker identity is assigned to the structured dialogue text according to the speaker tag. At least one audio segment that meets the preset quality conditions for each speaker is selected as a reference audio. Using a zero-shot speech cloning model, the structured dialogue text is synthesized into synthetic speech data with the speaker's timbre characteristics. The synthesized speech data, or a combination of the synthesized speech data and real labeled speech data, are used to train or fine-tune the basic speech recognition model. The system automatically evaluates the trained or fine-tuned speech recognition model, calculates the recognition error rate, and generates an evaluation report.

2. The power dispatching voice processing method based on voiceprint clustering and voice cloning according to claim 1, characterized in that, The step of extracting voiceprint features from the audio segment and clustering the extracted voiceprint features based on an unsupervised clustering algorithm specifically includes: The CAM++ model is used to extract the speaker embedding vector of the audio segment; The audio segments are screened for quality based on at least one of the following indicators: speech duration, text length, speech rate, and signal-to-noise ratio. Segments that meet preset quality conditions are then used for clustering. The voiceprint embedding vectors of the selected segments are clustered using the DBSCAN algorithm based on cosine distance, and the clustering radius parameter of the DBSCAN algorithm is determined by an adaptive parameter selection method.

3. The power dispatching voice processing method based on voiceprint clustering and voice cloning according to claim 2, characterized in that, The adaptive parameter selection method includes: Randomly sample a portion of the voiceprint embedding vector and calculate the cosine distance distribution between each pair of vectors. Calculate the 25th quantile and median of the cosine distance distribution; The cluster radius parameter is set to the average of the 25th quantile and the median.

4. The power dispatching voice processing method based on voiceprint clustering and voice cloning according to claim 1, characterized in that, Before generating the structured dialogue text, the initial identified text is further corrected for phonetic similarity based on a lexicon of professional terms in the field of power dispatching. The phonetic similarity correction includes: Convert the text to be corrected and the candidate words into pinyin sequences respectively; Calculate the Levenshtein edit distance between the pinyin sequences and perform normalization processing, wherein the normalization processing is the edit distance divided by the maximum value of the length of the text to be corrected and the length of the candidate word pinyin sequence; A dynamic threshold is calculated based on the length of the candidate word, where the dynamic threshold = 0.35 - 0.02 × L, and L is the length of the candidate word; When the normalized edit distance is less than the dynamic threshold, the text to be corrected is replaced with the corresponding candidate word.

5. The power dispatching voice processing method based on voiceprint clustering and voice cloning according to claim 4, characterized in that, The pinyin similarity correction also includes: using a greedy matching strategy to scan the text to be corrected from left to right, and prioritizing the matching of the longest candidate word.

6. The power dispatching voice processing method based on voiceprint clustering and voice cloning according to claim 1, characterized in that, The process of using a zero-shot speech cloning model to synthesize the structured dialogue text into synthetic speech data with the speaker's vocal timbre features specifically includes: The MaskGCT model is used as a zero-shot speech cloning model, and the model includes at least a semantic encoder, an acoustic encoder-decoder, a text-to-semantic model, and a semantic-to-acoustic model. The reference audio, the reference text corresponding to the reference audio, and the target text to be synthesized are input into the MaskGCT model to generate synthesized audio with the timbre features of the reference audio. The target text is the structured dialogue text. The generated synthesized audio is post-processed, including resampling and format conversion, to adapt it to the input requirements of the underlying speech recognition model.

7. The power dispatching voice processing method based on voiceprint clustering and voice cloning according to claim 1, characterized in that, The automated evaluation of the trained or fine-tuned speech recognition model, calculation of the recognition error rate, and generation of an evaluation report specifically include: The test set is input into the baseline speech recognition model and the trained or fine-tuned speech recognition model respectively for parallel inference to obtain the baseline recognition result and the fine-tuned recognition result. The baseline recognition results and the fine-tuned recognition results are subjected to text standardization, which includes at least converting the scheduling and uppercase readings of numbers into a standard reading. The edit distance between the standardized recognition result and the reference text is calculated based on the dynamic programming algorithm, and the number of replacement, deletion and insertion errors is counted to calculate the character error rate; Based on the character error rate, an evaluation report is generated that includes a performance comparison between the baseline model and the fine-tuned model, as well as the absolute improvement and relative improvement.

8. The power dispatching voice processing method based on voiceprint clustering and voice cloning according to claim 1, characterized in that, The preset dialogue templates include a variety of power dispatch scenario templates, and the scenario templates include at least one or more of the following: opening greeting, operation instructions, abnormal reports, repetition confirmation, and execution reports; The process of generating the structured dialogue text includes: randomly selecting variable values ​​from the knowledge base to fill the dialogue template, performing a rationality constraint check, and automatically extracting hot words from the structured dialogue text.

9. A power dispatching voice processing system based on voiceprint clustering and voice cloning, used to implement the processing method as described in any one of claims 1-8, characterized in that, include: The preprocessing module is used to acquire the raw voice data of the power dispatching scenario and perform preprocessing to obtain multiple audio segments and corresponding initial recognition text; The voiceprint clustering module is used to extract voiceprint features from the audio segments and cluster the extracted voiceprint features based on an unsupervised clustering algorithm to generate speaker tags. The dialogue text generation module is used to generate structured dialogue text based on a knowledge base in the field of power dispatching and a preset dialogue template, and to assign corresponding speaker identities to the structured dialogue text according to the speaker tags. The speech cloning module is used to select at least one audio segment that meets the preset quality conditions for each speaker as a reference audio, and use a zero-sample speech cloning model to synthesize the structured dialogue text into synthetic speech data with the speaker's timbre characteristics. The model training and fine-tuning module is used to train or fine-tune the basic speech recognition model using the synthesized speech data, or a combination of the synthesized speech data and real labeled speech data. The automated evaluation module is used to automatically evaluate the trained or fine-tuned speech recognition model, calculate the recognition error rate, and generate an evaluation report.

10. The power dispatching voice processing system based on voiceprint clustering and voice cloning according to claim 9, characterized in that, The voiceprint clustering module specifically includes a CAM++ voiceprint extraction unit, a multi-level quality control unit, and a DBSCAN clustering unit based on adaptive parameters; the speech cloning module specifically includes a MaskGCT model inference unit and an audio post-processing unit; the automated evaluation module specifically includes a dual-model parallel inference unit, a text normalization unit, a dynamic programming character error rate calculation unit, and a report generation unit.

Images