A small sample dialect corpus expansion and recognition method and system

Abstract

This invention proposes a method and system for expanding and recognizing small-sample dialect corpora. The method includes: acquiring source language corpora and phonological rules of the target dialect; converting the acoustic features of the source language corpora into acoustic features of the target dialect based on the phonological rules to generate expanded dialect corpora; performing quality assessment and screening on the expanded dialect corpora, removing corpora that do not meet preset quality standards to obtain a high-quality expanded dialect corpus; acquiring regional information of the speech to be recognized; adjusting a pre-trained acoustic model based on the regional information; and fine-tuning the adjusted pre-trained acoustic model using the high-quality expanded dialect corpus to obtain a target dialect recognition model; performing speech recognition using the target dialect recognition model; and optimizing the phonological rules and the quality assessment and screening standards based on the confidence level of the recognition results.

Description

Technical Field

[0001] This invention belongs to the field of speech recognition technology, specifically relating to a method and system for expanding and recognizing small-sample dialect corpora. Background Technology

[0002] Dialect speech recognition, as a key technology for intelligent interaction and cultural preservation, is widely used in public services and localized scenarios. With the development of deep learning, related technologies have constructed a small-sample recognition system through the collaborative operation of acoustic feature extraction, data augmentation, and model fine-tuning. Specifically, this system covers the entire process from raw corpus preprocessing to CTC decoding and recognition, including key steps such as noise reduction and normalization, time-frequency domain expansion, and pre-trained model adaptation.

[0003] However, existing corpus augmentation methods mostly employ general acoustic techniques such as noise addition and speed adjustment, without considering the unique phonological rules and tonal prosody of dialects. This results in insufficient authenticity of the generated corpus and its independence from the training of the recognition model. Due to the lack of closed-loop feedback mechanisms and regional variant modeling, low-quality augmented data is prone to model overfitting and struggles to adapt to pronunciation differences across different regions, thus severely limiting the improvement of accuracy and generalization ability of dialect recognition in small-sample scenarios. Summary of the Invention

[0004] The present invention aims to at least partially solve one of the technical problems in the related art.

[0005] Therefore, the first objective of this invention is to propose a method for expanding and recognizing small-sample dialect corpora.

[0006] The second objective of this invention is to propose a small-sample dialect corpus expansion and recognition system.

[0007] The third objective of this invention is to provide a computer device.

[0008] A fourth objective of this invention is to provide a non-transitory computer-readable storage medium.

[0009] To achieve the above objectives, a first aspect of the present invention provides a method for expanding and recognizing small-sample dialect corpora, comprising:

[0010] Obtain the phonological rules of the source language corpus and the target dialect, and convert the acoustic features of the source language corpus into the acoustic features of the target dialect based on the phonological rules to generate an expanded dialect corpus; The expanded dialect corpus is subjected to quality assessment and screening, and the corpus that does not meet the preset quality standards is removed to obtain a high-quality expanded dialect corpus; The geographical information of the speech to be recognized is obtained, the pre-trained acoustic model is adjusted based on the geographical information, and the adjusted pre-trained acoustic model is fine-tuned using the high-quality expanded dialect corpus to obtain the target dialect recognition model. Speech recognition is performed using the target dialect recognition model, and the phonological rules and the quality assessment and screening criteria are optimized based on the confidence level of the recognition results.

[0011] In one embodiment of the present invention, the step of acquiring source language corpus and phonological rules of target dialect, and converting the acoustic features of the source language corpus into acoustic features of target dialect based on the phonological rules to generate expanded dialect corpus, includes: The source language corpus is encoded into a discrete feature sequence containing a semantic layer and a timbre layer, and the semantic layer features are converted into a source language phoneme sequence with tone. Based on the phonological rules of the target dialect, a hybrid system of rule-based expert system and neural network-based adaptive model is used to convert the source language phoneme sequence into the target dialect phoneme sequence, and residual correction is performed on the conversion result. The target dialect phoneme sequence is fused with the timbre layer features and the target dialect prosodic features to generate the expanded dialect corpus.

[0012] In one embodiment of the present invention, the step of performing quality assessment and screening on the expanded dialect corpus, removing corpora that do not meet preset quality standards, and obtaining a high-quality expanded dialect corpus includes: Calculate the acoustic feature similarity between the expanded dialect corpus and the corresponding source language corpus, and remove corpora with similarity below a preset threshold; Perform at least one of objective quality assessment and subjective quality assessment on the remaining corpus to remove corpus that does not meet the preset quality standards, and obtain the high-quality expanded dialect corpus. The preset threshold for similarity and / or the preset quality standard are dynamically adjusted based on the confidence feedback in step S4.

[0013] In one embodiment of the present invention, adjusting the pre-trained acoustic model based on the geographic information includes: The geographic information is encoded into a geographic embedding vector, and the geographic embedding vector is injected into the pre-trained acoustic model to achieve geographic modulation of the model features; or, Based on the regional information, the model branch or parameter group corresponding to the pre-trained acoustic model is switched to adapt to the pronunciation features of the corresponding regional variant.

[0014] In one embodiment of the present invention, the fine-tuning of the adjusted pre-trained acoustic model includes: Freeze the backbone parameters of the pre-trained acoustic model and fine-tune only some parameters of the model using low-rank matrices; Phonological rule constraint loss is introduced during the fine-tuning process to force the model to learn the phonological features of the target dialect.

[0015] In one embodiment of the present invention, optimizing the phonological rules and the quality assessment and screening criteria based on the confidence level of the recognition results includes: Identification result samples with confidence scores below a preset threshold are labeled, and the error types of the samples are statistically analyzed. Based on the statistical results of the error types, the conversion parameters in the phonological rules are automatically updated, or the similarity threshold and quality standards on which the quality assessment and screening are based are adjusted.

[0016] To achieve the above objectives, a second aspect of the present invention provides a small-sample dialect corpus expansion and recognition system, comprising: The corpus expansion module is used to acquire the phonological rules of the source language corpus and the target dialect, and convert the acoustic features of the source language corpus into the acoustic features of the target dialect based on the phonological rules to generate expanded dialect corpus; The quality screening module is used to evaluate and screen the expanded dialect corpus, remove corpora that do not meet the preset quality standards, and obtain a high-quality expanded dialect corpus. The model fine-tuning module is used to obtain the regional information of the speech to be recognized, adjust the pre-trained acoustic model based on the regional information, and fine-tune the adjusted pre-trained acoustic model using the high-quality expanded dialect corpus to obtain the target dialect recognition model. The recognition and feedback module is used to perform speech recognition using the target dialect recognition model, and to optimize the phonological rules and the quality assessment and screening criteria based on the confidence level of the recognition results.

[0017] The present invention provides a method and system for expanding and recognizing small-sample dialect corpora, which can convert large-scale general corpora into high-quality expanded corpora that conform to dialect norms based on phonological rules, thus overcoming the bottleneck of scarce small-sample data. Through regional adaptive fine-tuning and closed-loop feedback mechanisms, it effectively solves the problem of regional variant recognition and prevents model overfitting, significantly improving the dialect recognition accuracy and generalization ability in small-sample scenarios.

[0018] To achieve the above objectives, a third aspect of this application provides a computer device, including a processor and a memory; wherein the processor reads executable program code stored in the memory to run a program corresponding to the executable program code, for implementing a method for expanding and recognizing small sample dialect corpora as described in the first aspect embodiment.

[0019] To achieve the above objectives, a fourth aspect of this application provides a non-transitory computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements a method for expanding and recognizing small-sample dialect corpora as described in the first aspect embodiment.

[0020] Additional aspects and advantages of the invention will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of the invention. Attached Figure Description

[0021] The above and / or additional aspects and advantages of the present invention will become apparent and readily understood from the following description of the embodiments taken in conjunction with the accompanying drawings, wherein: Figure 1 This is a flowchart of a method for expanding and recognizing small-sample dialect corpora according to an embodiment of the present invention; Figure 2 This is a structural diagram of a small-sample dialect corpus expansion and recognition system according to an embodiment of the present invention; Figure 3 It is a computer device according to an embodiment of the present invention. Detailed Implementation

[0022] It should be noted that, unless otherwise specified, the embodiments and features described in the present invention can be combined with each other. The present invention will now be described in detail with reference to the accompanying drawings and embodiments.

[0023] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0024] The following describes, with reference to the accompanying drawings, a method and system for expanding and recognizing small-sample dialect corpora according to embodiments of the present invention.

[0025] Example 1 Figure 1 This is a flowchart of a method for expanding and recognizing small-sample dialect corpora according to an embodiment of the present invention, such as... Figure 1 As shown, it includes: S1, Obtain the phonological rules of the source language corpus and the target dialect, and convert the acoustic features of the source language corpus into the acoustic features of the target dialect based on the phonological rules to generate an expanded dialect corpus; S2, perform quality assessment and screening on the expanded dialect corpus, remove corpora that do not meet the preset quality standards, and obtain a high-quality expanded dialect corpus; S3, obtain the regional information of the speech to be recognized, adjust the pre-trained acoustic model based on the regional information, and fine-tune the adjusted pre-trained acoustic model using the high-quality expanded dialect corpus to obtain the target dialect recognition model. S4. Speech recognition is performed using the target dialect recognition model, and the phonological rules and the quality assessment and screening criteria are optimized based on the confidence level of the recognition results.

[0026] This invention provides a method for expanding and recognizing small-sample dialect corpora. It can convert large-scale general corpora into high-quality expanded corpora that conform to dialect norms based on phonological rules, thus overcoming the bottleneck of scarce small-sample data. Through regional adaptive fine-tuning and closed-loop feedback mechanisms, it effectively solves the problem of regional variant recognition and prevents model overfitting, significantly improving the dialect recognition accuracy and generalization ability in small-sample scenarios.

[0027] Example 2 The following describes in detail an embodiment of the present invention, a system for expanding and recognizing small-sample dialect corpora, with reference to the accompanying drawings.

[0028] This system is an integrated hardware and software system, with its core consisting of six parts: a small-sample dialect corpus input module, a dialect corpus preprocessing module, a phonological perception corpus expansion module, an expanded corpus screening and optimization module, a dialect recognition model module, and a result output and feedback module. These modules are connected sequentially through data interfaces to collaboratively complete the corpus expansion and dialect recognition functions. Among them, the phonological perception corpus expansion module, the expanded corpus screening and optimization module, and the small-sample adaptation part of the dialect recognition model are the core innovations of this invention.

[0029] Appendix Figure 2 The system's overall structure is illustrated in a block diagram, showing the connections and data flow between modules and subsystems. The core principles are described below: The core function of the corpus expansion subsystem is to achieve accurate expansion based on small-sample dialect corpora. Its operational logic starts with the original small-sample dialect corpus, first using an acoustic feature extraction module to denoise the original corpus, removing environmental noise and extracting dialect-specific acoustic features, including Mel spectrum, fundamental frequency, and speech rate, while preserving the original speaker's timbre characteristics. This provides standardized, high-quality input for subsequent cross-language speech conversion. Subsequently, the cross-language speech conversion module, the core innovation of this invention, uses a self-constructed dialect phonological rule base combined with VQ-VAE discrete phoneme coding technology to encode the extracted acoustic features into discrete phoneme sequences. A phonological mapping unit achieves phoneme-level cross-dialect mapping, and a HiFi-GAN synthesis unit generates natural and fluent target dialect speech, thereby expanding the small-sample corpus to 10-20 times its original size without additional manual annotation. Its core principle is to utilize the systematic rules of the dialect phonological system to ensure that the phonemes and tones of the expanded corpus conform to the target dialect. This approach, which standardizes dialects, differs from traditional, simple acoustic enhancement methods. The expanded corpus enters a quality screening closed-loop module. This module calculates the acoustic similarity between the expanded and original corpora, combining quality assessments across three dimensions: signal-to-noise ratio, phoneme accuracy, and naturalness. It filters out invalid and low-quality expanded corpora and receives confidence information from the speech recognition subsystem, dynamically adjusting the screening criteria to form a closed-loop process. This ensures the quality of the expanded corpus matches the training requirements of the recognition model. The high-quality expanded corpus is then stored in a corpus storage module. This module uses distributed storage and index management to achieve orderly storage and rapid retrieval of the corpus, while also recording corpus version information for backtracking and optimization. Finally, the annotation information synchronization module aligns the annotation information of the expanded corpus with the original corpus to ensure annotation consistency. It also receives confidence feedback from the speech recognition subsystem, analyzes annotation issues of misidentified samples, updates the dialect phonology rule base, and synchronizes it to the cross-language speech conversion module, providing optimization data for the next round of corpus expansion.

[0030] The core function of the speech recognition subsystem is to achieve accurate recognition of dialect speech. Its operational logic starts with the dialect speech input by the user. First, it obtains the user's current geographical location through the geolocation module or receives manually input dialect regional information from the user, outputting corresponding regional labels to provide a basis for regional adaptation. This is based on the pronunciation differences of different regional variants of the same dialect, achieving accurate adaptation through regional information. The regional adaptation module, another core innovation of this invention, encodes regional labels as regional embedding vectors and injects them into the acoustic model. It switches the corresponding model branch based on regional information and simultaneously adjusts the input acoustic features for regional adaptation, such as fundamental frequency range mapping, to solve the problem of recognition differences between different regional variants of the same dialect. Subsequently, the acoustic model inference module loads a model pre-trained based on a large-scale multilingual corpus. Employing LoRA few-shot adaptation technology, this approach combines high-quality expanded corpus provided by the corpus augmentation subsystem with the original few-shot corpus to fine-tune the pre-trained model. This eliminates the need for full-parameter training, reducing training costs. Simultaneously, phonological rule constraint loss is introduced to ensure the model accurately learns the phonological features of the target dialect, addressing overfitting issues in few-shot scenarios. The model infers and outputs phoneme / text probability distributions. The decoding output module uses the CTC decoding algorithm, combined with the target dialect language model, to convert the probability distributions into text sequences. After post-processing and standardization, the final recognition results are output. The confidence feedback module calculates the confidence of each recognition result, marks low-confidence samples, statistically analyzes error types, and generates feedback data that is sent to the annotation information synchronization module of the corpus augmentation subsystem, triggering further corpus augmentation and model optimization.

[0031] The two subsystems form a closed-loop optimization through bidirectional data interaction. The core principle is that the high-quality expanded corpus generated by the corpus expansion subsystem is used as training data input to the speech recognition subsystem, providing sufficient and suitable data sources for fine-tuning the recognition model and improving recognition accuracy. The confidence feedback data of the speech recognition subsystem is fed back into the corpus expansion subsystem to update the phonological rule base, adjust the screening criteria, and optimize the quality of corpus expansion, forming a complete closed loop of "corpus expansion → model training → speech recognition → feedback optimization → further expansion". This achieves continuous improvement in system performance and can complete the entire process of automated operation without a lot of manual intervention.

[0032] The corpus augmentation subsystem includes the following modules: (1) Acoustic feature extraction module.

[0033] This module consists of a speech denoising unit, a Mel spectrum extraction unit, and a fundamental frequency / speech rate extraction unit. Its core principle is to purify and extract features from the original small-sample dialect data, removing invalid information while preserving the dialect's unique acoustic features and the original speaker's timbre, providing standardized input for subsequent cross-language speech conversion. In terms of implementation details, the speech denoising unit uses the Wiener filtering algorithm to effectively remove environmental noise, echoes, and other interference. The filtering parameters are set to a window size of 256 and an overlap rate of 50%. The Mel spectrum extraction unit uses the MFCC algorithm to extract 80-dimensional Mel spectrum features, with a uniform sampling rate of 44.1kHz and a bit depth of 16bit to avoid feature errors caused by inconsistent sampling parameters. The fundamental frequency / speech rate extraction unit extracts fundamental frequency (f0) and speech rate features using the Librosa library. The fundamental frequency range is set to 50-500Hz to accommodate pronunciation differences between speakers of different genders. The extracted features are normalized to the [0,1] interval using the Z-score normalization method to eliminate feature differences caused by different speakers and different acquisition environments. Simultaneously, shared memory enables rapid data transmission with the subsequent cross-language speech conversion module. The core function of this module is to provide high-quality, standardized acoustic feature input for cross-language speech conversion, preserving the unique phonological features of dialects and the speaker's timbre, thus laying the foundation for the accuracy of corpus expansion.

[0034] (2) Cross-language corpus conversion module.

[0035] This module, as the core innovation of this invention, adopts a three-layer architecture design, sequentially completing the three core functions of encoding, conversion, and generation. It achieves accurate conversion from input speech to target dialect speech while preserving the original speaker's characteristics and regional style. This part will be described in detail in Section 5, Key Innovations.

[0036] (3) Quality screening closed-loop module.

[0037] This module consists of a similarity calculation unit, a quality assessment unit, and a feedback adjustment unit. Its core principle is to eliminate low-quality expanded corpora through a dual screening mechanism, and at the same time, combine the confidence feedback of the speech recognition subsystem to dynamically adjust the screening criteria, forming a closed-loop optimization to ensure that the expanded corpora are accurately matched with the recognition requirements. In terms of implementation details, the similarity calculation unit uses the cosine similarity algorithm to calculate the acoustic feature similarity between the expanded corpus and the original corpus. The initial threshold is set to 0.8, which can be dynamically adjusted based on confidence feedback. The quality assessment unit adopts a combination of objective and subjective assessment. The objective assessment includes signal-to-noise ratio (≥30dB) and phoneme accuracy (≥95%). The subjective assessment is conducted by three native speakers of the dialect, who score the naturalness of the speech (out of 100 points, threshold ≥80 points). The feedback adjustment unit establishes a real-time feedback channel with the annotation information synchronization module to receive confidence feedback data (JSON format, including error sample ID, error type, and confidence value). If a certain type of phoneme error appears in the low-confidence sample set, the similarity threshold is automatically increased by 0.05-0.1 to optimize the screening criteria. After screening, the corpus is passed through the data verification unit to ensure that there is no redundancy and no errors before being transmitted to the corpus storage module. The core function of this module is to filter out invalid and low-quality augmented corpora, improve the quality of the augmented corpora, and dynamically optimize the screening criteria through feedback adjustment, so as to provide suitable training data for the recognition model.

[0038] (4) Corpus storage module. This module consists of a distributed storage unit, an index management unit, and a version control unit. Its core principle is to achieve orderly storage, rapid retrieval, and version tracking of high-quality expanded corpora, providing a stable and traceable training data source for the speech recognition subsystem. In terms of implementation details, the storage device uses a 1TB SSD, divided into independent partitions for storing corpora and model parameters to avoid storage conflicts. MySQL is used for table partitioning, with data tables divided according to dialect type, regional variant, and corpus quality level. The index is set as a combined index of corpus ID and regional tag, ensuring a corpus retrieval speed of ≤0.1 seconds / record, meeting the real-time access requirements of the recognition model. The stored corpus includes audio files (WAV format), labeled text, and metadata (speaker gender, age, regional variant, and collection time). The version control unit records each iteration of expansion and filtering, and each version saves the corresponding parameter configuration for subsequent backtracking and optimization. It also supports batch import and export of corpora, with export formats supporting WAV and TXT to adapt to the training needs of different models. The core function of this module is to provide stable and efficient training data support for the speech recognition subsystem, achieving orderly management and version tracking of the corpus.

[0039] (5) Annotation information synchronization module.

[0040] This module, as one of the core innovative parts of this invention, consists of an annotation alignment unit, a feedback receiving unit, and a rule update unit. Its core principle is to achieve consistent alignment of the annotation information in the expanded corpus, while simultaneously receiving confidence feedback from the speech recognition subsystem, updating the dialect phonological rule base, and triggering closed-loop optimization of the corpus expansion and model. In terms of implementation details, the annotation alignment unit uses the Dynamic Time Warping (DTW) algorithm to align the annotated text, phonemes, and tones of the expanded corpus with the original corpus, ensuring annotation consistency with an alignment error ≤ 0.01 seconds. The feedback receiving unit receives error samples and error types sent by the confidence feedback module in real time, and statistically analyzes the frequency and deviation values ​​of various errors. The rule update unit compares the phoneme features of the error samples with the corresponding rules in the dialect phonological rule base. If the deviation value exceeds 0.1, it automatically updates the rule parameters and synchronizes them to the cross-language speech conversion module without manual intervention. It also records a rule update log, including update time, update content, and triggered feedback type, facilitating subsequent debugging and optimization. The core function of this module is to connect the recognition feedback and corpus expansion, realize a complete closed loop of "expansion → training → recognition → feedback → further expansion", and continuously optimize the quality of corpus expansion and recognition accuracy.

[0041] The speech recognition subsystem includes the following modules: (1) Geographic location acquisition module.

[0042] This module consists of a GPS positioning unit, an IP address location unit, and a user input unit. Its core principle is to acquire the user's dialect and geographic location information, outputting corresponding geographic tags to provide accurate adaptation data for the geographic adaptation module and solve the problem of recognition differences between different geographic variants of the same dialect. In practice, the GPS positioning unit can quickly obtain the user's current geographic location. If the GPS signal is weak, it automatically switches to the IP address location unit, resolving the user's geographic information through the IP address with accuracy down to the prefecture-level city level. The user input unit supports manual input of geographic information, allowing users to select specific dialect geographic variants (e.g., Cantonese-Guangzhou dialect, Cantonese-Hong Kong dialect). The input information is automatically matched with the system's preset geographic tags. The geographic tags output by this module use the format "dialect type-geographic variant" (e.g., "Cantonese-Guangzhou dialect") and are transmitted to the geographic adaptation module in real time via a data interface with a transmission delay of ≤0.05 seconds, ensuring real-time geographic adaptation. The core function of this module is to provide accurate geographic information input for geographic adaptation, laying the foundation for improving the recognition accuracy of dialect geographic variants.

[0043] (2) Regional adaptive module.

[0044] This module, as one of the core innovative parts of this invention, consists of a regional embedding unit, a model branch switching unit, and a feature adaptation unit. Its core principle is based on regional tags to achieve regional adaptation of the recognition model, adjusting the input acoustic features and model parameters to solve the pronunciation differences between different regional variants of the same dialect, thereby improving recognition generalization ability. In terms of implementation details, the regional embedding unit encodes regional tags into 128-dimensional regional embedding vectors and injects them into the attention layer of the acoustic model. One-hot encoding is used to ensure accurate representation of regional features. The model branch switching unit adopts a multi-branch structure, with different regional variants corresponding to independent model branches. Branch parameters are stored in independent files, and the mapping relationship between regional tags and model branches is maintained through configuration files. When adding a new regional variant, only the corresponding parameter file and mapping relationship need to be added, without modifying the core system code. The feature adaptation unit adjusts parameters such as the fundamental frequency range and speech rate of the input acoustic features according to the regional tags. For example, the fundamental frequency range of Cantonese is adjusted to 100-400Hz, and the fundamental frequency range of Chongqing dialect is adjusted to 120-450Hz, ensuring that the features match the pronunciation characteristics of the regional variants. The core function of this module is to enable the recognition model to accurately adapt to different regional dialect variations and improve the generalization ability of dialect recognition in small sample scenarios.

[0045] (3) Acoustic model reasoning module.

[0046] This module consists of a pre-trained multilingual model unit and a LoRA few-shot adaptation unit. Its core principle is to leverage the general speech recognition capabilities of the pre-trained model, combined with LoRA few-shot adaptation technology, to fine-tune the model using expanded corpora. This addresses the overfitting problem in few-shot scenarios, achieving accurate inference for dialect speech. In terms of implementation details, the pre-trained multilingual model unit loads an XLS-R pre-trained model, which has been pre-trained on a large-scale multilingual corpus (1000 hours of Mandarin, 500 hours of Cantonese, and 300 hours of English). This preserves the core parameters of the pre-trained model, ensuring it possesses general speech recognition capabilities while reducing the difficulty and cost of few-shot fine-tuning. The LoRA few-shot adaptation unit, as the core adaptation component of this module, employs low-rank adaptation technology. It freezes the backbone parameters of the pre-trained model, fine-tuning only the low-rank matrix of the model's attention layer, effectively reducing the number of parameters to be fine-tuned and lowering training costs and the risk of overfitting. The learning rate uses a cosine annealing strategy, decreasing from 5e-5 to 1e-6, ensuring efficient parameter updates in the initial fine-tuning phase while avoiding parameter oscillations later. The training batch size is set to 16 to suit hardware computing power, with 30 training epochs. An early stopping strategy (patience=8) is also employed, automatically stopping training after 8 consecutive epochs of verification without improvement in accuracy, avoiding ineffective training that consumes computing power. During fine-tuning, a phonological rule constraint loss (weight 0.2) is introduced, incorporating dialect phonological rules (such as tone, final consonant, and medial rules) used in the cross-language speech conversion module into the loss function. This forces the model to learn the phonological features of the target dialect, further improving inference accuracy and preventing the model from deviating from dialect norms.

[0047] (4) Decoding output module.

[0048] This module consists of a CTC decoding unit, a language model unit, and a post-processing unit. The core principle is to convert the probability distribution output by the acoustic model inference into a natural and standardized text sequence, and combine the dialect language model to correct decoding errors and improve the accuracy and fluency of the recognized text. In terms of implementation details, the CTC decoding unit converts the phoneme / text probability distribution into an initial text sequence to solve the alignment problem between speech and text. By eliminating repeated phonemes and filling in missing semantics, it ensures that the initial text can initially match the input speech. The language model unit loads the target dialect language model (such as the Cantonese language model), which is trained based on the target dialect corpus (including the original small sample corpus and the augmented corpus), and contains information such as common dialect vocabulary, grammar rules, and spoken language habits. It can effectively correct problems such as phoneme confusion, word order errors, and non-standard spoken language in the decoding process. The weight of the language model is set to 0.3 to balance the outputs of the acoustic model and the language model, ensuring both the alignment accuracy between speech and text and the naturalness and standardization of the text. The post-processing unit performs standardization processing on the initial recognized text, specifically including the standardization of numbers and punctuation marks (such as correcting "one zero" in dialect spoken language to the standard number "10", and "廿" to "20"), the standardization of dialect spoken language vocabulary (such as correcting "系" in Cantonese to the common Chinese "是", and "嘅" to "的"), while filtering out invalid characters (such as meaningless characters corresponding to background noise) and correcting word order deviations, ensuring that the output text conforms to the common Chinese expression standard while retaining the core semantics of the dialect. The CTC decoding unit uses the beam search decoding algorithm, with the beam size set to 10 to balance the decoding efficiency and accuracy. The decoding path pruning threshold is 0.01 to eliminate decoding paths with too low probabilities and reduce invalid calculations. This unit outputs the phoneme / text probability distribution through model inference, providing the basic data for the subsequent decoding output module to ensure that the inference result can accurately reflect the semantic information of the input dialect speech. The core function of this module is to output the final dialect recognition text, ensuring the accuracy, fluency, and standardization of the recognition result to meet the text usage requirements in practical applications.

[0049] (5) Confidence feedback module.

[0050] This module consists of a confidence calculation unit, an error statistics unit, and a feedback generation unit. Its core principle is to quantify the reliability of the recognition results, statistically identify erroneous samples and error types, and generate feedback data to provide a basis for optimizing the corpus expansion subsystem, thus achieving closed-loop iteration of the system. In practical terms, the confidence calculation unit calculates the confidence level of each recognition result based on the probability distribution entropy value output by the acoustic model. The confidence level ranges from 0 to 1, with a threshold of 0.7. Samples below this threshold are marked as low-confidence samples, meaning samples with low reliability and a risk of error. The error statistics unit classifies and statistically analyzes low-confidence samples. Core error types include phoneme errors (such as errors in recognizing entering tone codas and vowel confusion), tone errors (such as Cantonese tone confusion), regional variant errors (such as recognizing Guangzhou dialect as Foshan dialect), and spoken vocabulary recognition errors. It also calculates the frequency and deviation values ​​of each type of error to clarify the key directions for system optimization. The feedback generation unit generates standardized feedback data in JSON format, using core information such as error sample ID, error type, confidence score, and deviation value. This data is then sent to the annotation information synchronization module of the corpus expansion subsystem via a real-time feedback channel. The feedback frequency is synchronized with the recognition rate (one feedback data point is generated for each recognized speech item), ensuring the real-time nature and timeliness of the feedback. Simultaneously, a feedback log is recorded, including feedback time, error statistics, feedback data content, and corresponding corpus information, facilitating subsequent system debugging, parameter optimization, and problem tracing. The core function of this module is to provide accurate feedback data to the corpus expansion subsystem, driving updates to the phonology rule base and adjustments to the filtering criteria, thus establishing a closed loop between speech recognition and corpus expansion and enabling continuous iterative optimization of system performance.

[0051] This implementation process is based on the twin closed-loop architecture of the system's "corpus expansion subsystem + speech recognition subsystem". Taking a small sample scenario of Cantonese (only 100 original Mandarin speech data points are obtained, the target dialect is Cantonese, covering three regional variants of Guangzhou, Foshan and Dongguan) as an example, it describes in detail the complete implementation process of the system from initialization to closed-loop optimization, and clarifies the operation content, parameter configuration, module call order and core requirements of each step. All steps follow the core principles and implementation details of each module, and can be implemented directly without creative work. At the same time, the parameters can be flexibly adjusted according to the target dialect type to adapt to different small sample dialect scenarios.

[0052] Step 1: System initialization and environment configuration (basic preparation stage).

[0053] The core of this step is to complete the hardware and software environment setup, parameter configuration, and basic resource preparation, providing stable support for subsequent corpus expansion and speech recognition. First, hardware equipment is procured and assembled according to the system hardware requirements, ensuring that the hardware performance meets the needs of model training, corpus processing, and real-time inference. Specifically, the configuration includes an Intel Core i7-12700H processor (14 cores, 20 threads, base frequency 2.7GHz, turbo frequency 4.7GHz) for overall system computation and module scheduling; 32GB DDR5 4800MHz memory to ensure parallel operation of multiple modules, large-scale corpus processing, and no memory bottleneck during model loading; and an NVIDIA RTX 3090 graphics card (24GB GDDR6X VRAM), with CUDA 11.6 driver installed and GPU acceleration enabled for model training and inference in the cross-language speech conversion module and the acoustic model inference module. Next, complete the software environment installation and configuration. Install the operating system: Install Ubuntu 20.04 LTS (64-bit). Install basic dependencies: Install Python 3.8, PyTorch 1.12.0 (with torchvision 0.13.0); Librosa 0.10.0, PyAudio 0.2.13; Scikit-learn 1.1.2, Pandas 1.5.2; MySQL 8.0, PyMySQL 1.0.2; PyQt5 5.15.9, FFmpeg 4.4.2. Finally, basic resource preparation and parameter configuration are performed, and the target dialect language system rule base is constructed: a Cantonese language system rule base (XML format) is created in the "Rule Base Directory" to store the rules for the six tones of Cantonese, the rules for the -p / -t / -k entering tone codas, the rules for medial simplification, and the rules for vowel transfer. The parameter thresholds for each rule are defined, and the rule call interface is configured. The pre-trained model weights are downloaded: the XLS-R pre-trained model, the residual VQ-VAE pre-trained weights, and the HiFi-GAN v2 pre-trained weights are downloaded in the "Model Parameter Directory". The XLS-R model loads multilingual pre-trained weights, while the residual VQ-VAE and HiFi-GAN models load cross-lingual pre-trained weights.

[0054] In addition, it is necessary to configure the core parameters of each module: create a parameter configuration file (JSON format) to uniformly configure the parameters of each module and adapt to the small sample scenario of Cantonese. The core parameters are as follows: Acoustic feature extraction module: window size 256, overlap rate 50%, MFCC feature dimension 80, sampling rate 44.1kHz, fundamental frequency range 50-500Hz, feature normalization method Z-score; Cross-language speech conversion module: residual VQ-VAE encoder 4-layer 1D-CNN (number of channels [32,64,128,256]), 3-layer residual VQ (codebook size 1024, dimension 256), frame rate 25Hz; Hybrid expert system basic expert freezing, adaptive expert 2-layer Transformer (number of parameters <1 M), training learning rate 5e-5 (cosine annealing to 1e-6); regional embedding vector 3D one-hot (Guangzhou / Foshan / Dongguan); quality screening closed-loop module: initial similarity threshold 0.8, signal-to-noise ratio ≥30dB, phoneme accuracy ≥95%, speech naturalness scoring threshold ≥80 points; acoustic model inference module: LoRA fine-tuning learning rate 5e-5 (cosine annealing to 1e-6), batch size 16, training 30 rounds, early stopping strategy patience=8, phonological rule constraint loss weight 0.2; CTC decoding beam size=10, pruning threshold 0.01; confidence feedback module: confidence threshold 0.7, feedback data format JSON, feedback frequency synchronized with recognition rate. After configuration, initialize the database, start the MySQL service, create the database "dialect_recognition", create 5 data tables (original corpus table, expanded corpus table, model parameter table, feedback log table, and rule update log table), set up a combined index of corpus ID and regional tag, and ensure that the corpus retrieval speed is ≤0.1 seconds / record.

[0055] Step 2: Collection and preprocessing of raw small sample corpus (corpus preparation stage).

[0056] The core of this step is to acquire a small sample of raw language data, perform data purification and standardization, and provide high-quality input for data expansion. The specific operations are as follows: During data collection, 100 sentences of Mandarin speech data covering commonly used vocabulary and short phrases (5-10 seconds each, without obscure words) should be acquired. In an environment with a signal-to-noise ratio ≥40dB and no echo noise, 5-10 native Mandarin speakers of different genders, aged 20-60, should each collect 10-20 sentences. The collection distance should be 30-50cm from the microphone. After collection, the data should be saved in WAV format (sampling rate 44.1kHz, bit depth 16bit), named as “Speaker ID-Data Sequence Number-Collection Time.wav” and stored in the “Original Data Directory”. In the preprocessing stage, the acoustic feature extraction module is started to read the corpus. The Wiener filtering algorithm is used to reduce noise and obtain cleaned speech. The MFCC algorithm is used to extract 80-Vimel spectrum features and unify the sampling rate and bit depth. The Librosa library is used to extract fundamental frequency (limited to 50-500Hz) and speech rate features, and corpus with abnormal fundamental frequency is removed. All acoustic features are normalized to the [0,1] interval using Z-score. Finally, annotation information is added to each corpus. The annotation information is associated with the normalized features and cleaned speech and stored in the MySQL original corpus table and backed up to the corresponding directory.

[0057] Step 3: The corpus expansion subsystem runs to generate high-quality expanded corpus (corpus expansion stage).

[0058] The core of this step is to launch the corpus expansion subsystem, sequentially calling each module to complete the transformation from the original corpus to the target dialect expanded corpus. Simultaneously, quality screening ensures corpus quality. The corpus expansion subsystem is launched via a Python script, loading parameter configuration files and the Cantonese phonological rule base, and establishing a data transmission channel between modules (shared memory, transmission rate 100MB / s). The specific modules involved are as follows: 1. Encoding layer (neural audio codec) operation: The preprocessed raw Mandarin speech waveform is input into the residual VQ-VAE model, which performs downsampling and feature extraction through a 4-layer 1D-CNN encoder, and then performs 3 layers of residual VQ quantization to generate a discrete semantic token sequence Z=[ ], where Z ( ¹ ) (Semantic layer), Z ( ² ) (Tone layer), Z ( ³ ) Extract and save the details (levels of detail) separately; 2. The conversion layer (phonics conversion core) is running: (1) Phoneme Recognizer: Z ( ¹) The input is a lightweight decoder based on CTC, which parses it into a Mandarin IPA phoneme sequence with tones, and outputs it to a phonology converter. (2) Phonetic converter: Load the Cantonese phonetic rule base, and convert the Mandarin phoneme sequence into the Cantonese phoneme sequence by combining FST hard rules and Transformer soft rules. At the same time, call the hybrid expert system, and based on the annotation information of 100 original corpora, complete the residual correction of phoneme conversion through the collaborative output of basic experts and adaptive experts. (3) Prosody transformation: Extract the fundamental frequency profile, energy envelope, and duration pattern of the original Mandarin corpus, map them to the Cantonese prosody template library, generate smooth Cantonese prosody features R' through spline interpolation, and adjust the duration of the entering syllable (shortened by 30%) and the duration of the sentence-ending modal particle (extended). (4) Quantization encoder: Maps the converted Cantonese phoneme sequence back to the VQ-VAE codebook space to generate Z ( ¹ ) '_cantonese (Cantonese semantic layer token), and Z ( ² ) (Original speaker timbre), R' (Cantonese rhythm), and G (regional embedding, default Guangzhou style) are concatenated and output to the decoding layer.

[0059] (5) Decoding layer (neural vocoder) operation: The features output from the conversion layer are input into the improved HiFi-GAN v2 vocoder. The features are upsampled to 16kHz through a 4-layer MRF upsampling module. Combined with MPD+MRD adversarial training, a high-quality Cantonese speech waveform (WAV format) is generated, preserving the original speaker's timbre and Guangzhou regional style. (6) Batch conversion: Batch conversion of 100 original corpora to generate 1000 initial expanded corpora, which are stored in "Expanded Corpus Directory - Initial Subdirectory".

[0060] 3. Quality screening and closed-loop adjustment.

[0061] (1) Similarity calculation: Call the similarity calculation unit, use the cosine similarity algorithm to calculate the acoustic feature similarity between each initial expanded corpus and the corresponding original corpus, and remove corpus with similarity <0.8; (2) Quality assessment: Objective assessment: Call the quality assessment unit to test the signal-to-noise ratio of each corpus (removing corpus with <30dB) and phoneme accuracy (removing corpus with <95%); Subjective assessment: Three native Cantonese speakers score the naturalness of the remaining corpus (out of 100 points), and remove corpus with a score <80 points. (3) Screening results: 800 high-quality expanded corpora were selected from the initial 1000 expanded corpora and stored in the "Expanded Corpus Directory - High-Quality Subdirectory"; (4) Initial feedback adjustment: If the number of high-quality corpora after screening is less than 8 times that of the original corpora (i.e. <800), the similarity threshold will be automatically reduced by 0.05 and the corpora will be screened again.

[0062] 4. Corpus storage and annotation are synchronized.

[0063] (1) Corpus storage: Call the corpus storage module to import 800 high-quality expanded corpus (speech files, acoustic features) and annotation information into the MySQL expanded corpus table in batches, create an index, and back up to "Expanded Corpus Directory - High-Quality Subdirectory" and enable version control; (2) Annotation information synchronization: Call the annotation information synchronization module, use the DTW algorithm to align the annotated text, phonemes, and tones of the expanded corpus with the original corpus to ensure annotation consistency (alignment error ≤ 0.01 seconds); synchronously update the call log of the Cantonese phonological rule base.

[0064] Step 4: The speech recognition subsystem runs, completing dialect recognition and confidence feedback (recognition and feedback stage).

[0065] The core of this step is to start the speech recognition subsystem, call various modules to complete Cantonese speech recognition, and generate confidence feedback data to provide a basis for optimizing the corpus expansion subsystem. The specific operations are as follows: Start the speech recognition subsystem.

[0066] The speech recognition subsystem is started by a Python script, which loads the parameter configuration file and pre-trained model weights (XLS-R model) and establishes a bidirectional data channel with the corpus expansion subsystem.

[0067] Model fine-tuning.

[0068] (1) Load the pre-trained model: Load the XLS-R pre-trained model, freeze the backbone parameters of the model, and retain only the low-rank matrix of the attention layer for training; (2) Prepare training data: Read the acoustic features and labeled text of 800 high-quality expanded corpora from the MySQL expanded corpus table, and divide them into training set (640 entries) and validation set (160 entries) in an 8:2 ratio. (3) Model fine-tuning: Set training parameters (batch size 16, learning rate 5e-5, cosine annealing to 1e-6, training rounds 30), introduce tone system rule constraint loss (weight 0.2), start fine-tuning training; adopt early stopping strategy, if the accuracy does not improve for 8 consecutive rounds, automatically stop training; (4) Model saving: Save the fine-tuned model parameters to the "Model Parameters Directory" and name it "xls-r-dialect-finetune.pth".

[0069] 3. Dialect speech recognition (real-world application scenarios).

[0070] (1) Location acquisition: Call the location acquisition module to obtain the user's current location (assuming it is Guangzhou) through GPS positioning, and output the region label "Cantonese-Guangzhou dialect"; if the GPS signal is weak, it will automatically switch to IP address resolution; users can manually enter the region label. (2) Regional adaptation: Call the regional adaptation module to encode the regional label "Cantonese-Guangzhou dialect" into a 128-dimensional one-hot regional embedding vector and inject it into the attention layer of the acoustic model; switch to the Guangzhou dialect model branch and adjust the fundamental frequency range of the input acoustic features (100-400Hz). (3) Acoustic model inference: The acoustic features of the Cantonese speech to be recognized are input into the fine-tuned XLS-R model, and the phoneme / text probability distribution is output through the LoRA adaptation layer and the CTC decoding unit. (4) Decoding output: Call the decoding output module, input the probability distribution into the CTC decoding unit to generate the initial recognition text; load the Cantonese language model to correct phoneme confusion, word order errors and other problems; through the post-processing unit, standardize the initial text (numbers, punctuation, colloquial vocabulary standardization) and output the final recognition text; (5) Recognition and verification: Compare the recognized text with the labeled text of the speech to be recognized, calculate the recognition accuracy, and ensure that the accuracy is ≥82%; batch recognize 100 speech samples to be recognized and record the recognition results.

[0071] 4. Confidence feedback generation and transmission.

[0072] (1) Confidence calculation: Based on the probability distribution entropy value output by the acoustic model, calculate the confidence level (range 0-1) of each recognition result and mark low confidence samples with confidence level <0.7; (2) Error statistics: Classify low-confidence samples, count error types (phoneme errors, tone errors, regional variant errors, etc.), and calculate the frequency and deviation of each type of error; (3) Feedback generation: The error sample ID, error type, confidence value, deviation value and other information are generated in JSON format to generate feedback data. Each low confidence sample corresponds to 1 feedback data. (4) Data transmission: Feedback data is transmitted in real time to the annotation information synchronization module of the corpus expansion subsystem through a two-way data channel, and the feedback log is stored in the MySQL feedback log table.

[0073] Step 5: Closed-loop optimization and continuous iteration (optimization phase).

[0074] The core of this step is to optimize the rules and parameters of the corpus augmentation subsystem based on the confidence feedback of the speech recognition subsystem, achieve closed-loop iteration of the system, and improve the quality of corpus augmentation and recognition accuracy. The specific operations are as follows: (1) Rule and parameter update: Call the rule update unit of the annotation information synchronization module, receive confidence feedback data, compare the phoneme features of the erroneous sample with the corresponding rules in the Cantonese phonological rule library, if the deviation value is >0.1, automatically update the corresponding phonological rule parameters, update the rule library synchronously, and record the rule update log; at the same time, adjust the similarity threshold of the quality screening closed-loop module (e.g., increase the initial threshold from 0.8 to 0.85). (2) New round of corpus expansion: Based on the updated phonological rule base and screening criteria, the corpus expansion subsystem is restarted to expand 100 original corpora in a new round, generating new high-quality expanded corpora, replacing the original expanded corpora, and updating the corpus; (3) Model readjustment: Call the acoustic model inference module of the speech recognition subsystem, use the new expanded corpus to readjust the model and optimize the model parameters; (4) Iterative loop: Repeat the process of “identification → feedback → rule update → expansion → model fine-tuning”. After each iteration, the recognition accuracy is calculated until the recognition accuracy is stable at ≥85%, and then the iteration stops. The above closed loop process can be repeated according to new original corpus and new regional variant requirements.

[0075] Step 6: System deployment and maintenance (application implementation phase).

[0076] The core of this step is to complete the actual deployment and daily maintenance of the system to ensure stable operation and adaptability to actual application scenarios. The specific operations are as follows: (1) System deployment: Deploy the optimized system (including all modules, model parameters, rule base, and database) to the target device, configure the automatic startup script, set up the remote monitoring interface, and monitor the running status of each module in real time; (2) Daily maintenance: Corpus maintenance, regularly clean up the corpus, delete invalid and redundant corpus, supplement new original small sample corpus, trigger a new round of closed-loop optimization; regularly update the weights of the pre-trained model, optimize and fine-tune the parameters to adapt to new dialect scenarios; regularly back up the MySQL database logs and system operation logs, and clean up expired logs; if module operation is abnormal, troubleshoot the problem through the logs, adjust the parameters or repair the rule base to ensure that the system can quickly resume operation.

[0077] (3) Scene adaptation: If other dialects (such as Sichuan dialect and Wenzhou dialect) need to be adapted, only the phonological rule library needs to be replaced, the regional embedding vector and related parameters need to be adjusted, and the above implementation steps need to be repeated. There is no need to modify the core code of the system.

[0078] The above outlines the complete implementation steps of the system, covering the entire process from initialization to closed-loop optimization, deployment, and maintenance. Each step clearly defines the operational details, parameter configurations, and module calling logic, strictly adhering to the core principles of each module. Technical personnel in the relevant field can directly implement the system according to the above steps to achieve accurate expansion and efficient recognition of small-sample dialect data, thus achieving the invention's objectives.

[0079] The key technical points of this invention are as follows: First, a cross-language corpus conversion module, which realizes the conversion between Mandarin and small sample dialects; second, a speech recognition module, which realizes acoustic reasoning based on geographical location and regional adaptive module; and third, a confidence feedback mechanism, which combines dialect conversion rules with human confidence scoring to achieve synchronous correspondence between prediction expansion and recognition.

[0080] Cross-language corpus conversion module: This module, the core innovation of this invention, consists of a three-layer architecture, sequentially performing the three core functions of encoding, conversion, and generation. It achieves accurate conversion from input speech to target dialect speech while preserving the original speaker's characteristics and regional style. The overall process is as follows: the input Mandarin speech sequentially passes through the first encoding layer, the second conversion layer, and the third decoding layer, ultimately outputting the target dialect speech. The architecture and functional details of each layer are as follows: First layer: Coding layer (neural audio codec).

[0081] The function of this layer is to convert continuous Mandarin speech waveforms into discrete semantic tokens, laying the foundation for cross-linguistic phoneme representation. The core technology chosen is Residual VQ-VAE. Compared to single-layer VQ-VAE, which is prone to large quantization loss and significant sound quality degradation, Residual VQ-VAE achieves complementary advantages through multi-layer progressive quantization. The top layer captures phonological semantic features, while the bottom layer captures speech detail features, ensuring the integrity and naturalness of the converted speech.

[0082] The specific architecture parameters are as follows: The encoder adopts a 4-layer 1D-CNN structure with a convolution stride of 2 and the number of channels configured sequentially as [32, 64, 128, 256], progressively downsampling and extracting features from the input speech; the quantization layer contains a 3-layer residual VQ structure, with each layer having a codebook size of 1024 and a codebook dimension of 256, achieving accurate encoding of features at different levels through layered quantization; the decoder adopts a 4-layer transposed CNN structure, forming a symmetrical structure with the encoder, realizing the restoration from discrete tokens to feature vectors; the overall frame rate is set to 25Hz, that is, one frame of token is generated every 40ms, which is precisely matched with the phoneme generation rate to ensure the temporal consistency of subsequent conversion processes.

[0083] This layer employs a cross-lingual pre-training strategy to enhance generalization ability. The pre-training objectives include two core aspects: First, a multilingual reconstruction task, which takes inputs in multiple languages ​​such as Mandarin, Cantonese, and English, and reconstructs the original speech waveform through the model. At the same time, it constrains the same phoneme to be mapped to similar codebook vectors in different languages, thereby achieving cross-lingual semantic alignment. Second, cross-lingual alignment constraints, which calculate the phonological similarity between phonemes through IPA (International Phonetic Alphabet) features, construct an alignment loss function, and strengthen the feature association between phonemes in different languages.

[0084]

[0085] The pre-training data covers multilingual scenarios: for Mandarin, the AISHELL-1 and THCHS-30 datasets (total 150 hours) are used; for Cantonese, the CantoMap and HKUST datasets (total 50 hours) are used; and for English, the Librispeech dataset (100 hours, used to enhance the model's generalization ability) is used.

[0086] The final output is represented as follows: For the input speech X, the encoder outputs a discrete semantic token sequence:

[0087] The three-layer codebook features are further decomposed into three levels: This is a semantic layer token, corresponding to core semantic information related to the phonological system; For the timbre layer token, corresponding to the timbre characteristics of the original speaker; These are detail layer tokens, corresponding to auxiliary detail information such as noise and breath in speech, providing hierarchical feature support for subsequent conversion and generation.

[0088] Second layer: Conversion layer (phonetic conversion core).

[0089] This layer is the core innovation module for cross-lingual speech conversion. Its core function is to perform cross-dialect phonological conversion on the semantic layer token sequence output by the first-layer encoding, achieving accurate conversion from Mandarin to the target dialect (such as Cantonese), while adapting to the learning needs of small-sample corpora and regional style differences. The overall process sequentially passes through three sub-units: a phoneme recognizer, a phonological converter, and a quantization encoder. The specific details are as follows: Phoneme recognizer: The core function of this unit is to map the discrete semantic tokens output by the neural audio encoding to an interpretable IPA (International Phonetic Alphabet) phoneme sequence, providing a clear semantic basis for subsequent phonological conversion. The implementation method is a lightweight decoder based on CTC. By using the formula to calculate the probability distribution of the phonemes corresponding to the tokens, it outputs a Mandarin phoneme sequence with tone information; during the training stage, a Mandarin phoneme annotation dataset with forced alignment is used to ensure the accuracy of phoneme recognition. The output example is to parse the sequence into tone-bearing phoneme combinations such as [ni3, hao3], accurately restoring the phonological features of Mandarin.

[0090] Phonological converter: This unit is the core for achieving phonological conversion from Mandarin to the target dialect. It adopts a hybrid system of a finite-state transducer (FST) and a neural network, and through the combination of deterministic rules and probabilistic models, it completes the accurate conversion of phonological features while adapting to the small-sample learning scenario.

[0091] Phonological rules are divided into two major types: one is deterministic phonological mapping (hard rules), which formulate standardized conversion rules for fixed phonological differences between dialects, and solidify the core conversion logic in tabular form. For example, Mandarin tone T1 (55Hz) is mapped to Cantonese T1 (53 / 55Hz) ("妈 ma1" → "maa1"), Mandarin words without coda are predicted to have an entering tone coda in Cantonese ("十 shi2" → "sap6"), and the medial / i / in Mandarin is simplified and deleted in Cantonese ("家 jia1" → "gaa1"), etc.; the other is probabilistic sound change (soft rules), which, for phonological variants in some complex contexts, makes dynamic decisions through a neural network. The formula is , calculates the phoneme conversion probability based on context information; the specific implementation uses a conditional generation model based on Transformer to complete the conversion from the Mandarin phoneme sequence to the Cantonese phoneme sequence.

[0092] To address the learning challenges in small-sample scenarios, this unit introduces a Mixture of Experts (MoE) system to achieve accurate adaptation: the base expert is a pre-trained rule system, forming an FST rule base based on a large-scale linguistic knowledge base, covering the systematic phonological correspondence between Mandarin and the target dialect. Parameters are frozen after training to ensure the accuracy of the base conversion. The adaptation expert is a trainable two-layer Transformer model with parameters controlled within 1M. Inputs are the output of the base expert and contextual information, and the output is a residual correction for the base conversion, adapting to personalized phonological features in the small-sample corpus. A gating network dynamically fuses the outputs of the two experts based on the complexity of the input phonemes; simple phoneme conversions prioritize the base expert, while complex contexts activate the adaptation expert for correction. The training phase is optimized using only 100 small-sample paired corpora, with a loss function of... The cross-entropy loss ensures the consistency between the transformation result and the real corpus, while the KL divergence constraint ensures that the output of the expert does not deviate too far from the basic expert, thus avoiding overfitting in small sample scenarios.

[0093] In addition, this unit also includes a prosodic conversion function. The core is to preserve the original speaker's prosodic intention while adapting to the prosodic pattern of the target dialect. Specifically, it converts three major prosodic features: In terms of fundamental frequency contour (F0 contour), it first extracts the F0 curve of Mandarin, identifies tone inflection points, and then maps them to the six-tone template library of the target dialect. It then generates a smooth F0 curve of the target dialect through spline interpolation. In terms of duration pattern, it automatically adjusts based on phonological rules. For example, the duration of entering tone syllables is shortened by 30%, and the duration of sentence-final particles is appropriately extended to ensure the prosodic naturalness and dialect adaptability of the converted speech.

[0094] Quantization Encoder: The core function of this unit is to map the converted target dialect phoneme sequence back to the codebook space of VQ-VAE, generate a semantic feature sequence adapted to the third layer of decoding, provide standardized discrete token input for subsequent speech generation, and ensure that the decoding layer can accurately restore the phonological features of the target dialect speech.

[0095] The third layer: the decoding layer (neural vocoder).

[0096] The function of this layer is to integrate the discrete semantic tokens, timbre features, prosodic features, and other information output by the conversion layer and restore them into a high-quality target dialect speech waveform. The core technology selected is an improved version of the HiFi-GAN v2 vocoder, which improves the sound quality and naturalness of the generated speech through adversarial training.

[0097] The model's conditional input consists of four parts: the transformed quantized semantic features. The original speaker's vocal characteristics are preserved when copying from the source speech. The transformed prosodic features The generator input consists of four parts: a target dialect feature vector (G) and a regional embedding vector (G) (corresponding to different regional styles such as Guangzhou, Foshan, and Dongguan). These features are concatenated to ensure that the generated speech simultaneously includes the target dialect features, the original speaker's identity, and the regional style. The generator structure employs a multi-receptive field upsampling (MRF) module, containing a four-layer upsampling structure that progressively increases the sampling rate of the feature sequence to 16kHz, achieving the restoration of high-dimensional speech waveforms from low-dimensional features. The discriminator uses a combination of a multi-period discriminator (MPD) and a multi-receptive field discriminator (MRD). Adversarial training is used to enhance the detail and naturalness of the generated speech, avoiding issues such as noise and distortion.

[0098] This layer achieves precise control over regional styles through geo-embedding: geo-embedding uses 3D one-hot vectors to represent different regions (Guangzhou / Foshan / Dongguan), and injects the geo-embedded vectors into each layer of the generator through a FiLM (Feature Linear Modulation) layer, using the formula... Generate modulation parameters, and then through A linear transformation is applied to the features, where G is the regional one-hot encoding, enabling the same model to output different regional accents. For different regions, the Guangzhou style is characterized by a slightly lower pitch and a slightly faster speech rate, the Foshan style is characterized by some vowels exhibiting nasalization features, and the Dongguan style is characterized by an overall lower pitch, meeting the needs of dialect speech generation in different regional scenarios.

[0099] Geographic fusion acoustic reasoning and feedback mechanism: Traditional speech recognition treats geographic location as external metadata, only performing simple model selection later. This invention uses geographic location as a dynamic prior deeply embedded in the acoustic inference process. The acoustic inference module leverages hardware advantages and integrates regional probability calculations of phoneme priors. A three-layer adaptive architecture is adopted: a geographic semantic layer, a phonological prior layer, and an acoustic inference layer.

[0100] First, geographic location information is acquired through the device and input into the geographic semantic layer. This module uses soft geographic boundary modeling and a Gaussian mixture model to calculate the probability of geographic affiliation.

[0101] in Using the coordinates of the regional center, Define the boundary range for the covariance matrix. As a priori weight for the region, The probability density function is Gaussian. When the calculated maximum regional affiliation probability max P(Geo|x) < 0.6, multi-region parallel recognition is triggered, and the output candidate results are reordered by the language model.

[0102] To achieve dynamic adjustment for regional adaptation, the present invention integrates the calculation of regional probability and the prior of the final phoneme. Meanwhile, the adaptive coefficient α is dynamically adjusted according to the location confidence. The specific process is as follows: First, integrate the calculation of regional probability to obtain the fused phoneme prior:

[0103] Among them, P(Geo = g|x) is the attribution probability of region g output by the geographical semantic layer, is the prior probability of phoneme ph corresponding to region g output by the phonological prior layer, is the fused phoneme prior.

[0104] Then, combine the general phoneme prior to obtain the final phoneme prior:

[0105] Among them, is the general phoneme prior (without distinguishing regions), is the adaptive coefficient, which is dynamically adjusted according to the location confidence Conf_geo. The specific adjustment rules are as follows: When Conf_geo > 0.8 (high location confidence): α = 0.7 (strong regional adaptation, mainly relying on the regional phoneme prior); When 0.5 < Conf_geo ≤ 0.8 (medium location confidence): α = 0.5 Conf_geo (medium regional adaptation, dynamically matching the location confidence); When Conf_geo ≤ 0.5 (low location confidence): α = 0.2 (weak regional adaptation, mainly relying on the general phoneme prior).

[0106] To integrate the regional phoneme prior into the recognition process, the present invention corrects the output of the acoustic model for each decoding frame. The specific formula is as follows:

[0107] Among them, is the phoneme originally output by the acoustic model at the probability of the t-th frame, is the final phoneme prior after dynamic weighting, β is the prior intensity coefficient (used to adjust the influence weight of the phoneme prior), is the context-related correction term (used to combine the context phoneme relationship and correct the phoneme recognition error), is the corrected phoneme probability, ensuring that the decoding process fully combines the regional phonological characteristics and context information.

[0108] To achieve deep fusion of geographic information and acoustic features, this invention injects geographic information into the Conformer encoder through a FiLM (Feature Linear Modulation) layer. The specific process is as follows:

[0109] in, The geographic probability distribution output by the geographic semantic layer is transformed into modulation parameters γ (scaling factor) and β (offset factor) by the MLP (Multilayer Perceptron). LayerNorm(X) is the layer normalization result of the encoder input feature X. By scaling the normalized feature with γ and offsetting it with β, the regional information is dynamically modulated to the acoustic feature, enabling the encoder to adapt to the pronunciation features of different regions.

[0110] After completing dialect recognition, the confidence feedback module calculates the machine confidence score (based on the probability distribution entropy value output by the acoustic model) for each recognition result. Simultaneously, it combines human confidence scores for the recognition results and the expanded corpus to comprehensively assess the recognition effect and the quality of the expanded corpus. Low-confidence samples (machine confidence score < 0.7 or human score < 80 points) are marked, and error types (phoneme errors, tone errors, regional adaptation errors, etc.) and deviation values ​​are statistically analyzed. This feedback information is synchronized to the annotation information synchronization module of the corpus expansion subsystem. This module dynamically corrects the dialect conversion rules (including phonological mapping rules, tone conversion rules, prosody conversion parameters, etc.) based on the error distribution and deviation values. Simultaneously, the quality screening closed-loop module adjusts the similarity threshold and quality threshold for corpus screening based on the confidence score, ensuring that the new round of expanded corpus focuses on addressing the weaknesses of the recognition model (such as high-frequency erroneous phonemes and regional adaptation deviations). This mechanism enables the corpus expansion rules to be dynamically optimized based on the recognition results, and the recognition model to be continuously improved as the quality of the expanded corpus increases. The two form a strong correspondence and strong synchronization in a collaborative optimization relationship, which completely solves the problem of the two being disconnected in the existing technology.

[0111] To achieve the core objectives of this invention—automatic expansion of dialect corpora under small sample conditions, high-precision dialect recognition, and continuous optimization of the closed-loop system—there are various complete or partial alternative technical solutions that can achieve the same technical effect. These solutions can achieve the same inventive effect without using the specific technical means of this invention by replacing some modules, algorithms, model architectures, or implementation processes. The specific situations are as follows.

[0112] At the overall technical level, there are implementation methods that can completely replace the dual-system architecture of this invention. For example, a multilingual speech synthesis model combined with small-sample prompting learning and timbre replication technology can be used to directly generate expanded corpora with the same speaker and regional style based on a small amount of original dialect data, without the need to build a dedicated dialect phonological rule base and VQ. The VAE discrete phoneme coding structure, relying on the generalization ability of large models, can expand the corpus size to 10 to 20 from the original. This approach, combined with lightweight fine-tuning of a pre-trained speech recognition model, feeds back the confidence results from the recognition process to the synthesis stage to optimize the generation quality, achieving continuous improvement in dialect recognition accuracy even with limited samples. Alternatively, a pure acoustic data augmentation approach can be adopted, without cross-dialect or cross-language conversion of the original corpus. Instead, conventional acoustic enhancement techniques such as speech rate adjustment, volume transformation, reverberation simulation, spectrum distortion, and fundamental frequency perturbation are used to expand the corpus size. This is combined with domain adaptive algorithms to handle pronunciation differences in different regions of the same dialect. The enhanced data is used to train the recognition model and construct an iterative optimization process, which can also alleviate the overfitting problem in small sample training and meet the practical needs of dialect recognition. Another approach is to use an integrated large model combined with active learning architecture, eliminating the independent corpus expansion subsystem and directly using a multilingual speech recognition large model. The model filters low-confidence recognition samples, and incremental fine-tuning is performed after a small amount of manual review. The accent embedding module is used to adapt to different regional dialect variants, and pseudo-label iteration replaces the dedicated corpus expansion process. This approach can also achieve high-precision dialect recognition and system performance iteration even in scenarios with scarce samples.

[0113] At the level of local modules and technical steps, there are also numerous alternative solutions that will not affect the achievement of the overall invention objective. In the core step of corpus expansion, this invention employs VQ... VAE Discrete Encoding, Phonological Rule Mapping and HiFi GAN synthesis structures can be synthesized by VITS, Soft Other discrete representation methods, such as VQ and HuBERT deep feature encoding, can be replaced, and the transformation method combining phonological rules and Transformer can also be achieved using CycleGAN. VC, StarGAN End-to-end speech conversion models such as VC or direct speech translation models can be used instead. The vocoder part can also use models such as WaveNet, WaveGlow, and BigVSAN to achieve speech generation with equivalent sound quality. In the acoustic feature extraction module, Wiener filtering noise reduction can be replaced by spectral subtraction, wavelet noise reduction, or deep learning noise reduction models. MFCC features can be replaced by deep acoustic features such as Fbank, WavLM, and HuBERT. Fundamental frequency extraction can also be achieved using other algorithms such as PYIN and CREPE, all of which can achieve the goal of corpus purification and standardized feature extraction. In the quality screening and closed-loop optimization part, the cosine similarity calculation used in this invention can be replaced by KL divergence, model embedding distance, sentence error rate calculation, etc. Multi-dimensional quality assessment can replace manual scoring with an automatic MOS prediction model, relying on the logic of dynamically adjusting the screening threshold based on recognition confidence. Alternatively, a reinforcement learning reward mechanism can achieve the same effect.

[0114] There are also many alternatives to the dialect recognition module. A location-driven regional adaptation mechanism can replace the regional embedding vector and multi-branch model structure with accent embedding, domain adaptive adapter, conditional normalization, etc., achieving regional variant adaptation without the need to divide independent model branches; for the acoustic model fine-tuning stage, XLS... The R pre-trained model can be replaced with general speech models such as Whisper, WavLM, and Conformer. LoRA lightweight fine-tuning can be replaced with adapter fine-tuning, prefix fine-tuning, and partial parameter fine-tuning. Phonological rule constraint loss can also be replaced by dictionary constraints and regularization constraints, which can also reduce training costs and avoid small sample overfitting. In the decoding and confidence feedback parts, CTC decoding can be replaced with RNN. T, joint attention decoding, and the confidence calculation based on entropy can be replaced by N. Methods such as best-case difference comparison and empty token probability calculation can be used to optimize the feedback process, which can also be achieved through weakly supervised learning and self-training iteration without affecting the closed-loop optimization effect. Furthermore, the dual-system bidirectional closed-loop architecture of this invention can also be replaced by an integrated single-model architecture, a knowledge distillation architecture, or a federated learning architecture, integrating corpus expansion and recognition reasoning within the same system. Iteration can be completed through pseudo-label generation and error feedback, achieving automated performance improvement without significant manual intervention.

[0115] Whether used as a complete technical approach or as a partial module replacement, the above-mentioned alternatives can achieve the core objectives of this invention: efficient expansion of dialect corpora in small sample scenarios, accurate dialect recognition, and system closed-loop optimization.

[0116] To implement the methods of the above embodiments, the present invention also provides a computer device, such as... Figure 3 As shown, the computer device 600 includes a memory 601 and a processor 602; wherein, the processor 602 reads executable program code stored in the memory 601 to run a program corresponding to the executable program code, so as to implement the various steps of the method described above.

[0117] To implement the above embodiments, this application also proposes a non-transitory computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the method described in the foregoing embodiments.

[0118] In the description of this specification, the references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., refer to specific features, structures, materials, or characteristics described in connection with that embodiment or example, which are included in at least one embodiment or example of the present invention. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. Moreover, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification, as well as the features of different embodiments or examples.

[0119] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one of that feature. In the description of this invention, "a plurality of" means at least two, such as two, three, etc., unless otherwise explicitly specified.

Claims

1. A method for expanding and recognizing small-sample dialect corpora, characterized in that, Includes the following steps: Obtain the phonological rules of the source language corpus and the target dialect, and convert the acoustic features of the source language corpus into the acoustic features of the target dialect based on the phonological rules to generate an expanded dialect corpus; The expanded dialect corpus is subjected to quality assessment and screening, and the corpus that does not meet the preset quality standards is removed to obtain a high-quality expanded dialect corpus; The geographical information of the speech to be recognized is obtained, the pre-trained acoustic model is adjusted based on the geographical information, and the adjusted pre-trained acoustic model is fine-tuned using the high-quality expanded dialect corpus to obtain the target dialect recognition model. Speech recognition is performed using the target dialect recognition model, and the phonological rules and the quality assessment and screening criteria are optimized based on the confidence level of the recognition results.

2. The method as described in claim 1, characterized in that, The process of acquiring source language corpus and target dialect phonological rules, and converting the acoustic features of the source language corpus into the acoustic features of the target dialect based on the phonological rules to generate expanded dialect corpus, includes: The source language corpus is encoded into a discrete feature sequence containing a semantic layer and a timbre layer, and the semantic layer features are converted into a source language phoneme sequence with tone. Based on the phonological rules of the target dialect, a hybrid system of rule-based expert system and neural network-based adaptive model is used to convert the source language phoneme sequence into the target dialect phoneme sequence, and residual correction is performed on the conversion result. The target dialect phoneme sequence is fused with the timbre layer features and the target dialect prosodic features to generate the expanded dialect corpus.

3. The method as described in claim 1, characterized in that, The process of quality assessment and screening of the expanded dialect corpus, removing corpora that do not meet preset quality standards, and obtaining a high-quality expanded dialect corpus includes: Calculate the acoustic feature similarity between the expanded dialect corpus and the corresponding source language corpus, and remove corpora with similarity below a preset threshold; Perform at least one of objective quality assessment and subjective quality assessment on the remaining corpus to remove corpus that does not meet the preset quality standards, and obtain the high-quality expanded dialect corpus. The preset threshold for similarity and / or the preset quality standard are dynamically adjusted based on the confidence feedback in step S4.

4. The method as described in claim 1, characterized in that, The adjustment of the pre-trained acoustic model based on the geographical information includes: The geographic information is encoded into a geographic embedding vector, and the geographic embedding vector is injected into the pre-trained acoustic model to achieve geographic modulation of the model features; or, Based on the regional information, the model branch or parameter group corresponding to the pre-trained acoustic model is switched to adapt to the pronunciation features of the corresponding regional variant.

5. The method according to claim 1, characterized in that, The fine-tuning of the adjusted pre-trained acoustic model includes: Freeze the backbone parameters of the pre-trained acoustic model and fine-tune only some parameters of the model using low-rank matrices; Phonological rule constraint loss is introduced during the fine-tuning process to force the model to learn the phonological features of the target dialect.

6. The method as described in claim 1, characterized in that, The optimization of the phonological rules and the quality assessment and screening criteria based on the confidence level of the recognition results includes: Identification result samples with confidence scores below a preset threshold are labeled, and the error types of the samples are statistically analyzed. Based on the statistical results of the error types, the conversion parameters in the phonological rules are automatically updated, or the similarity threshold and quality standards on which the quality assessment and screening are based are adjusted.

7. A system for expanding and recognizing small-sample dialect corpora, characterized in that, include: The corpus expansion module is used to acquire the phonological rules of the source language corpus and the target dialect, and convert the acoustic features of the source language corpus into the acoustic features of the target dialect based on the phonological rules to generate expanded dialect corpus; The quality screening module is used to evaluate and screen the expanded dialect corpus, remove corpora that do not meet the preset quality standards, and obtain a high-quality expanded dialect corpus. The model fine-tuning module is used to obtain the regional information of the speech to be recognized, adjust the pre-trained acoustic model based on the regional information, and fine-tune the adjusted pre-trained acoustic model using the high-quality expanded dialect corpus to obtain the target dialect recognition model. The recognition and feedback module is used to perform speech recognition using the target dialect recognition model, and to optimize the phonological rules and the quality assessment and screening criteria based on the confidence level of the recognition results.

8. The system as described in claim 7, characterized in that, The corpus expansion module is used for: The source language corpus is encoded into a discrete feature sequence containing a semantic layer and a timbre layer, and the semantic layer features are converted into a source language phoneme sequence with tone. Based on the phonological rules of the target dialect, a hybrid system of rule-based expert system and neural network-based adaptive model is used to convert the source language phoneme sequence into the target dialect phoneme sequence, and residual correction is performed on the conversion result. The target dialect phoneme sequence is fused with the timbre layer features and the target dialect prosodic features to generate the expanded dialect corpus.

9. A computer device, characterized in that, Including processor and memory; The processor reads executable program code stored in the memory to run a program corresponding to the executable program code, so as to implement a method for expanding and recognizing small sample dialect corpora as described in any one of claims 1-6.

10. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When executed by the processor, the program implements a method for expanding and recognizing small-sample dialect corpora as described in any one of claims 1-6.

Images