A model training method for minnan dialect recognition
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- XIAMEN TIANCONG INTELLIGENT SOFTWARE CO LTD
- Filing Date
- 2026-05-21
- Publication Date
- 2026-07-14
Smart Images
Figure CN122392494A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of speech recognition model training technology, specifically to a method for training a Minnan (Southern Fujian) speech recognition model for low-resource dialects. Background Technology
[0002] Minnan dialect, as a regional dialect that retains many features of ancient Chinese, possesses extremely high cultural research and daily use value. While automatic speech recognition (ASR) technology is rapidly developing and standard Mandarin speech recognition has achieved large-scale commercial application, Minnan dialect is a typical low-resource dialect, and related recognition technologies in the industry still suffer from many unavoidable technical shortcomings. Minnan dialect exhibits regional pronunciation differences, with variations even within short distances, making it extremely difficult to uniformly annotate speech samples across the entire region. Existing mainstream dialect databases all use Minnan-specific scripts for speech annotation, but different institutions' Minnan script systems are not interchangeable, resulting in extremely poor data compatibility. Furthermore, dialect-specific script annotation requires a high level of professional dialect literacy from the annotators, leading to long annotation cycles, high labor costs, and an inability to quickly expand large-scale annotated corpora.
[0003] To build a Minnan dialect recognition system, the classic approach is a hybrid framework based on Hidden Markov Models (HMMs), commonly trained using the Kaldi tool. First, a Minnan dialect dictionary is designed, then the acoustic and language models are merged into a Weighted Finite State Transformer (WFST) decoder. This hybrid architecture is suitable for scenarios with limited training data, but it requires a specially designed dictionary, making maintenance cumbersome. Furthermore, the model consumes a huge amount of memory after loading, typically exceeding 20GB, which is unsuitable for lightweight deployments on mobile devices and embedded systems.
[0004] Most existing end-to-end (E2E) dialect recognition models rely on dialect phonemes and Minnan characters for modeling and training, resulting in dialect text as the final output. Ordinary users cannot intuitively understand the recognized content, leading to poor practical applicability. Currently, most systems only use Minnan characters for modeling, making it difficult to display as Mandarin text, which is also detrimental to comprehension. Furthermore, the limited Minnan annotation data often results in poor training performance for end-to-end systems, sometimes even failing to surpass hybrid framework systems. Some existing patented technologies rely on Mandarin phonemes for dialect modeling, requiring additional forced conversion from dialect text to common phonemes. This conversion process is prone to semantic bias and necessitates the addition of alignment models, further increasing training costs and the complexity of the system architecture.
[0005] Most mainstream speech recognition models are not specifically adapted to Minnan dialect recognition scenarios. A few adapted models suffer from problems such as low recognition accuracy, semantic illusion, and language misidentification. At the same time, under low resource conditions, conventional end-to-end model training converges slowly, and the recognition accuracy cannot surpass that of traditional hybrid architectures, which seriously restricts the practical application of Minnan dialect speech recognition technology.
[0006] Chinese patent CN202110615995 proposes a method, system, device, and medium for Minnan (Hokkien) speech recognition. It uses Mandarin phonemes as modeling units, significantly reducing the number of phoneme sequences compared to traditional methods using Minnan phonemes. During training, the Connected Temporal Classification (CTC) standard is used, combined with Conditional Random Fields (CRF) to optimize the acoustic model. During recognition, the speech feature sequence to be recognized is input into the trained acoustic model to obtain the probability of different predicted phoneme sequences corresponding to the speech feature sequence. This probability is then combined with the language model to decode and search for the optimal recognition result. However, this method requires converting Minnan characters into phonemes, which introduces inaccuracies. Furthermore, it necessitates separate training of the acoustic and language models, making the process complex and maintenance cumbersome.
[0007] Chinese patent 202411486618 proposes a dialect recognition method, apparatus, device, and program product. The dialect speech recognition model is obtained through dialect speech recognition training based on dialect speech samples and dialect text prediction training based on dialect text pronunciation samples. The dialect text pronunciation samples include the corresponding Mandarin pronunciation of the dialect text. Its purpose is to improve the dialect speech recognition effect. However, it requires generating dialect text from Mandarin text and generating a mapping table from dialect text to Mandarin pronunciation; then, dialect text pronunciation samples corresponding to the dialect text are generated based on the mapping table. Additionally, a phoneme forced alignment model is required, increasing the training cost. Summary of the Invention
[0008] The purpose of this invention is to address the problems in existing Minnan dialect speech recognition technologies, such as high annotation costs, poor data compatibility, bloated model structures, high deployment difficulty, weak readability of recognition results, insufficient recognition accuracy in low-resource scenarios, and poor recognition ability for mixed languages. The invention provides a low-resource Minnan dialect speech recognition model training method that features a simple training process, low annotation threshold, high recognition accuracy, lightweight and easy deployment, and adaptability to multiple scenarios.
[0009] To achieve the above-mentioned objectives, the present invention adopts the following technical solution:
[0010] A model training method for Minnan dialect recognition is proposed. The core model relies on the Zipformer-Transducer streaming end-to-end speech recognition architecture. This model integrates a multi-scale compressed attention mechanism and eliminates the separate construction mode of independent dialect dictionaries, independent acoustic models, and independent language models, simplifying the overall model structure. At the data level, it abandons the industry-standard Minnan character annotation and dialect phoneme annotation mode and uses Mandarin Chinese characters to perform unified semantic annotation of Minnan speech samples across the entire domain, reducing the annotation difficulty and cost from the source.
[0011] A hierarchical progressive training system is built at the model training level. First, the model base is pre-trained based on a massive general Chinese speech corpus, enabling the model to learn the basic acoustic features of general Chinese. Then, according to the quality classification of Minnan speech data, a three-level hierarchical fine-tuning training is completed from coarse to fine, gradually realizing the optimization of the entire process of dialect pronunciation feature adaptation, sample generalization ability improvement, and model noise correction calibration. Finally, combined with a semi-supervised learning mode, high-confidence pseudo-labels are generated using unlabeled folk Minnan speech corpus to continuously expand the number of training samples. Relying on iterative training, the model recognition error rate is continuously compressed, solving the core pain point of scarce Minnan speech labeled corpus.
[0012] Meanwhile, this invention adds a standardized dual preprocessing flow for both speech and text, unifies the format and quality of all training samples, eliminates inferior and invalid training samples, and ensures the stability of model training. The trained recognition model can be simultaneously compatible with three types of actual use scenarios: pure Minnan dialect, pure Mandarin, and mixed Minnan and Mandarin speech. All recognition results are uniformly output as standard Mandarin Chinese text, which is in line with the reading habits of the general public.
[0013] Compared with the prior art, the present invention has the following outstanding technical effects and advantages:
[0014] The annotation cost is greatly reduced: the entire process of annotation is completed using Mandarin Chinese characters, which eliminates the need for annotators to have professional knowledge of Minnan characters and dialect phonemes. The annotation threshold is significantly lowered, the annotation efficiency is improved, and the incompatibility problem of dialect annotation systems of different institutions is completely solved.
[0015] Lightweight model deployment: It adopts an integrated Zipformer-Transducer end-to-end architecture. The model has a built-in multi-scale compressed attention mechanism, eliminating the need to split and build multiple independent functional models and dedicated dialect dictionaries. The overall model size is small and the memory usage is low, making it suitable for lightweight deployment in multiple scenarios such as cloud, edge devices, and mobile terminals.
[0016] Outstanding adaptability to low-resource conditions: Relying on a triple training strategy of pre-trained transfer learning, multi-stage fine-tuning, and semi-supervised pseudo-label iteration, the model can still quickly complete convergence training under the condition of limited artificially labeled Minnan dialect data, effectively improving the accuracy of dialect recognition.
[0017] The recognition results are highly practical: the recognition results uniformly output standard Mandarin Chinese text, abandoning the obscure and difficult-to-understand Minnan script output mode, and ordinary users can directly read the recognized content, greatly improving the actual user experience.
[0018] Wide range of scenarios: It can accurately recognize single Minnan dialect, single Mandarin, and mixed Minnan and Mandarin speech commonly found in daily communication, which fits the actual scenarios of daily oral communication in the Minnan region.
[0019] High versatility and scalability: The overall training process and model adaptation logic of this invention can be directly transferred and applied to speech recognition training of various Southeast low-resource dialects such as Hakka, Fuzhou, and Putian, making it widely applicable. Attached Figure Description
[0020] Figure 1 This is a schematic diagram of the overall structure of the Minnan dialect speech recognition system in an embodiment of the present invention;
[0021] Figure 2 This is a schematic diagram of the pre-training-multi-stage fine-tuning-semi-supervised iterative progressive training strategy in an embodiment of the present invention. Detailed Implementation
[0022] The present invention will be further described in detail below with reference to specific embodiments. Those skilled in the art can completely replicate the invention based on the content described herein.
[0023] The embodiments of the present invention include the following steps:
[0024] Step 1: Acquisition and Standardization Preprocessing of Multi-Source Minnan Dialect Speech Data
[0025] 1.1 Data Collection Classification
[0026] The invention involves the batch collection of four types of raw Minnan dialect speech data: the first type is manually tagged speech data recorded and proofread by professionals; the second type is coarse-quality speech data from film and television dramas and local programs in the Minnan region; the third type is pseudo-labeled speech data generated by preliminary basic model inference; and the fourth type is Minnan dialect TTS synthesized speech data generated based on speech synthesis technology. In other words, the Minnan dialect data sources of this invention cover ASR annotation (pseudo-labeled data), film and television subtitles (coarse-quality data), manually tagged (tagged data), and text-to-speech (TTS) synthesized data (tagged data).
[0027] 1.2 Hierarchical Data Classification
[0028] All collected data were divided into three levels—high-precision, normal, and poor-quality—based on speech clarity, annotation accuracy, and pronunciation standard. Poor-quality samples were removed, and high-precision and normal-quality samples were retained for subsequent training.
[0029] 1.3 Unified Preprocessing Operations
[0030] (1) Text preprocessing: All the labeled texts corresponding to the samples are uniformly standardized, non-standard characters, punctuation and redundant modal words are removed, the text format is unified, special garbled symbols, dialect redundant modal words without actual meaning and messy and redundant punctuation marks are deleted, and all labeled texts are uniformly organized into standard simplified Mandarin Chinese characters to ensure that the text format is completely unified.
[0031] (2)Speech preprocessing: Filter out abnormal samples (too fast / slow) based on the speech rate distribution, perform long speech segmentation and silence removal to ensure the quality consistency of the input data; count the normal speech rate range of all speech samples (the normal speech rate range can be set to 120 - 240 words per minute), and filter out abnormal and invalid speech samples with too fast or too slow speech rates; uniformly segment long speeches with durations exceeding the set threshold into fixed short-duration speech segments (long speeches with durations exceeding 15s can be uniformly segmented into fixed 3 - 8s short-duration speech segments); accurately detect and remove silent sections at the beginning and end of the speech samples that are greater than 200ms to ensure the integrity and consistency of the speech features input to the model.
[0032] 1.4 Unified annotation execution
[0033] Different from the prior art, the present invention abandons the Minnan character annotation scheme and phoneme system, and uses Mandarin characters to annotate Minnan dialect speech. For example, the Minnan dialect spoken language "你要食甚物" is directly annotated as the Mandarin Chinese characters "你要吃什么". No Minnan-exclusive characters or dialect-specific phonemes are used as the annotation carriers throughout the process, significantly reducing the annotation cost and shortening the annotation time. As Figure 1 shown, the Minnan dialect recognition result also directly outputs the Mandarin Chinese character text, which is easier for users to understand and provides a more friendly experience. At the same time, the present invention jointly models Minnan dialect and Mandarin, enabling the trained Minnan dialect recognition system to also recognize Mandarin, effectively dealing with the phenomenon of mixed languages.
[0034] Step 2: Pre-training of the Zipformer-Transducer model base
[0035] Select Zipformer-Transducer as the only core training model. This model incorporates a multi-scale compression attention mechanism, combining the high-precision advantages of Transformer-like models with low computational cost and low latency, and also has a natural streaming real-time speech inference ability. The overall structure is an integrated end-to-end structure, without the need to independently build external dedicated dialect acoustic models, language models, and dialect dictionaries. This model is customized for training on this low-resource Minnan dialect. It not only inherits the natural streaming inference ability of the Transducer framework but also realizes the comprehensive advantages of fast speed, low memory occupancy, and high recognition accuracy through the efficient encoder design of Zipformer.
[0036] As Figure 2 shown, at the uppermost stream of the training process, there is a data input node "a large number of Chinese language datasets (including Minnan dialect)". In this stage, based on this dataset, mixed corpus pre-training work is first carried out, corresponding to<The "Hybrid Corpus Pre-training (Base Model)" process node. This stage utilizes massive amounts of annotated Chinese speech data (mainly Mandarin, supplemented by a small amount of Minnan dialect) to learn general Chinese acoustic features, compensate for the lack of Minnan dialect data, and construct a base model with strong generalization ability. A massive amount of publicly available, universally annotated Mandarin speech data is selected as the main pre-training corpus, combined with a small batch of pre-processed, precisely annotated Minnan dialect speech data to form a hybrid pre-training corpus. Full-scale pre-training is conducted with general Chinese speech recognition as the training objective. In this embodiment, the pre-training learning rate is set to 1e-4, with 80 iterations, allowing the model to fully learn general Chinese pronunciation features, basic syllable features, and basic semantic features. After training, a base model with strong generalization ability is obtained, completing the accumulation of basic features in the early stages of dialect training and compensating for the lack of Minnan dialect-specific data.
[0037] Step 3: Three-level tiered progressive fine-tuning training
[0038] Based on the pre-trained model, fine-tuning of dialect adaptation is carried out in three levels according to data quality from low to high. The overall training layout logic is as follows: Figure 2 As shown, the present invention adopts a closed-loop progressive training process of pre-training - multi-stage fine-tuning - semi-supervised iteration. Figure 2 The model is arranged with three levels of fine-tuning data input nodes and model output nodes: coarse-quality + fine-labeled dataset, pseudo-label + coarse-quality + fine-labeled dataset, and fine-labeled dataset. These correspond to the first, second, and third stages of fine-tuning, respectively, with the purity of the training data and the training accuracy increasing progressively. The fine-tuning stage uses the Minnan dialect corpus as the core training data, combined with standard Mandarin annotated corpus for joint training. The model is trained in stages according to the quality level of the Minnan dialect data.
[0039] 3.1 First-level basic dialect adaptation and fine-tuning
[0040] This step corresponds to Figure 2 The process involves inputting the "coarse-quality + fine-labeled dataset" into the "first-stage fine-tuning (Pretrained_base)". Based on the base model, coarse-quality and fine-labeled data are used for training. In this embodiment, the fine-tuning learning rate is set to 5e-5 to allow the model to initially adapt to dialect features, resulting in the dialect Pretrained_base model. During training, a joint training set is formed by retrieving ordinary-level film and television coarse-quality speech data and high-precision artificially labeled speech data. This allows the model to initially adapt to the unique pronunciation intonation and regional pronunciation habits of Minnan dialect, completing the basic dialect feature implantation and obtaining a primary dialect adaptation model.
[0041] 3.2 Fine-tuning of the second-level global generalization capability
[0042] This step corresponds to Figure 2 The process flow is as follows: inputting "pseudo-labels + coarse-quality + fine-labeled datasets" to "second-stage fine-tuning (Pretrained)". Large-scale pseudo-label data is introduced on top of the Pretrained_base, and training is performed using coarse-quality and fine-labeled data. The learning rate is lowered to 2e-5 to improve the model's generalization ability in diverse scenarios, resulting in the Pretrained version. Based on the initial dialect adaptation model, a large-scale introduction of cleaned pseudo-labeled speech data, combined with coarse-quality speech data and manually calibrated speech data, forms an expanded training set. This further trains the model's network weights, enriching the Minnan dialect pronunciation and spoken sentence scenarios encountered by the model. This comprehensively improves the model's recognition and generalization ability under different regional accents and communication scenarios, resulting in an intermediate-optimized dialect recognition model.
[0043] 3.3 Third-level high-precision error correction calibration fine-tuning
[0044] This step corresponds to Figure 2 The process flow from inputting the "precisely labeled dataset" to the "third-stage fine-tuning" is described. While the introduction of large-scale pseudo-labeled data effectively expanded the training corpus in the pretrained version, it also introduced the risk of noise accumulation and model confusion. Therefore, this invention sets up a fine-tuning stage based on high-quality data, with a learning rate of 1e-5, aiming to "correct" the model and calibrate its features. All low-quality coarse data and noisy pseudo-labeled data are removed, and only the precisely labeled data is used as the sole training sample for the final fine-tuning training. This training stage mainly performs global error correction and calibration on the semantic noise and pronunciation recognition deviations introduced by the pseudo-labeled data during the second-stage training, correcting the model's recognition bias.
[0045] Meanwhile, in this training level, additional Hokkien TTS synthesized speech data is added. Using TTS synthesized data enhancement strategies, domain adaptation is performed for some special scenarios (such as number recognition and literary written language) to solve the problem of insufficient coverage of general language corpus. Targeted adaptation training is completed for niche scenarios such as digital broadcasting, traditional written Hokkien, and folk-specific spoken language, comprehensively filling the blind spots in the scene coverage of general language corpus. Finally, a high-precision and mature Hokkien recognition model is trained.
[0046] Step 4: Semi-supervised iterative optimization training
[0047] 4.1 Labeling for Inference Using Unlabeled Data
[0048] We collect a large number of unlabeled native Minnan dialect spoken voices from publicly available online sources and daily recordings. We then use a high-precision recognition model with third-level fine-tuning to perform offline batch inference and recognition of the unlabeled voices to generate initial semantic labels.
[0049] 4.2 Screening of high-confidence samples
[0050] Set a threshold for inference confidence (in this embodiment, the threshold is set to 0.85) to retain only pseudo-labeled speech samples with inference confidence that reach the standard threshold. Then, perform a second cleaning of speech and text to remove unqualified samples with inference errors and semantic deviations.
[0051] 4.3 Iterative Training
[0052] Combination Figure 2 As can be seen, this invention sets up a closed-loop feedback structure at the end of the three-stage fine-tuning, which is labeled as "semi-supervised iteration (Finetune model is used to generate pseudo-labels, clean and label the data, and iterate the obtained dataset in the second and third stages of fine-tuning)". This invention further adopts a semi-supervised learning mechanism, using the Finetune model trained in the third stage of fine-tuning to perform inference and recognition on unlabeled or low-quality data, generating pseudo-labels with higher confidence. After cleaning, new data is added to the training set for iterative training to achieve self-evolution of model performance. The newly selected qualified pseudo-labeled samples are incorporated into the original training dataset, and the second and third stages of fine-tuning training process in step three are repeated according to the closed-loop logic to complete one round of model iteration optimization; multiple rounds of iterative iteration are performed according to this mode to continuously expand high-quality training samples and continuously reduce the model word error rate. After multiple rounds of iteration, the Minnan dialect recognition performance is tested. In this embodiment, when the word error rate fluctuation is less than 0.5% for 5 consecutive rounds of iteration, the convergence threshold is reached, the final version is output, and the model recognition performance is determined to have fully converged, finally generating Figure 2 The final version of the model is marked with a "final version" annotation at the end.
[0053] Step 5: Model Deployment and Practical Recognition Application
[0054] 5.1 Lightweight Model Export
[0055] The final Minnan dialect speech recognition model, after training convergence, is exported in a lightweight format, removing redundant training structures and retaining the pure inference runtime architecture.
[0056] 5.2 Multi-scenario speech recognition application
[0057] The exported model is deployed to a speech recognition terminal device. Users directly input the audio signal to be recognized. After the audio signal is processed by front-end speech noise reduction, it is input into the model. The model relies on the built-in trained feature mapping logic to directly complete the extraction of Minnan speech features and semantic decoding. No additional translation and conversion module is required throughout the process. It directly outputs standard Mandarin Chinese character recognition text that is consistent with the meaning of Minnan speech.
[0058] 5.3 Mixed Speech Compatibility Recognition
[0059] During actual use, the model can automatically distinguish the Minnan dialect pronunciation segments and Mandarin pronunciation segments in the input speech. After unified feature recognition, it integrates and outputs a coherent and smooth Mandarin Chinese text, perfectly adapting to the speech recognition scenario of daily Minnan-Mandarin mixed conversations in Minnan regions.
[0060] Figure 1 The following is a schematic diagram of the overall system structure for Minnan dialect speech recognition. The audio signals to be recognized input by the user can be divided into three categories: ① Pure Minnan dialect speech (such as the spoken language "你要食甚物"); ② Pure Mandarin speech (such as the spoken language "你要吃什么"); ③ Minnan-Mandarin mixed speech. Both types of speech inputs are sent into the same core model. After the audio signals are processed by front-end speech noise reduction, they are input into the E2E ASR (such as Zipformer-Transducer) module. Relying on the built-in feature mapping logic completed through training, it directly completes the extraction of Minnan dialect speech features and semantic decoding without the need for an additional translation and conversion module. The model directly outputs a standard Mandarin Chinese text that is semantically consistent with the input speech. For example, for the Minnan dialect "你要食甚物" and Mandarin "你要吃什么", the final recognition result is both "你要吃什么", which can be directly read by the user without any reading comprehension obstacles.
[0061] Compatible recognition of mixed speech: During actual use, the model can automatically distinguish the Minnan dialect pronunciation segments and Mandarin pronunciation segments in the input speech. After unified feature recognition, it integrates and outputs a coherent and smooth Mandarin Chinese text, perfectly adapting to the speech recognition scenario of daily Minnan-Mandarin mixed conversations in Minnan regions.
[0062] When the present invention is actually used, the speech signals in the on-site environment are directly collected as the input content. The collected speech can be either pure Minnan dialect speech, standard Mandarin speech in daily conversations, or the most common Minnan-Mandarin mixed speech in Minnan regions. The collected speech signals are first subjected to front-end feature extraction and then sent into the already deployed Zipformer-Transducer end-to-end recognition model.
[0063] Relying on various speech feature and semantic mapping relationships learned in the early stage, the model independently completes the whole process of deep speech feature extraction, language feature differentiation, and semantic sequence decoding. The entire operation process does not require additional intermediate processing links such as dialect text translation, phoneme forced alignment, and dedicated dictionary matching. After the operation is completed, it directly outputs a standard Mandarin Chinese text that is completely semantically corresponding to the input speech. Even if the speech content has a situation of alternating use of dialect and Mandarin, the model can independently distinguish different language pronunciation segments and integrate and generate a smooth and semantically accurate Mandarin text content, fully meeting the speech recognition usage requirements in real life. The solution of the present invention is not only applicable to Minnan dialect recognition but also applicable to other dialects such as Hakka dialect, Fuzhou dialect, and Putian dialect.
[0064] Simulation implementation verification:
[0065] Under the same hardware operating environment, the speech recognition model trained by this invention was compared with the traditional HMM hybrid architecture recognition model and the conventional end-to-end dialect recognition model in a comparative simulation test. The test samples covered a variety of practical application scenarios, including everyday Minnan dialect, professional terms, and mixed Minnan and Mandarin speech.
[0066] Tests have verified that this invention adopts an integrated network architecture with multi-scale compressed attention mechanism, which does not require additional configuration of dialect-specific dictionaries and multiple independent split function models. The model deployment consumes less memory resources and has significant advantages in lightweight deployment. Under the condition of training with low-resource, small-sample, manually tagged corpora, the model of this invention has higher convergence efficiency, higher overall dialect speech recognition accuracy, and lower speech recognition word error rate.
[0067] This invention adopts a unified annotation method for Mandarin Chinese characters, which significantly reduces the threshold for annotating dialect speech data, effectively shortens the dataset construction cycle, and reduces the overall R&D and training costs. The model can stably adapt to speech recognition scenarios of pure Minnan dialect, pure Mandarin, and mixed Minnan and Mandarin, and outputs standard Mandarin text in a unified manner. The recognition results are intuitive and easy to understand, and have strong user adaptability.
[0068] In summary, this invention effectively addresses many shortcomings of existing low-resource Minnan dialect speech recognition technologies by optimizing data construction patterns, unifying annotation standards, and combining a training strategy that integrates pre-training, three-level progressive fine-tuning, and semi-supervised iteration. This invention demonstrates strong stability, good applicability, and broad prospects for practical promotion and application.
[0069] The above description is merely a preferred embodiment of the present invention and is not intended to limit the scope of protection of the present invention. The training learning rate, iteration rounds, confidence threshold, and speech processing related parameters described herein are preferred values. Those skilled in the art can flexibly adjust the corresponding parameters according to actual application needs, and such reasonable adjustments fall within the scope of protection of the present invention. Any equivalent transformations, simplified substitutions, and process optimizations made within the spirit and principles of the technical solution of the present invention should be covered within the scope of protection defined by the claims of the present invention.
Claims
1. A model training method for Minnan dialect recognition, characterized in that, Includes the following steps: S1. Collect multi-source Minnan dialect speech data and complete data annotation and unified preprocessing to construct a hierarchical Minnan dialect speech training dataset; S2. Select Zipformer-Transducer as the basic recognition model and use a general Chinese-Min mixed corpus to complete the pre-training of the model base; S3. Based on the quality level of the dataset, conduct multi-stage hierarchical fine-tuning training on the pre-trained base model to obtain the dialect adaptation base model. S4. A semi-supervised learning approach is adopted to perform semi-supervised iteration using unlabeled Minnan dialect speech data. The training samples are iteratively expanded and the dialect adaptation basic model is continuously optimized until the model recognition index reaches the preset convergence threshold. S5. Deploy and apply the final recognition model after convergence. Input the pure Minnan dialect speech, pure Mandarin speech, or mixed Minnan and Mandarin speech to be recognized, and directly output the corresponding semantic Mandarin Chinese character recognition text.
2. The model training method for Minnan dialect recognition according to claim 1, characterized in that, The data annotation method in S1 is as follows: all Minnan dialect speech samples are semantically annotated using standard Mandarin Chinese characters, without using Minnan-specific characters or dialect phonemes. The annotated text corresponds one-to-one with the actual semantic expression of the Minnan dialect speech.
3. The model training method for Minnan dialect recognition according to claim 1, characterized in that, The multi-source Minnan dialect speech data in S1 includes four categories: manually tagged speech data, coarse-quality speech data from film and television subtitles, speech pseudo-label data, and Minnan dialect TTS synthesized speech data. Data preprocessing includes text preprocessing and speech preprocessing.
4. The model training method for Minnan dialect recognition according to claim 3, characterized in that, The text preprocessing involves removing abnormal characters, redundant modal particles, and disordered punctuation marks from the samples, and standardizing the text to a standard simplified Chinese format; the speech preprocessing involves filtering speech samples with abnormal speech rates, segmenting long speech segments, and removing silent sections at the beginning and end of the speech.
5. The model training method for Minnan dialect recognition according to claim 1, characterized in that, The S2 base pre-training uses a massive amount of Mandarin speech data combined with a small amount of Minnan speech data for joint training. In the pre-training stage, it learns general Chinese acoustic features and builds a model base with cross-dialect generalization ability.
6. The model training method for Minnan dialect recognition according to claim 1, characterized in that, The S3 multi-stage hierarchical fine-tuning training is specifically divided into three levels of training: the first level combines coarse-quality data and fine-labeled data to complete the adaptation of basic dialect features; the second level introduces pseudo-labeled data to expand the sample size and improve the model's generalization ability; the third level uses only high-precision artificial fine-labeled data to complete model noise correction and feature calibration.
7. The model training method for Minnan dialect recognition according to claim 6, characterized in that, During the third-level fine-tuning training, Minnan dialect TTS synthesized speech data was added to complete the specialized domain adaptation training for digital speech and written speech.
8. The model training method for Minnan dialect recognition according to claim 1, characterized in that, The S4 semi-supervised iteration specifically involves: using the optimized dialect adaptation base model to perform offline inference on unlabeled Minnan speech to generate high-confidence pseudo-labels, selecting qualified high-confidence pseudo-label samples and incorporating them into the original training dataset, performing multi-level fine-tuning training again, and iteratively optimizing the model parameters.
9. The model training method for Minnan dialect recognition according to claim 1, characterized in that, The Zipformer-Transducer in S2 is an integrated end-to-end speech recognition architecture that integrates a multi-scale compressed attention mechanism. It does not require the separate construction of dedicated dialect acoustic models, dialect language models, and dialect-specific dictionaries. It directly completes the end-to-end mapping of speech features to Mandarin text, making it suitable for low-resource devices and low-latency recognition application scenarios.
10. The model training method for Minnan dialect recognition according to claim 1, characterized in that, The final recognition model that converges in S4 has the ability to recognize three types of speech: pure Minnan dialect speech, pure Mandarin speech, and mixed Minnan and Mandarin speech. During the recognition stage, the model directly maps the acoustic features of the three types of speech into standard Mandarin Chinese text and outputs it.