A speech recognition method and post-processing system for low-resource languages

Abstract

This invention discloses a speech recognition and post-processing method and system for low-resource languages, belonging to the field of speech recognition and natural language processing technology. First, candidate transcription results are obtained by recognizing the input speech signal. A bias correction library is constructed based on high-frequency error patterns from a statistical model of labeled data. New candidates are generated through approximate matching to enhance the results. Then, a basic large-scale language model is trained to be customized for low-resource languages. Constrained inference is achieved by combining structured prompts and multiple constraints, outputting the optimal transcription result. The system includes modules for candidate result acquisition, bias correction library construction, candidate result enhancement, language model inference optimization, and result output. This invention does not significantly increase the cost of labeled data, can significantly reduce the recognition error rate, avoid miscorrection, improve the accuracy and stability of the results, and can be quickly adapted to various low-resource languages, possessing strong engineering practicality.

Description

Technical Field

[0001] This invention relates to a speech recognition and post-processing method and system for low-resource languages, belonging to the field of computer speech recognition and natural language processing technology. Background Technology

[0002] Low-resource languages ​​refer to languages ​​lacking large-scale labeled speech-text corpora and with weak foundational research in related technologies. Speech recognition tasks for these languages ​​face challenges such as low recognition accuracy, complex error patterns, and poor model generalization ability. Traditional speech recognition techniques largely rely on training models with large-scale labeled data, which struggles to achieve ideal results for low-resource languages. Existing post-processing techniques primarily use general language models for correction, failing to optimize for the systematic recognition biases inherent in low-resource languages, leading to problems such as word-level errors and grammatical errors in the models.

[0003] In addition, low-resource languages ​​are characterized by numerous dialect variations, significant accent differences, and substantial impact from ambient noise. Existing technologies lack targeted deviation correction mechanisms, making it difficult to effectively adapt to these complex scenarios and ultimately affecting the reliability and practicality of speech recognition results. Summary of the Invention

[0004] The purpose of this invention is to address the technical problems of existing low-resource language speech recognition technologies, such as limited training data scale, significant systematic recognition bias in models, high error rate of speech recognition results, and insufficient post-processing error correction capabilities. This invention creatively proposes a speech recognition and post-processing method and system for low-resource languages. Without significantly increasing the cost of annotated speech data, this invention can effectively improve the accuracy, stability, and practicality of low-resource language speech recognition results.

[0005] The innovations of this invention include: constructing a systematic bias correction library for speech recognition models in low-resource language scenarios, introducing a candidate result enhancement mechanism, and combining it with a large language model specializing in low-resource languages ​​for constrained inference optimization.

[0006] A speech recognition and post-processing method for low-resource languages ​​includes the following steps:

[0007] Step S1: Speech recognition and candidate result acquisition.

[0008] First, speech recognition processing is performed on the input low-resource language speech signal. Low-resource languages ​​are those lacking large-scale labeled corpora and for which technical research is relatively weak.

[0009] Next, it is determined whether the speech recognition model used supports multiple candidate outputs. If it does, the N-best transcription result output by the model is obtained, and the weight or confidence value corresponding to each candidate result is saved simultaneously when the model has the ability to output candidate weights or confidence scores; if it does not support this, only the 1-best transcription result with the highest confidence score output by the model is obtained as the initial candidate result.

[0010] Step S2: Construct a deviation correction library.

[0011] First, we collected labeled low-resource language speech-text pairs. The data included clear speech signals and corresponding accurately labeled text to ensure the reliability and stability of the statistical analysis results.

[0012] Then, using the same speech recognition model and model parameters as in step S1, the labeled speech data is transcribed to obtain the model transcription result, so that the transcription result has consistent systematic bias characteristics with the output of the inference stage.

[0013] Next, the model transcription results are aligned sentence by sentence with the corresponding manually annotated reference text. The differences between the two are calculated at the character and word levels (which can be achieved using the minimum edit distance algorithm). Three types of error operations are identified: insertion, deletion, and replacement. The error text, the corrected text, and the position information in the sentence corresponding to each error operation are recorded.

[0014] Based on error operation records, statistical analysis is performed on error patterns. A word-level confusion statistics table is constructed, using word-level errors as the statistical object. Simultaneously, a frequency threshold for error occurrence is set, retaining only high-frequency error patterns that occur more frequently than the threshold to eliminate random errors.

[0015] Based on the high-frequency word-level error patterns obtained through screening, a deviation correction library is established. The deviation correction library includes at least the erroneous text fragment, the corresponding corrected text fragment, and the error occurrence frequency information, which are used for approximate matching and candidate generation in the subsequent candidate result enhancement stage.

[0016] Step S3: Enhance candidate results.

[0017] Based on the candidate transcription results obtained in step S1, the candidate transcription results are segmented using words as the basic processing unit to obtain several candidate word-level text fragments.

[0018] The candidate word-level text fragments are approximated by word-level error patterns in the deviation correction library. During the matching process, a word-level difference not exceeding a preset CER threshold is allowed between the candidate text fragments and the error patterns. When the CER is less than the threshold, the match is considered successful.

[0019] When a successfully matched word-level error pattern is detected, the original candidate transcription result is not deterministically corrected. Instead, at least one new candidate transcription result is generated by replacing the error pattern with the corresponding corrected form, thereby expanding the candidate search space.

[0020] Finally, the original candidate transcription results are merged with the newly generated candidate transcription results to form a preliminary enhancement candidate set. The preliminary enhancement candidate set is then deduplicated, retaining only one instance of each identical candidate, resulting in the final deduplicated enhancement candidate set.

[0021] By generating new candidate expanded search spaces, the problem of the optimal result not appearing in the 1-best search space due to insufficient training data in low-resource language models is addressed.

[0022] Step S4: Specialize in large language model inference optimization.

[0023] First, based on the general corpus of the target low-resource language and the automatic speech recognition error correction corpus, the basic large language model is sequentially pre-trained with low-resource language customization and fine-tuned with automatic speech recognition error correction task-specific instructions to obtain a specialized large language model that is adapted to low-resource languages ​​and has error correction capabilities.

[0024] Then, the regularized enhancement candidate set and the corresponding normalized confidence scores are structured and encoded to form structured prompt text, which is then input into the specialized large language model. Under preset output constraints, the optimal candidate result is selected or local error correction is performed based on the bias correction library, without introducing words outside the enhancement candidate set, and the final transcription result is output.

[0025] By employing methods such as vocabulary constraints and length constraints, we address the technical shortcomings of large language models in generating unfounded content in low-resource language scenarios.

[0026] Step S5: Output the results.

[0027] The optimal transcription result obtained in step S4 is used as the final low-resource language speech recognition output.

[0028] To implement the above method, this invention further proposes a speech recognition and post-processing system for low-resource languages, including a candidate result acquisition module, a deviation correction library construction module, a candidate result enhancement module, a language model inference optimization module, and a result output module.

[0029] The candidate result acquisition module is responsible for speech recognition and candidate result acquisition. Specifically, this module includes a speech preprocessing unit, a feature extraction unit, and an ASR decoding unit.

[0030] The deviation correction library construction module is responsible for building the deviation correction library. Specifically, this module includes a data transcription unit, a text alignment unit, an error statistics unit, and a library storage unit.

[0031] The candidate result enhancement module is responsible for enhancing the candidate results. Specifically, this module includes a word-level segmentation unit, an approximate matching unit, a new candidate generation unit, and a deduplication optimization unit.

[0032] The language model inference optimization module is responsible for optimizing the inference of specialized large language models. Specifically, this module includes a model training and fine-tuning unit, a prompting construction unit, and an inference decision-making unit.

[0033] The output module is responsible for using the best transcription result as the final low-resource language speech recognition output. Specifically, this module includes a visualization unit, a file storage unit, and an API interface unit.

[0034] The relationships between the above modules are as follows:

[0035] The output of the candidate result acquisition module is connected to the input of the bias correction library construction module. The output of the bias correction library construction module is connected to the input of the candidate result enhancement module. The output of the candidate result enhancement module is connected to the input of the language model inference optimization module. The output of the language model inference optimization module is connected to the input of the result output module.

[0036] The modules communicate with each other through a unified data interaction interface, and the transmitted data is encapsulated in a structured data format. The structured data format includes at least candidate text content, candidate weights or confidence levels, error mode identifiers, and contextual information to ensure the consistency and scalability of the overall system process.

[0037] Beneficial effects

[0038] Compared with the prior art, the present invention has the following beneficial effects:

[0039] 1. By obtaining 1-best or N-best candidate results and combining them with subsequent candidate enhancement and language model inference optimization mechanisms, this invention can effectively alleviate the recognition instability problem caused by insufficient training data in low-resource languages; by expanding the candidate search space and selecting the best candidate from multiple candidates, it can significantly reduce the word-level misidentification rate and improve the overall transcription accuracy.

[0040] 2. This invention constructs a deviation correction library based on statistical analysis of real labeled data to specifically characterize high-frequency systematic error patterns of speech recognition models in low-resource language scenarios, rather than relying on general language rules or manual experience rules for correction. This invention only expands and corrects candidates for high-frequency, stable error patterns, effectively avoiding miscorrection problems caused by accidental errors and improving the reliability and stability of post-processing.

[0041] 3. In the candidate result enhancement stage, this invention does not directly perform deterministic replacement of the recognition result, but expands it by "generating new candidates" and constrains it through approximate matching and deduplication mechanisms. This technique preserves the original recognition result while introducing potential correct expressions, increasing the probability of correct results being selected and reducing the risk of false positives without destroying the original information.

[0042] 4. This invention fully leverages the language modeling advantages of specialized large language models. By performing specialized pre-training and instruction fine-tuning on the target low-resource language, this invention enables the large language model to possess stronger grammatical, semantic, and contextual understanding capabilities of the target language. During the inference phase, structured prompts and strict output constraints guide the large language model to select or locally modify only within the enhanced candidate set, utilizing the inference and discrimination capabilities of the large language model while avoiding the problem of generating uncontrolled content. Unified modeling and correction of model misidentification phenomena improve the system's generalization ability in real-world application environments.

[0043] 5. The overall process of this invention is scalable and highly portable. The method of this invention has a clear structure, and the various modules of the system interact through a unified data interface. It allows for flexible replacement of different speech recognition models or language models, and continuous improvement of system performance through updates to the deviation correction library. For different low-resource languages, only the corresponding deviation correction library needs to be rebuilt and the language model adapted for training, enabling rapid application migration and demonstrating good engineering practical value. Attached Figure Description

[0044] Figure 1 This is a flowchart illustrating the implementation of the method described in an embodiment of the present invention.

[0045] Figure 2 This is a schematic diagram of the structural composition of the system described in an embodiment of the present invention. Detailed Implementation

[0046] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0047] Example

[0048] like Figure 1 As shown, a speech recognition and post-processing method for low-resource languages ​​is presented. This method can be directly transferred to other low-resource language scenarios, requiring only the replacement of the corresponding language corpus and model adaptation parameters.

[0049] This embodiment focuses on low-resource languages ​​and details the complete implementation process of the method. This method is adaptable to various low-resource language scenarios, requiring only the replacement of the corresponding language corpus and model adaptation parameters. It uses an end-to-end speech recognition model as the core recognition carrier, is compatible with mainstream models of the same architecture, and ensures the method's universality and engineering feasibility.

[0050] This method includes the following steps:

[0051] Step S1: Speech recognition and candidate result acquisition.

[0052] Standardized preprocessing and automatic speech recognition (ASR) are performed on the input low-resource language speech signal to obtain 1-best or N-best candidate transcription results and corresponding confidence scores, providing basic data for subsequent post-processing.

[0053] Step S2: Construct a deviation correction library.

[0054] Based on a low-resource language annotation dataset, this study compares model-transcribed text with manually annotated text to statistically analyze high-frequency systematic error patterns in the model, quantifies the frequency of errors, and establishes a bias correction library containing erroneous text fragments, standard correction formats, and frequency weights to achieve accurate characterization of error patterns.

[0055] Step S3: Enhance candidate results.

[0056] The candidate transcription results obtained in step S1 are subjected to word-level segmentation and feature matching, and approximate matching is performed with high-frequency error patterns in the bias correction library to generate potentially correct candidate texts. Redundant terms are eliminated through deduplication optimization to construct a high-quality enhanced candidate set and expand the search space of the recognition results.

[0057] Step S4: Specialize in large language model adaptation training and structured restricted reasoning.

[0058] An adaptive training approach is adopted for the basic large language model, combining customized pre-training for low-resource languages ​​with fine-tuning based on ASR error correction task-specific instructions. This enhances the model's semantic understanding, syntactic modeling, and accurate ASR error correction capabilities for the target low-resource language. Based on the rule-based enhanced candidate set and corresponding normalized confidence scores, structured prompt text is constructed. Structured constrained reasoning is performed under multiple inference constraints, including vocabulary constraints, length constraints, and error correction constraints, to achieve optimal candidate selection or local error correction consistent with the bias correction library. Finally, the optimal transcription result is output, which is accurate, reliable, and context-coherent.

[0059] Step S5: Output the results.

[0060] The optimal transcription results are output in the form of visualization, file storage, and interface calls to adapt to the needs of different application scenarios and complete the closed loop of low-resource language speech recognition and post-processing.

[0061] Considering the common problems of scarce labeled corpora, complex semantic rules, and diverse accents in low-resource languages, traditional speech recognition methods rely on massive amounts of labeled data to train models. This leads to overfitting and high error rates in low-resource scenarios. Existing post-processing techniques mostly use general language models for correction, failing to specifically optimize for the systematic biases of low-resource languages ​​(such as misjudgment of grammatical structures, confusion of high-frequency words, and errors in replacing homophones). Furthermore, they lack candidate result expansion mechanisms, making it difficult to cover potentially correct expressions, resulting in limited post-processing effectiveness and poor generalization ability. To address the shortcomings of existing technologies, this method mainly considers the following three core points:

[0062] 1. How to accurately capture systematic errors in a model and avoid interference from random errors, given limited labeled data.

[0063] 2. How to expand the search space through a controllable candidate enhancement mechanism while reducing the risk of miscorrection.

[0064] 3. How to leverage the semantic modeling capabilities of large language models to improve contextual coherence while constraining the output range.

[0065] To address the aforementioned core pain points, this embodiment achieves efficient and reliable post-processing optimization through the construction of a bias correction library, hierarchical candidate enhancement, and a specialized large language model cascade architecture. It does not require the reconstruction of the existing ASR model and has strong engineering feasibility.

[0066] Step S1 specifically includes three core stages: speech signal preprocessing, speech feature extraction, and candidate result decoding and output. The entire process relies on an end-to-end ASR model, requiring no modification to the model's network structure. Existing model weights and inference frameworks can be directly reused, significantly reducing development costs and deployment difficulty, and adapting to computational constraints in low-resource scenarios.

[0067] Specifically, standardization is performed in the speech signal preprocessing stage. First, the input low-resource speech signal is uniformly converted into mono audio with a sampling rate of 16kHz and a bit depth of 16bit, consistent with the audio parameters during the ASR model training stage, avoiding feature distortion caused by sampling rate mismatch. A speech endpoint detection algorithm based on energy threshold and zero-crossing rate dual criteria is adopted. By traversing the audio through a sliding window, silence segments and noise fragments are removed, and effective speech intervals are retained. Subsequently, spectral subtraction is used for noise reduction. The audio power spectrum is extracted based on short-time Fourier transform (STFT), the noise spectrum is estimated and subtracted from the original power spectrum, and the denoised speech signal is reconstructed, improving the signal-to-noise ratio of the input speech and ensuring the accuracy of feature extraction and decoding.

[0068] The speech feature extraction stage supports multiple feature adaptation schemes, which can be flexibly selected according to the input requirements of the target ASR model. This embodiment adopts an end-to-end waveform input model, which does not require additional feature extraction and directly uses the original waveform data with a sampling rate of 16kHz and a bit depth of 16bit as the model input. All feature schemes need to be normalized (mean of 0, variance of 1) to eliminate the differences in feature distribution caused by different speakers, devices and scenarios. In the candidate result decoding output stage, the ASR model uses CTC as the training objective function, and the decoding stage uses a beam search algorithm to generate 10-best candidate results.

[0069] For step S2, a deviation correction library is constructed through data statistics, error comparison, and threshold screening. The core is the ability to accurately capture high-frequency systematic errors in ASR models on low-resource languages, ensuring the data in the library has generalization value, and that the specific execution process is reproducible and quantifiable. First, a low-resource language annotation dataset is selected, covering typical pronunciation scenarios such as daily conversations, public broadcasts, and professional settings, ensuring the diversity of data distribution and avoiding statistical bias in error patterns.

[0070] Specifically, using the same ASR model and parameters as in step S1, the speech samples in the labeled dataset are transcribed one by one to obtain the model's predicted text. The predicted text is then aligned sentence by sentence with the manually labeled standard text. The alignment uses the minimum edit distance algorithm to calculate the character-level edit distance between the predicted text and the standard text. Three types of error operations are broken down: insertion (I), deletion (D), and replacement (S). The specific location of each error, the erroneous text segment (1-5 words in length), and the corresponding standard corrected text segment are recorded. The frequency of each error pattern is also counted. For example, for the "similar word replacement" error, the erroneous segment "abc", the corrected segment "abd", and their corresponding occurrence counts are recorded; for the "omitted grammatical suffix" error, the erroneous segment "def", the corrected segment "defx", and their occurrence counts are recorded.

[0071] In the error pattern filtering stage, an error frequency threshold is set, retaining only error patterns with a frequency higher than the threshold and eliminating random errors. The final deviation correction library is stored in a key-value pair structure, where the key is the hash value of the error text fragment, and the value contains the error text, the correction text, and the frequency of occurrence. The data is stored in JSON files, supporting efficient fuzzy matching and query operations, providing data support for the approximate matching in step S3.

[0072] For step S3, candidate result enhancement is achieved through four hierarchical operations: word-level segmentation, approximate matching, new candidate generation, and deduplication optimization. This not only preserves the effective information of the original recognition results but also introduces potentially correct expressions. At the same time, strict deduplication avoids candidate redundancy, ensuring a balance between the quality and efficiency of the enhancement set.

[0073] Specifically, the candidate transcription results obtained in step S1 are first segmented at the word level. The appropriate word segmentation strategy is selected according to the writing characteristics of the target language: for language types where words are separated by spaces, spaces combined with punctuation marks are used directly as the segmentation basis to quickly split independent word units; for language types without explicit word separators, a word segmentation algorithm based on dictionary matching and bidirectional maximum positive matching is used; for words outside the dictionary (such as personal names and place names), character boundaries are used to segment them into single-character or double-character units to ensure segmentation accuracy.

[0074] In the approximate matching stage, a similarity determination mechanism based on Character Error Rate (CER) is adopted. The segmented text fragments are compared one by one with erroneous text fragments in the deviation correction library. The similarity between the two text fragments is measured by calculating the CER. At the character level, the text fragment to be matched is converted into a reference text fragment. The minimum number of editing operations required during the conversion is calculated, and the ratio of this number of editing operations to the total number of characters in the reference text fragment is used as the CER. Editing operations include character insertion, character deletion, and character replacement. In the specific implementation, the text fragment to be matched and the reference text fragment are first aligned at the character level. Then, the number of character replacements, character deletions, and character insertions required to make them consistent are calculated, and the total number of these three types of operations is accumulated. The total number of editing operations is then divided by the total number of characters in the reference text fragment to obtain the corresponding CER. Differentiated CER thresholds are set based on the text fragment length: when the text fragment length is less than or equal to 3 characters, the CER threshold is set to 60%; when the text fragment length is greater than 3 characters, the CER threshold is set to 40%. When the calculated character error rate is less than or equal to the corresponding threshold, the current text fragment is determined to have successfully matched the erroneous text fragment in the deviation correction library. For successfully matched text fragments, the original candidate text is not directly replaced. Instead, a new candidate text is generated based on the corrected text corresponding to the erroneous text fragment in the deviation correction library. That is, while keeping the original candidate text unchanged, a new candidate text is constructed after replacing the matched text fragment with the corrected text, forming multiple candidate results for subsequent processing. If the original candidate text has confidence data, the confidence of the candidate text is updated simultaneously. The update formula is: New confidence = Original confidence × (1 + Error occurrence frequency / Total error frequency), where the total error frequency is the sum of the occurrence frequencies of all error patterns in the deviation correction library. This formula strengthens the confidence after correcting high-frequency errors, providing a weighting basis for subsequent inference. If the candidate text has no completely matching error patterns, the text and confidence remain unchanged to ensure the controllability of optimization and avoid over-correction.

[0075] During the deduplication optimization phase, for several identical candidate options, only one is retained.

[0076] Step S4 includes specialized large language model adaptation training and structured constrained inference. Customized pre-training and fine-tuning strategies are designed for the characteristics of low-resource language corpora to enhance the model's adaptability to the target low-resource language. At the same time, a strict inference constraint mechanism is used to ensure the reliability of the output results.

[0077] In this embodiment, the adaptation training phase of the specialized large language model uses an open-source basic large language model with 7B parameters as the initial model. The basic large language model can be an LLaMA series model, a Qwen series model, or other generative language models with the same or similar structure. The training data adopts a hybrid scheme of low-resource language general corpus and ASR error correction corpus. The scale of the low-resource language general corpus is no less than 5 million texts, covering multiple application fields such as news, dialogue, and literature, which is used to improve the model's semantic understanding and grammatical modeling capabilities for the target low-resource language. The ASR error correction corpus adopts a candidate set annotation form that matches the fixed format of instruction fine-tuning, with a corpus size of no less than 100,000 texts. The aforementioned ASR error correction corpus comes from a manually constructed enhancement candidate set, and the error patterns in the candidate set are all set based on the high-frequency error patterns in the deviation correction library constructed in step S2. This is used to allow the model to learn the core logic of selecting the optimal result or locally correcting errors from the candidate set, thereby strengthening the model's ability to accurately correct errors in automatic speech recognition. Before model training, word segmentation is performed on various corpora of low-resource languages. The segmentation method can utilize the built-in segmenter of the basic large language model or a language-specific segmenter trained on the target low-resource language corpus. This ensures that the segmentation results closely match the writing characteristics and semantic expression habits of the target low-resource language. The model training process is divided into a low-resource language-customized pre-training stage and an ASR error correction task-specific instruction fine-tuning stage. This allows the basic large language model to deeply adapt to low-resource language scenarios and possess accurate ASR error correction capabilities, ultimately resulting in a low-resource language-specialized large language model.

[0078] In the pre-training phase for low-resource languages, the core objective is to train an autoregressive language model. Addressing the scarcity of low-resource language corpora and the dispersion of effective semantic and grammatical features, customized strategies such as refined preparation and fusion of multiple corpora, targeted mask generation, and lightweight training optimization are employed. These strategies enable the model to deeply learn the basic semantic and grammatical features of the target low-resource language, fully leveraging the feature value of various corpora. During pre-training, the mixed pre-training data is randomly divided into blocks for mask prediction training. A dynamic masking strategy is used to mask the words in the text, with the masking ratio controlled between 15% and 20%. 80% of the masked positions are replaced with model-specific mask symbols, 10% are replaced with random words, and 10% remain unchanged. The model then predicts the true words at the masked positions based on contextual information. Through this self-supervised training method, the model fully learns the word associations, semantic context, and sequence generation rules of the target language from the mixed data. Meanwhile, this stage abandons the common random masking method and adopts a low-resource language feature-oriented targeted masking strategy. It extracts the core vocabulary set and high-frequency error character fragments of the target low-resource language from the pre-training data. The core vocabulary set includes high-frequency nouns, verbs, grammatical particles, etc. of the target language. The high-frequency error character fragments are obtained from the feature enhancement data. A high masking probability of 60%-80% is set for the word units of the aforementioned core vocabulary set and the high-frequency error character fragments, and a low masking probability of 10%-20% is set for ordinary word units. After masking, the masked word units are predicted by context information, which enhances the model's ability to model the core vocabulary of low-resource language and the features related to high-frequency errors in ASR. In the autoregressive generation stage, the core vocabulary of the target language is subject to a forced priority generation constraint to ensure that the model masters the basic vocabulary association and sequence expression rules of the target language. To address the limited computing power issue commonly encountered in low-resource language development scenarios, this stage employs a lightweight training scheme combining low-rank adaptation and incremental parameter training. Low-rank fine-tuning is performed only on the attention layer and feature fusion layer of the basic large language model, freezing the remaining core parameters and reducing the number of training parameters to less than 5% of the original model, significantly reducing computing power consumption. This stage involves at least one training epoch, with an initial learning rate of 1e-5 to 5e-5. The batch size is flexibly configured based on actual computing resources, and random shuffling of data within each batch is used to prevent the model from learning sequential features. To ensure training stability, this stage introduces a learning rate preheating and cosine annealing decay strategy. The preheating phase accounts for 10% of the total training steps, during which the learning rate linearly increases from 1 / 10 of its initial value to its initial value. After preheating, the learning rate gradually decays according to a cosine function to prevent gradient explosion in the early stages of model training. Simultaneously, weight decay and gradient clipping strategies are introduced, with a weight decay coefficient set to 0.01 and a gradient norm threshold set to 1.0, effectively suppressing overfitting.In this stage, the perplexity of a low-resource language general corpus is used as the training effect verification index. After each training round, the effect is verified once. When the perplexity tends to stabilize and there is no significant decrease in three consecutive verification steps, the pre-training stage is terminated, and the pre-trained model weights are saved as the base model for subsequent ASR error correction task-specific instruction fine-tuning.

[0079] The ASR error correction task-specific instruction fine-tuning phase focuses on the accurate correction of ASR errors in low-resource languages. Based on a pre-trained base model, and combined with a regularized enhanced candidate set and high-frequency error patterns in low-resource language ASR, a dedicated instruction data format and refined training strategy are designed. This enables the model to master the core ability to select the optimal result from the candidate set and locally correct high-frequency errors in the candidate text. Simultaneously, constraints on parameters and training logic standardize the model's output behavior, preventing the model from generating data without a basis. This phase is task-oriented, guiding the model to learn to select the optimal transcription result or perform local corrections on the candidate text that conform to the rules of the bias correction library based on the regularized candidate set, normalized confidence, and task instructions. This ensures that the output conforms to the syntax and semantics of low-resource languages, contains no extra redundant information, and is compatible with the input requirements of subsequent structured inference. Based on the high-frequency error patterns of low-resource language ASR statistically analyzed in the deviation correction corpus of step S2, and combined with the regularized enhanced candidate set, dedicated instruction data is constructed. The instruction data adopts a fixed format: "Candidate set: [Candidate 1, confidence: X1; Candidate 2, confidence: X2; ... Candidate N, confidence (if any): XN]; Task: Select the optimal result from the above candidate set, or perform local correction on the candidate text (correcting only the erroneous segments matching the deviation correction corpus, without introducing words outside the candidate set), the output must conform to the low-resource language's syntax, semantics, and contextual coherence, returning only the final transcribed text without additional explanations." The instruction data includes the regularized candidate set, the normalized confidence score corresponding to each candidate, and a clear task instruction. The total size of the instruction data is consistent with the ASR error correction corpus, not less than 100,000 entries. The initial learning rate in this stage is further reduced compared to the continued pre-training stage, set to 1e-6~3e-6, ensuring fine-tuning of parameters based on the pre-trained model and avoiding damage to the basic low-resource language features already learned by the model. This stage continues the lightweight training approach of low-rank adaptation. The fine-tuning layers remain consistent with the pre-training stage, and the batch size is the same as or appropriately reduced compared to the pre-training stage. The learning rate strategy employs cosine annealing decay, with no learning rate warm-up process. Stability strategies such as weight decay and gradient pruning remain consistent with the pre-training stage to ensure training stability. This stage has at least one training epoch, which can be increased appropriately based on model convergence to ensure the model fully learns the inference logic of the ASR error correction task. A low-resource language ASR error correction validation set is constructed for training effect verification. The validation set contains at least 10,000 labeled, regularized candidate sets matched with the optimal results. Error correction accuracy and candidate selection accuracy are used as core validation metrics. Performance verification is performed after each training epoch. When both metrics stabilize and show no significant improvement after three consecutive validation steps, the instruction fine-tuning stage is terminated, and the final low-resource language-specialized large language model weights are saved.

[0080] In the structured constrained inference stage, based on the regularized enhanced candidate set, structured prompt text is designed. Combined with multiple strict output constraint mechanisms, this guides the specialized large language model to make inference decisions within a limited scope, ultimately outputting the optimal transcription result. The regularized enhanced candidate set and the normalized confidence scores of each candidate are structured and encoded to form a fixed-format structured prompt text. The prompt text clearly indicates the target low-resource language name, the complete candidate set, the confidence scores of each candidate, and the specific inference task instructions. The inference task instructions explicitly require the model to select the optimal result from the candidate set or to locally correct only the high-frequency error segments that match the target low-resource language deviation correction library. No words outside the candidate set may be introduced. The output result must conform to the grammatical semantics and contextual coherence of the target low-resource language, and only the final transcribed text is returned without any additional explanations. The structured prompt text is input into a low-resource language-specialized large language model. Multiple output constraint mechanisms are set to ensure the model's reasoning behavior is controllable. The first is a vocabulary constraint, implemented through model vocabulary filtering, prohibiting the model from generating any words outside the enhanced candidate set, allowing only local corrections to text fragments within the candidate set. The second is a length constraint, setting the maximum generated length to 1.2 times the average length of the input candidate text to avoid generating excessively long and redundant content. The third is an error correction constraint, bound to the deviation correction library constructed in step S2. Through model reasoning logic constraints, the model is limited to correcting only high-frequency error patterns existing in the library, and the correction method must be consistent with the correction form in the library. After the model completes inference and outputs preliminary results, the validity of the results is verified. If the preliminary result is one of the enhancement candidate sets, the result is directly taken as the optimal transcription result. If the preliminary result is a local correction result of the candidate text, it is verified whether the corrected fragment matches the high-frequency error pattern in the deviation correction library and whether the correction form conforms to the correction rules in the library. If the verification passes, the correction result is taken as the optimal transcription result. If the verification fails, the candidate with the highest confidence in the enhancement candidate set is selected as the optimal transcription result.

[0081] For step S5, a multi-format output interface is designed to enable flexible application of the results. The visualization interface supports real-time display of the optimal transcribed text on the terminal interface, simultaneously annotating the confidence level (retaining two decimal places) and correction records (e.g., "Corrected error segment: XXX→XXX"), facilitating intuitive viewing and verification by users. The file storage interface supports saving the transcribed text in both TXT and JSON formats. TXT format stores only plain text content, while JSON format includes complete information such as text content, confidence level, recognition time, and correction records. The storage path can be customized by the user, and batch storage and batch export are supported. The API interface adopts a RESTful architecture, with POST requests and JSON responses. Return parameters include the text result, confidence level, and status code, supporting third-party application calls, adapting to distributed deployment and integration requirements, and completing the entire closed-loop process.

[0082] like Figure 2 The diagram shown is a structural composition of a speech recognition and post-processing system for low-resource languages ​​proposed in this embodiment. The system can execute the methods described in the above embodiments and can be configured in terminals, servers or edge devices to adapt to different deployment scenarios.

[0083] This system includes a candidate result acquisition module, a bias correction library construction module, a candidate result enhancement module, a language model inference optimization module, and a result output module. All modules communicate through standardized interfaces, supporting modular deployment and expansion.

[0084] The candidate result acquisition module, which performs step S1, includes a speech preprocessing unit, a feature extraction unit, and an ASR decoding unit. The speech preprocessing unit performs sampling rate conversion, endpoint detection, and noise reduction, outputting a standardized speech signal. The feature extraction unit performs feature extraction and normalization, outputting a frame-level feature vector. The ASR decoding unit loads the ASR model, decodes and generates 1-best or N-best candidate results and their confidence scores. The module's input is a low-resource language speech signal, and its output is a list of candidate transcription results and their corresponding confidence scores.

[0085] The deviation correction library construction module is used to execute step S2 and includes a data transcription unit, a text alignment unit, an error statistics unit, and a library storage unit. The data transcription unit calls the ASR model from the candidate result acquisition module to perform batch transcription of the labeled dataset. The text alignment unit aligns the transcribed text with the standard text and breaks down error types. The error statistics unit counts the frequency of error patterns and filters high-frequency errors by threshold. The library storage unit uses JSON to store the deviation correction library, supporting efficient querying and updating. The module's data input is a low-resource language-annotated speech-text pair dataset, and the output is a deviation correction library file.

[0086] The candidate result enhancement module is used to execute step S3, and includes a word-level segmentation unit, an approximate matching unit, a new candidate generation unit, and a deduplication optimization unit. Specifically, the word-level segmentation unit performs word segmentation of the candidate text; the approximate matching unit corrects error patterns in the matching library based on the CER index; the new candidate generation unit generates potentially correct candidates based on the matching results and adjusts the confidence level; and the deduplication optimization unit removes redundant candidates. The module's input data consists of prompt words and a list of candidate transcription results, and its output is an enhanced candidate set.

[0087] The language model inference optimization module executes step S4 and includes a model training unit, a prompting construction unit, and an inference decision unit. The model training unit performs continued pre-training and instruction fine-tuning of the specialized large language model, outputting an adapted model. The prompting construction unit generates structured prompt text. The inference decision unit calls the specialized model to perform restricted inference, verifies, and outputs the optimal transcribed text. The module's input is the structured prompt text, and its output is the optimal transcribed text.

[0088] The results output module is used to execute step S5 and includes a visualization unit, a file storage unit, and an API interface unit. The visualization unit displays the optimal text, confidence level, and correction records in real time. The file storage unit supports TXT and JSON format storage and batch export. The API interface unit provides a RESTful interface, supporting calls from third-party applications. The module's input is the optimal transcribed text, and the output is visualized text, a stored file, or API response data.

[0089] In this system, the modules interact with each other via JSON format. The interaction fields include core information such as text content, confidence level, error mode identifier, and context label.

[0090] Furthermore, embodiments of the present invention also provide a non-volatile computer storage medium storing computer-executable instructions that can execute the speech recognition and post-processing methods for low-resource languages ​​in any of the above method embodiments.

[0091] Furthermore, as one implementation, the non-volatile computer storage medium of the present invention stores computer-executable instructions, which are configured to: perform standardized preprocessing on the input low-resource language speech signal, extract features and decode them through an ASR model to obtain candidate transcription results and confidence levels; based on the low-resource language labeled dataset, statistically analyze the high-frequency systematic error patterns of the ASR model and construct a bias correction library; perform word-level segmentation and approximate matching on the candidate results, generate new candidates and optimize for deduplication, and construct an enhanced candidate set; perform low-resource language adaptation training on the basic large language model, perform rule-based optimization and restricted inference based on the enhanced candidate set, and determine the optimal transcribed text; and output the optimal transcription result through a multi-morphological interface.

[0092] As a non-volatile computer-readable storage medium, it can be used to store non-volatile software programs, non-volatile computer-executable programs, and modules, such as the program instructions / modules corresponding to the methods in the embodiments of this invention. One or more program instructions are stored in the non-volatile computer-readable storage medium, and when executed by a processor, they perform the speech recognition and post-processing methods for low-resource languages ​​in any of the above method embodiments, adapting to the operating environment of various computing devices.

[0093] The above-described product can perform the methods provided in the embodiments of this application, and has the corresponding functional modules and beneficial effects for performing the methods. Technical details not described in detail in this embodiment can be found in the methods provided in the embodiments of this application.

[0094] Furthermore, embodiments of the present invention also provide an electronic device, including at least one processor and a memory communicatively connected to the at least one processor, wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform the steps of the speech recognition and post-processing method for low-resource languages ​​according to any embodiment of the present invention.

[0095] The electronic devices in the embodiments of this application exist in various forms, including but not limited to: (1) Server: a device that provides computing services, which typically includes components such as processor, memory, and network interface, and can be deployed in the cloud or locally, supporting batch speech recognition processing; (2) Edge computing device: a device with localized data processing capabilities, such as edge gateways and smart terminals, which are adapted to low latency and high privacy protection scenarios; (3) Portable computing device: such as laptops and tablets, which support lightweight deployment and on-site recognition; (4) Other electronic devices with data processing functions.

[0096] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0097] The above embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A speech recognition and post-processing method for low-resource languages, characterized in that, Includes the following steps: Step S1: Speech recognition and candidate result acquisition; Standardized preprocessing and automatic speech recognition are performed on the input low-resource language speech signal to obtain 1-best or N-best candidate transcription results and corresponding confidence scores, providing basic data for subsequent post-processing; Step S2: Construct a deviation correction library; First, we collect labeled low-resource language speech-text pairs; the data includes clear speech signals and corresponding accurate manually labeled text. Then, using the same speech recognition model and model parameters as in step S1, the labeled speech data is transcribed to obtain the model transcription result, so that the transcription result has consistent systematic bias characteristics with the output of the inference stage. Then, the model transcription results were aligned sentence by sentence with the corresponding manually annotated reference text. The differences between the two were calculated at the character level and word level. Three types of error operations, namely insertion, deletion and replacement, were identified. The error text, the corrected text and its position information in the sentence were recorded for each error operation. Based on the error operation records, statistical analysis is performed on error patterns; a word-level confusion statistics table is constructed with word-level errors as the statistical object; at the same time, an error occurrence frequency threshold is set, and only high-frequency error patterns that occur more frequently than the threshold are retained to eliminate random errors; Based on the high-frequency word-level error patterns obtained through screening, a deviation correction library is established. The deviation correction library includes erroneous text fragments, corresponding corrected text fragments, and error occurrence frequency information, which are used for approximate matching and candidate generation in the subsequent candidate result enhancement stage. Step S3: Enhance candidate results; Based on the candidate transcription results obtained in step S1, the candidate transcription results are segmented using words as the basic processing unit to obtain several candidate word-level text fragments. Approximate matching is performed between candidate word-level text fragments and word-level error patterns in the deviation correction library. During the matching process, a word-level difference not exceeding a preset CER threshold is allowed between the candidate text fragments and the error patterns. When the CER is less than the threshold, the match is considered successful. When a successfully matched word-level error pattern is detected, at least one new candidate transcription result is generated by replacing the error pattern with the corresponding corrected form, thus expanding the candidate search space. Finally, the original candidate transcription results are merged with the newly generated candidate transcription results to form a preliminary enhancement candidate set; the preliminary enhancement candidate set is then deduplicated, and only one of the identical candidates is retained, thus forming the final deduplicated enhancement candidate set; Step S4: Specialize in large language model inference optimization; First, based on the general corpus of the target low-resource language and the automatic speech recognition error correction corpus, the basic large language model is pre-trained in a low-resource language-customized manner and fine-tuned with instructions specific to the automatic speech recognition error correction task, so as to obtain a specialized large language model that is adapted to low-resource languages ​​and has error correction capabilities. Then, the regularized enhancement candidate set and the corresponding normalized confidence are structured and encoded to form a structured prompt text, which is then input into the specialized large language model. Under the preset output constraints, the optimal candidate result is selected or the local error is corrected based on the bias correction library. No words outside the enhancement candidate set are introduced, and the final transcription result is output. Step S5: Output the result; The optimal transcription result obtained in step S4 is used as the final low-resource language speech recognition output.

2. The speech recognition and post-processing method for low-resource languages ​​as described in claim 1, characterized in that, Step S1 includes three core steps: speech signal preprocessing, speech feature extraction, and candidate result decoding output, which are all completed using an end-to-end ASR model. In the speech signal preprocessing stage, standardized operations are performed. First, the input low-resource language speech signal is uniformly converted into mono audio with a sampling rate of 16kHz and a bit depth of 16bit, which is consistent with the audio parameters in the ASR model training stage. A speech endpoint detection algorithm based on energy threshold and zero-crossing rate is adopted. The audio is traversed by sliding window to remove silent segments and noise segments and retain the effective speech interval. Subsequently, spectral subtraction is used for noise reduction. The audio power spectrum is extracted based on short-time Fourier transform, the noise spectrum is estimated and subtracted from the original power spectrum, and the noise-reduced speech signal is reconstructed to improve the signal-to-noise ratio of the input speech. The feature adaptation scheme supported in the speech feature extraction stage is selected according to the input requirements of the target ASR model. The feature adaptation scheme eliminates the feature distribution differences caused by different speakers, devices and scenarios through feature normalization. In the candidate result decoding output stage, the ASR model uses CTC as the training objective function and the beam search algorithm is used in the decoding stage to generate 10-best candidate results.

3. The speech recognition and post-processing method for low-resource languages ​​as described in claim 1, characterized in that, In step S2, the alignment uses the minimum edit distance algorithm to calculate the character-level edit distance between the predicted text and the standard text, decompose the three types of error operations: insertion (I), deletion (D), and replacement (S), record the specific location of each error, the erroneous text fragment, and the corresponding standard corrected text fragment, and at the same time, count the frequency of occurrence of each type of error pattern.

4. A speech recognition and post-processing system for low-resource languages ​​that implements the method of claim 1, characterized in that, It includes a candidate result acquisition module, a bias correction library construction module, a candidate result enhancement module, a language model inference optimization module, and a result output module; The candidate result acquisition module is responsible for speech recognition and candidate result acquisition; this module includes a speech preprocessing unit, a feature extraction unit, and an ASR decoding unit. The deviation correction library construction module is responsible for building the deviation correction library; this module includes a data transcription unit, a text alignment unit, an error statistics unit, and a library storage unit. The candidate result enhancement module is responsible for enhancing the candidate results; this module includes a word-level segmentation unit, an approximate matching unit, a new candidate generation unit, and a deduplication optimization unit. The language model inference optimization module is responsible for optimizing the inference of specialized large language models; this module includes a model training and fine-tuning unit, a prompting construction unit, and an inference decision-making unit. The output module is responsible for using the best transcription result as the final low-resource language speech recognition output; this module includes a visualization unit, a file storage unit, and an API interface unit. The relationships between the above modules are as follows: The output of the candidate result acquisition module is connected to the input of the bias correction library construction module; the output of the bias correction library construction module is connected to the input of the candidate result enhancement module; the output of the candidate result enhancement module is connected to the input of the language model inference optimization module; and the output of the language model inference optimization module is connected to the input of the result output module. The modules communicate with each other through a unified data interaction interface, and the transmitted data is encapsulated in a structured data format. The structured data format includes candidate text content, candidate weights or confidence levels, error mode identifiers, and context information to ensure the consistency and scalability of the overall system process.

5. A speech recognition and post-processing system for low-resource languages ​​as described in claim 4, characterized in that, The modules exchange data via JSON format.

6. A speech recognition and post-processing system for low-resource languages ​​as described in claim 4, characterized in that, In the deviation correction library construction module, the data transcription unit calls the ASR model of the candidate result acquisition module to perform batch transcription of the labeled dataset; the text alignment unit uses the minimum edit distance algorithm to align the transcribed text with the standard text and decompose the error types. The error statistics unit counts the frequency of error patterns and filters high-frequency errors according to a preset threshold. The library storage unit uses JSON format to store the deviation correction library, supporting efficient querying and updating.

7. A speech recognition and post-processing system for low-resource languages ​​as described in claim 4, characterized in that, In the language model inference optimization module: The model training and fine-tuning unit performs low-resource language customization for specialized large language models, continuing pre-training and ASR error correction task-specific instruction fine-tuning. The prompt building unit will perform structured encoding of the enhanced candidate set and the corresponding normalized confidence to generate structured prompt text in a fixed format; The reasoning decision unit calls a specialized large language model to perform restricted reasoning, verifies the validity of the output results, and outputs the optimal transcribed text.

8. A non-volatile computer storage medium, characterized in that, A computer-readable storage medium stores computer-executable instructions that, when executed by a processor, implement the speech recognition and post-processing method for low-resource languages ​​as described in any one of claims 1-3.

Images