A Mongolian speech recognition method for complex scenarios
By optimizing the Mongolian speech recognition model through multimodal feature fusion and transfer learning, and combining acoustic and visual features, the robustness and decoding efficiency of Mongolian in complex acoustic environments are solved, achieving efficient speech recognition and high recognition accuracy.
Patent Information
- Application Number
- CN202611589629.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2026-10-14
- Publication Date
- 2026-11-10
AI Technical Summary
Mongolian speech recognition suffers from insufficient feature robustness in complex acoustic environments, and its agglutinative nature leads to a bottleneck in decoding efficiency. Traditional methods struggle to adapt to noise interference, overlapping speech from multiple speakers, and insufficient training of language models under low-resource data conditions.
We employ multimodal feature fusion technology, combining acoustic and visual features, and use a dynamic weight allocation strategy to enhance the weight of visual features in noisy environments. We optimize the model using transfer learning and fine-tuning strategies, and construct a lightweight decoding network by combining stemming, case suffixes, and verb segmentation strategies other than verbs.
It improves speech recognition accuracy in noisy environments, reduces reliance on large-scale labeled data, enhances adaptability to Mongolian dialects and grammatical features, and solves decoding efficiency bottlenecks and recognition error problems.
Smart Images

Figure CN123072376A_ABST
Abstract
Description
Technical Field
[0001] This invention belongs to the field of speech recognition technology and relates to a Mongolian speech recognition method for complex scenarios. Background Technology
[0002] Speech recognition is a key technology for human-computer interaction, involving multiple disciplines such as acoustic analysis, linguistic modeling, signal processing, and machine learning. Its core objective is to convert speech signals into corresponding text information. Depending on the task requirements, speech recognition can be categorized into speaker recognition, keyword detection, and continuous speech recognition. Currently, this technology is widely used in smart homes, automotive systems, industrial automation, and medical diagnostics, significantly improving the efficiency of human-computer interaction and the level of scene intelligence.
[0003] Currently, speech recognition research mainly focuses on widely used languages such as English and Chinese, while research on languages like Mongolian, which have a smaller usage and population, is still in its early stages. As a representative of agglutinative languages, Mongolian is characterized by complex affix combinations and diverse word forms (e.g., hundreds of morphemes derived from single words), making traditional speech recognition technologies challenging to adapt. However, Mongolian has significant application value in scenarios such as minority education, public services in border regions (e.g., traffic guidance, medical consultation), and cross-border trade. Researching Mongolian speech recognition technology for complex environments can not only promote the informatization of ethnic minority areas but also provide technical reference for the recognition of other agglutinative languages such as Finnish and Turkish, offering valuable insights.
[0004] Acoustic feature extraction requires extracting key, discriminative parameters (such as spectrum and formants) from speech signals. Mainstream methods include Mel frequency cepstral coefficients (MFCC), linear predictive cepstral coefficients (LPCC), and Mel filter bank energy features (FBank). These features perform well in quiet environments, but in complex acoustic scenarios (such as background noise, device channel distortion, and reverberation interference), the spectral energy distribution is easily contaminated, leading to decreased feature robustness. Although compensation algorithms such as linear discriminant analysis (LDA) and feature space maximum likelihood linear regression (fMLLR) can improve feature stability, they lack specific optimization for the unique phonological characteristics of Mongolian, such as palatalized consonants and vowel harmonics, making it difficult to suppress the coupling interference between noise and speech components.
[0005] Traditional speech recognition systems employ a Gaussian Mixture Model-HMM framework, using a Gaussian Mixture Model (GMM) to model acoustic features and a Hidden Markov Model (HMM) to describe state transitions. With the development of deep learning, the DNN-HMM hybrid model has gradually become mainstream, utilizing deep neural networks (DNNs) to replace GMMs and enhance acoustic modeling capabilities. In terms of language models, N-gram models provide contextual constraints through statistical word sequence probabilities, while models based on recurrent neural networks (RNNs) can capture long-range dependencies. However, the generalization ability of these models in complex scenarios is still limited by the quality of acoustic features and decoding efficiency.
[0006] The speech recognition system constructs a decoding network (Weighted Finite-State Transducer, WFST) by integrating an acoustic model, a language model, and a pronunciation dictionary, and uses the Viterbi algorithm to search for the optimal path to output the recognition result. During optimization, the language model weights and long-word penalty coefficients need to be adjusted to balance the acoustic score and the language model probability. To address the complex word inflections in Mongolian, a stemming-affix decomposition strategy is often used to compress the vocabulary size, but this introduces ambiguity at morpheme boundaries, increasing the risk of decoding errors. Furthermore, interference from multiple candidate paths in noisy environments further exacerbates the expansion of the search space, making real-time performance difficult to guarantee. In implementing this embodiment of the invention, two key problems were found to exist in Mongolian speech recognition technology in complex scenarios:
[0007] The robustness limitations of traditional acoustic features in complex acoustic environments: Traditional acoustic features lose discriminative power under noise and reverberation interference, especially for easily confused phonemes in Mongolian (such as voiced / voiced consonants and long / short vowels). It is necessary to introduce multimodal information (such as lip movements and tongue positions in videos) to supplement pronunciation details and enhance phoneme discrimination accuracy in noisy environments through cross-modal feature fusion.
[0008] The characteristics of agglutinative languages lead to a bottleneck in decoding efficiency: the complexity of word inflections makes the decoding network much larger than that of conventional languages, and traditional pruning algorithms are prone to losing low-frequency word paths when compressing the search space. A lightweight dynamic decoding mechanism needs to be designed, combined with multimodal temporal alignment constraints (such as lip-movement synchronization) to reduce the expansion of invalid paths and achieve a balance between efficiency and accuracy. Summary of the Invention
[0009] To achieve the above objectives, this invention provides a Mongolian speech recognition method for complex scenarios, which solves the problems in the prior art such as acoustic feature distortion caused by noise interference, difficulty in separating overlapping speech from multiple speakers, insufficient model generalization caused by dialect accent differences, and insufficient training of language models under low-resource data.
[0010] The technical solution adopted in this invention is a Mongolian speech recognition method for complex scenarios, which consists of a dictionary construction stage, a multimodal feature extraction stage, a multimodal feature fusion stage, a transfer learning and fine-tuning stage, a decoding stage, and a post-processing stage. The specific steps are as follows:
[0011] Step S1: Dictionary construction stage; Segment the words in the Mongolian training text into stems (excluding verbs), case suffixes, and verb forms; Simultaneously, establish a whole-word pronunciation dictionary and a word-segmentation pronunciation dictionary based on stems (excluding verbs), case suffixes, and verb forms; Use the segmented training text to train the language model;
[0012] Step S2: Multimodal feature extraction stage; Deep noise reduction network is used to perform deep noise suppression on the input speech signal, auditory encoder is used to extract acoustic features, visual encoder is used to extract lip movement temporal features from synchronous video stream, and the extracted lip movement temporal features are temporally aligned with the acoustic features.
[0013] Step S3: Multimodal feature fusion stage; The acoustic-visual fusion model receives the noise-suppressed acoustic features and the temporal features of lip movement after temporal alignment, and constructs an acoustic-visual joint representation through a dynamic weight allocation strategy;
[0014] Step S4: Transfer learning and fine-tuning stage; First, the acoustic-visual fusion model is pre-trained using a multilingual general speech dataset. Then, the parameters of the low-level feature extraction network of the acoustic-visual fusion model are frozen, and the parameters of the classification layer and language adaptation module of the model are optimized on the Mongolian dataset.
[0015] Step S5: Decoding stage; a weighted finite state converter decoding network is constructed using an acoustic model, a language model, and a dedicated pronunciation dictionary. The optimal path is searched using the Viterbi algorithm, and the intermediate text is output in the form of stems other than verbs, case suffixes, and verbs. The path penalty is applied to low-frequency stems and low-frequency case suffixes by combining the probability distribution of the language model.
[0016] Step S6: Post-processing stage; Based on the matching rules of Mongolian positive stems and case suffixes, the case suffix errors in the decoded text are corrected and the corresponding Latin words are formed. At the same time, the recognized Latin words are segmented and punctuation marks are added. Finally, through the comparison relationship between Latin and Mongolian words, they are converted into actual Mongolian words, and then Mongolian sentences are formed.
[0017] Furthermore, the specific steps for segmenting words in the Mongolian training text into stems (excluding verbs), case suffixes, and verb forms are as follows:
[0018] First, the Mongolian text is converted into Latin form, and the stems of words other than verbs and case suffixes are separated by narrow, unbroken spaces.
[0019] The obtained whole-word pronunciation dictionary stores complete Mongolian words and their corresponding pronunciations for acoustic model training; the word-segmentation pronunciation dictionary contains segmented stems, case suffixes, verbs and their corresponding pronunciations, and all possible pronunciations of case suffixes, for the decoding stage.
[0020] Furthermore, the acoustic-visual fusion model adopts a cascaded structure. First, a bidirectional long short-term memory network is used to locally temporally encode the acoustic features and the temporal features of lip movements, respectively. Then, the output features of the bidirectional long short-term memory network are input into the Transformer encoder, and a multi-head self-attention mechanism is used to capture cross-modal long-distance dependencies. Finally, an acoustic-visual joint representation is generated according to a dynamic weight allocation strategy.
[0021] The specific implementation of the dynamic weight allocation strategy is as follows: calculate the residual noise intensity in the acoustic features. If the noise intensity still exceeds the preset threshold after noise reduction and suppression in step S2, increase the weight ratio of the lip movement temporal features. If the noise intensity does not exceed the preset threshold, maintain the default weight allocation or give priority to the use of acoustic features.
[0022] Furthermore, the classification layer computation maps the acoustic-visual joint representation to the posterior probabilities of phonemes, stems, case suffixes, and verbs; the language adaptation module introduces Mongolian linguistic constraints, including vowel harmony, palatalization rules, and case suffix co-occurrence probabilities, and corrects the classification layer output through a learnable constraint matrix.
[0023] Furthermore, the specific process for path penalty of low-frequency stems and low-frequency case suffixes is as follows: combining the probability distribution of the language model, the frequency of each stem and case suffix in the training corpus is counted. If the frequency is lower than a preset threshold, it is marked as a low-frequency unit; a penalty is applied to the path containing low-frequency stems or case suffixes, and the penalty intensity is negatively correlated with the frequency of the low-frequency unit; finally, the path with the highest total score is selected as the optimal recognition result.
[0024] The beneficial effects of this invention are as follows:
[0025] The system addresses the robustness issue of features in complex acoustic environments by introducing multimodal feature extraction and fusion techniques, combining acoustic and visual features (lip movements), and employing a dynamic weight allocation strategy. When noise intensity exceeds a threshold, the system prioritizes increasing the weight of visual features, effectively mitigating noise interference. Even when noise levels remain high after deep denoising, the system automatically increases the weight of visual features, prompting the model to focus more on the speaker's lip movements, significantly improving speech recognition accuracy in noisy environments.
[0026] This approach addresses the issue of sparse data for low-resource languages by employing transfer learning and fine-tuning strategies. It utilizes a large-scale, multilingual, general-purpose speech dataset to pre-train an acoustic-visual fusion model, freezes the underlying cross-lingual feature extraction network, and fine-tunes the top-level encoder and language adaptation module only on a low-resource Mongolian dataset. This design significantly reduces reliance on large-scale labeled Mongolian data while enhancing the model's adaptability to Mongolian dialects, accents, and grammatical features, resulting in a substantial reduction in training time and computational resource consumption.
[0027] This approach addresses the bottleneck in decoding efficiency caused by the complex inflectional changes in agglutinative languages: Through an innovative segmentation strategy of "stem other than verb + case suffix + verb", Mongolian words are divided into three recognition units. At the same time, a dual-dictionary strategy is adopted (the whole word pronunciation dictionary is used for training, and the segmented pronunciation dictionary is used for decoding), which balances the phoneme accuracy of the acoustic model training with the efficiency of the decoding stage, thus solving the problem of recognizing large-scale Mongolian words.
[0028] The problem of incorrect case suffix recognition and Mongolian text display has been solved: the grammatical rule correction mechanism in the post-processing stage finally generates Mongolian sentences that conform to grammatical norms, which solves the problem that traditional acoustic models and language models cannot completely distinguish similar case suffixes, and at the same time achieves the correct display of Mongolian text. Attached Figure Description
[0029] Figure 1 This is a flowchart of the Mongolian speech recognition method proposed in this invention.
[0030] Figure 2 This is a diagram illustrating the word formation method in Mongolian script.
[0031] Figure 3 This is an example diagram of the Mongolian text preprocessing process in this invention.
[0032] Figure 4 This is a comparison table of some contents of the two pronunciation dictionaries in this invention.
[0033] Figure 5 This is an example diagram of the post-processing stage in this invention.
[0034] Figure 6 This is a rule diagram for the rule correction of the ending suffix in this invention. Detailed Implementation
[0035] Mongolian, as a typical agglutinative language, relies heavily on the combination of roots and affixes in its word structure. For example... Figure 2As shown, the combination of a root word with a word-forming affix or inflectional suffix changes the semantics, while the combination with a final suffix primarily has grammatical meaning, and the final suffix always appears at the end of the word. Final suffixes include case suffixes for nouns, possessive suffixes, verbal suffixes, and adverbial suffixes. Verbal suffixes are considered final suffixes when functioning as the predicate of a main clause, while they are considered stem suffixes when used as nouns (especially when followed by a case suffix). Generally, the order of suffixes is word-forming suffixes first, inflectional suffixes second, and final suffixes last. By combining roots, word-forming suffixes, and inflectional suffixes to form a stem, and using the stem and final affix as the basis for Mongolian word formation, different stems and different final suffixes can form the vast majority of Mongolian words. Therefore, in a speech recognition system, word recognition can be transformed into the recognition of stems and final suffixes.
[0036] However, the recognition method based solely on stems and suffixes has some problems. First, Mongolian verbs exhibit vowel dropout and insertion during stem and suffix segmentation, resulting in low segmentation accuracy. Second, the pronunciation of verb stems and suffixes changes phonemically under different splicing conditions, posing a challenge to the construction of a pronunciation dictionary. In contrast, the pronunciation of case suffixes attached to stems other than verbs is relatively stable and does not affect the pronunciation of the stem. Therefore, this method separates the verb into a separate unit, using it together with stems and case suffixes other than verbs as recognition units, constructing a speech recognition system based on stems other than verbs, case suffixes, and the verb itself.
[0037] This embodiment provides a Mongolian speech recognition method for complex scenarios, such as... Figure 1 As shown, the process includes the dictionary construction stage, the multimodal feature extraction stage, the multimodal feature fusion stage, the transfer learning and fine-tuning stage, the decoding stage, and the post-processing stage, as detailed below:
[0038] Step S1: Dictionary construction stage; the words in the Mongolian training text are segmented into stems other than verbs, case suffixes and verb forms, and a word segmentation pronunciation dictionary is built based on stems other than verbs, case suffixes and verb forms, covering polyphonic variants in complex scenarios.
[0039] In written Mongolian, the suffix "-ge" is separated using special symbols, which can be easily used for segmentation when converted to Latin form. For example... Figure 3 As shown, the Mongolian text is specifically converted to Latin form. The stems of words other than verbs and case suffixes are separated using narrow, unbroken spaces. Since Mongolian case suffixes are written using narrow, unbroken spaces (one-third the width of a full-width character, slightly shorter than a regular space), the Latin form uses "-". The text corpus used to train the language model after conversion to Latin form can then be easily segmented into stems of words other than verbs and case suffixes based on the "-" letter.
[0040] Two pronunciation dictionaries are established: one is a traditional whole-word pronunciation dictionary, which stores complete Mongolian words and their corresponding pronunciations for acoustic model training; the other is a word-segmented pronunciation dictionary, which includes the segmented stems, case suffixes, verbs and their corresponding pronunciations, and all possible pronunciations of case suffixes, for the decoding stage.
[0041] The language model is trained using segmented training text, enabling it to effectively match non-verb stems, case suffixes, and verb pronunciation dictionaries during decoding. This ensures that the decoded results are presented as non-verb stems, case suffixes, and verbs. Non-verb stems and case suffixes can combine to form a large number of Mongolian words, and the total number of non-verb stems, case suffixes, and common verbs is within tens of thousands. This method addresses the data sparsity problem during language model training and the challenge of recognizing a large number of Mongolian words.
[0042] Figure 4 The comparison of portions of the two dictionaries is shown. It can be seen that in the whole-word pronunciation dictionary, the storage form of certain words remains unchanged. For example, the verbs “sagvjv” (sit) and “svrqv” (study), as well as the stem-only word “elqi” (ambassador), are stored in their complete form in both dictionaries. These words are represented consistently in both dictionaries, requiring no further splitting. In contrast, non-verb words consisting of stems and case suffixes are split into stems and case suffixes separately in the pronunciation dictionary based on stems other than verbs, case suffixes, and verbs. For example, the words “tarihi-ban” (tarihi-ban, head) and “tere-yi” (tere-yi, he / she / it) are stored in their complete form in the whole-word pronunciation dictionary, while in the word-segmentation pronunciation dictionary, they are split into the stems “tarihi” (head) and “tere” (tere), and the case suffixes “-ban” and “-yi”, stored separately. This splitting method allows the system to handle combinations of stems and case suffixes more flexibly, thereby improving the accuracy and efficiency of recognition.
[0043] This design allows the system to use a whole-word pronunciation dictionary during the training phase to ensure accurate phoneme pronunciation of training sentences, while using a word-segmentation pronunciation dictionary during the decoding phase. This reduces dictionary size, computational load, and the problem of recognizing large-scale Mongolian words. This dual-dictionary strategy not only improves the system's recognition capability but also significantly optimizes its performance.
[0044] Step S2: Multimodal feature extraction stage; A deep noise reduction network is used to perform deep noise suppression on the input speech signal to eliminate environmental interference. An auditory encoder is used to extract robust acoustic features (such as Mel frequency cepstral coefficients and their dynamic differences). A visual encoder is used to extract lip movement temporal features from the synchronous video stream at the same time, which enhances the information complementarity in noisy environments and ensures that the extracted lip movement temporal features are strictly temporally aligned with the acoustic features.
[0045] Step S3: Multimodal Feature Fusion Stage; Multimodal feature fusion is a key step in this system. This stage receives acoustic features after noise suppression and temporal features of lip movement after temporal alignment, and constructs an acoustic-visual joint representation through a dynamic weight allocation strategy.
[0046] The specific implementation of the dynamic weight allocation strategy is as follows: calculate the residual noise intensity in the acoustic features. If the noise intensity still exceeds the preset threshold after the noise reduction operation in step S2, increase the weight ratio of the lip movement temporal features to encourage the acoustic-visual fusion model to pay more attention to the speaker's lip movement information to avoid the influence of noise. If the noise intensity does not exceed the threshold, maintain the default weight allocation or give priority to the use of acoustic features.
[0047] An acoustic-visual fusion model is used to fuse acoustic features and lip movement temporal features in a temporal context. Temporal context features refer to the information of adjacent frames of acoustic features and lip movement temporal features within a temporal window. The acoustic-visual fusion model adopts a cascaded structure: first, a bidirectional long short-term memory network is used to perform local temporal encoding on acoustic features and lip movement temporal features respectively; then, the hidden state sequence output by the bidirectional long short-term memory network is input into a Transformer encoder, and a multi-head self-attention mechanism is used to capture long-distance dependencies across modalities, finally generating an acoustic-visual joint representation and outputting it to step S4.
[0048] Step S4: This stage implements a two-stage training strategy for the acoustic-visual fusion model constructed in Step S3. Pre-training stage: The acoustic-visual fusion model is pre-trained using a multilingual general speech dataset to learn cross-lingual acoustic-visual joint representation capabilities. Fine-tuning stage: The parameters of the low-level feature extraction network of the acoustic-visual fusion model are frozen to preserve cross-lingual capabilities, and the parameters of the model's classification layer and language adaptation module are optimized on a Mongolian low-resource dataset.
[0049] The classification layer maps the joint representations generated by the acoustic-visual fusion model to the posterior probabilities of phonemes, stems, case suffixes, and verbs. The language adaptation module introduces linguistic constraints specific to Mongolian, including vowel harmony, palatalization rules, and case suffix co-occurrence probabilities. It then uses a learnable constraint matrix to correct the classification layer output, adapting the model to Mongolian dialects and grammatical characteristics.
[0050] Step S5: The input for this stage is the phoneme posterior probability sequence output by the acoustic-visual fusion model after fine-tuning in Step S4. The specific decoding process is as follows: A weighted finite-state converter decoding network is constructed using the acoustic model, language model, and a dedicated pronunciation dictionary. The optimal path is searched using the Viterbi algorithm, and the intermediate text is output as stems (excluding verbs), case suffixes, and verb forms. Path penalties are applied to low-frequency stems and low-frequency case suffixes based on the probability distribution of the language model.
[0051] The specific process of the path penalty mechanism is as follows: combining the probability distribution of the language model, the frequency of each stem and case suffix in the training corpus is counted. If the frequency is lower than the preset threshold, it is marked as a low-frequency unit. A penalty is applied to the path containing low-frequency stems or case suffixes. The penalty intensity is negatively correlated with the frequency of low-frequency units. Finally, the path with the highest total score is selected as the optimal recognition result to optimize the decoding accuracy.
[0052] Step S6: Post-processing stage; such as Figure 5 As shown, based on the matching rules of Mongolian positive stems and case suffixes, the case suffix errors in the decoded text are corrected and the corresponding Latin words are formed. At the same time, the Conditional Random Field (CRF) model is used to segment the recognized Latin words and add punctuation marks. Finally, through the comparison relationship between Latin and Mongolian words, they are converted into actual Mongolian words, and then Mongolian sentences are formed.
[0053] In the actual decoding and post-processing process, the decoded error results exhibit certain patterns, primarily in the recognition of Mongolian case suffixes. To address these errors, this embodiment employs a matching rule based on Mongolian masculine stems and case suffixes. For example... Figure 6 As shown, for example, when determining the choice of case suffixes "-dv", "-du", "-tv", and "-tu", for masculine stems, if the stem ends in a consonant (excluding vowels or "n", "N", "l", and "m"), the case suffix "-tv" is selected; if the stem ends in a vowel or "n", the case suffix "-dv" is selected. For non-masculine stems, if the stem does not end in a vowel or "n", "N", "l", and "m", the case suffix "-tu" is selected; if the stem ends in a vowel or "n", "N", "l", and "m", the case suffix "-du" is selected. Correcting incorrectly identified case suffixes using Mongolian rules can further improve the accuracy of speech recognition. Furthermore, the recognition results can be displayed in Mongolian script. This solves the problem that some traditional acoustic and language models cannot completely distinguish similar case suffixes, and also resolves the issue of displaying Mongolian script.
[0054] It is understood that the present invention has been described through some embodiments, and those skilled in the art will recognize that various changes or equivalent substitutions can be made to these features and embodiments without departing from the spirit and scope of the invention. Furthermore, under the teachings of the present invention, these features and embodiments can be modified to adapt to specific situations and materials without departing from the spirit and scope of the invention. Therefore, the present invention is not limited to the specific embodiments disclosed herein, and all embodiments falling within the scope of the claims of this application are within the protection scope of the present invention.
Claims
1. A Mongolian speech recognition method for complex scenarios, characterized in that, Includes the following steps: Step S1: Dictionary construction stage; Segment the words in the Mongolian training text into stems (excluding verbs), case suffixes, and verb forms; Simultaneously, establish a whole-word pronunciation dictionary and a word-segmentation pronunciation dictionary based on stems (excluding verbs), case suffixes, and verb forms; Use the segmented training text to train the language model; Step S2: Multimodal feature extraction stage; Deep noise reduction network is used to perform deep noise suppression on the input speech signal, auditory encoder is used to extract acoustic features, visual encoder is used to extract lip movement temporal features from synchronous video stream, and the extracted lip movement temporal features are temporally aligned with the acoustic features. Step S3: Multimodal feature fusion stage; The acoustic-visual fusion model receives the noise-suppressed acoustic features and the temporal features of lip movement after temporal alignment, and constructs an acoustic-visual joint representation through a dynamic weight allocation strategy; Step S4: Transfer learning and fine-tuning stage; First, the acoustic-visual fusion model is pre-trained using a multilingual general speech dataset. Then, the parameters of the low-level feature extraction network of the acoustic-visual fusion model are frozen, and the parameters of the classification layer and language adaptation module of the model are optimized on the Mongolian dataset. Step S5: Decoding stage; A weighted finite-state converter decoding network is constructed using an acoustic model, a language model, and a dedicated pronunciation dictionary. The optimal path is searched using the Viterbi algorithm, and the intermediate text is output in the form of stems other than verbs, case suffixes, and verbs. The path penalty is applied to low-frequency stems and low-frequency case suffixes by combining the probability distribution of the language model. Step S6: Post-processing stage; Based on the matching rules of Mongolian positive stems and case suffixes, the case suffix errors in the decoded text are corrected and the corresponding Latin words are formed. At the same time, the recognized Latin words are segmented and punctuation marks are added. Finally, through the comparison relationship between Latin and Mongolian words, they are converted into actual Mongolian words, and then Mongolian sentences are formed.
2. The Mongolian speech recognition method for complex scenarios according to claim 1, characterized in that, The specific steps for segmenting words in the Mongolian training text into stems (excluding verbs), case suffixes, and verb forms are as follows: First, the Mongolian text is converted into Latin form, and the stems of words other than verbs and case suffixes are separated by narrow, unbroken spaces. The obtained whole-word pronunciation dictionary stores complete Mongolian words and their corresponding pronunciations for acoustic model training; the word-segmentation pronunciation dictionary contains segmented stems, case suffixes, verbs and their corresponding pronunciations, and all possible pronunciations of case suffixes, for the decoding stage.
3. The Mongolian speech recognition method for complex scenarios according to claim 2, characterized in that, The acoustic-visual fusion model adopts a cascaded structure. First, a bidirectional long short-term memory network is used to locally temporally encode the acoustic features and the temporal features of lip movements, respectively. Then, the output features of the bidirectional long short-term memory network are input into the Transformer encoder. A multi-head self-attention mechanism is used to capture long-distance dependencies across modalities. Finally, an acoustic-visual joint representation is generated according to a dynamic weight allocation strategy. The specific implementation of the dynamic weight allocation strategy is as follows: calculate the residual noise intensity in the acoustic features. If the noise intensity still exceeds the preset threshold after noise reduction and suppression in step S2, increase the weight ratio of the lip movement temporal features. If the noise intensity does not exceed the preset threshold, maintain the default weight allocation or give priority to the use of acoustic features.
4. The Mongolian speech recognition method for complex scenarios according to claim 3, characterized in that, The classification layer computation maps the acoustic-visual joint representation to the posterior probabilities of phonemes, stems, case suffixes, and verbs; the language adaptation module introduces Mongolian linguistic constraints, including vowel harmony, palatalization rules, and case suffix co-occurrence probabilities, and corrects the classification layer output through a learnable constraint matrix.
5. The Mongolian speech recognition method for complex scenarios according to claim 4, characterized in that, The specific process for path penaltying of low-frequency stems and low-frequency case suffixes is as follows: combining the probability distribution of the language model, the frequency of each stem and case suffix in the training corpus is counted. If the frequency is lower than a preset threshold, it is marked as a low-frequency unit. Penalties are applied to paths containing low-frequency stems or case suffixes, and the penalty intensity is negatively correlated with the frequency of the low-frequency unit. Finally, the path with the highest total score is selected as the optimal recognition result.
6. The Mongolian speech recognition method for complex scenarios according to claim 5, characterized in that, The specific rules for matching Mongolian masculine stems and case suffixes are as follows: Taking the choice of case suffixes -dv, -du, -tv, and -tu as an example, for masculine stems, if the stem ends in a consonant, choose the -tv case suffix; if the stem ends in a vowel or n, choose the -dv case suffix. For nonmasculine stems, if the stem does not end in a consonant, choose the -tu case suffix; if the stem ends in a vowel or n, N, l, or m, choose the -du case suffix. Consonants do not include vowels or n, N, l, or m.
