Intelligent ancient Tibetan character dividing method based on character combination structure and Attention BiLSTM

Abstract

The invention discloses an ancient Tibetan intelligent character dividing method based on a character combination structure and Attention BiLSTM, and relates to the technical field of natural language processing, and the method comprises the steps: collecting electronic literatures, ancient book digital achievements and non-divided Tibetan texts in a standard corpus; performing cleaning, format regularization and VCC sequence extraction on the text, labeling a configuration role and completing character standardization processing; through Unicode coding standardization and VCC structure verification, illegal combinations are corrected, and homomorphic and heteromorphic codes are unified; calculating a structure specification coefficient, a semantic aggregation coefficient and a structure legality coefficient, dynamically evaluating data cleaning, model semantic consistency and language path legality, and automatically adjusting a strategy for unqualified conditions; according to the method, an enhanced model fusing residual connection and a multi-head attention mechanism is combined with a CRF layer, based on a BMES label system and a Viterbi algorithm, high-precision word segmentation prediction and error correction are achieved, and key technical support is provided for digital processing of the ancient Tibetan.

Description

Technical Field

[0001] The present invention relates to the technical field of natural language processing, and in particular to an ancient Tibetan intelligent word segmentation method based on word group structure and AttentionBiLSTM. Background Art

[0002] With the continuous advancement of digital humanities research, the digitization of ancient texts has become a crucial tool for preserving and inheriting cultural heritage. Among my country's rich and diverse ancient textual resources, Tibetan texts, due to their unique cultural value and linguistic characteristics, have become a key research topic. Digitizing ancient Tibetan texts not only helps preserve precious historical materials but also provides a convenient platform for query and analysis for academic research.

[0003] Tibetan, a script with a long history and complex structure, is far more complex than Chinese and English. Tibetan features a multi-layered structure, from Unicode characters and compound structures like VCC to characters, words, and sentences. This multi-layered structure makes automated processing of Tibetan text challenging. This is particularly true during optical character recognition (OCR), where Tibetan character separators are often misidentified or omitted due to their unclear morphology, leading to significant errors in subsequent text processing.

[0004] Tibetan character segmentation, a fundamental step in Tibetan natural language processing, directly impacts the quality and effectiveness of multiple downstream tasks, including lexical analysis, syntactic parsing, semantic understanding, and machine translation. Existing Tibetan character segmentation technologies often rely on rules or simple statistical models, which struggle to accurately capture the complex structure and contextual semantics of Tibetan characters, resulting in insufficient character segmentation accuracy. Summary of the Invention

[0005] In view of the shortcomings of the existing technology, the present invention provides an ancient Tibetan intelligent word segmentation method based on word group structure and AttentionBiLSTM to solve the problems mentioned in the background technology.

[0006] To achieve the above objectives, the present invention is implemented through the following technical solutions: an ancient Tibetan intelligent word segmentation method based on word group structure and AttentionBiLSTM, comprising the following steps:

[0007] Step 1: Collect ancient Tibetan document data, including extracting unclassified text from electronic documents, digitized ancient books, and standard corpora; perform OCR recognition and manual proofreading on image documents, preserving the original character structure; and introduce manually annotated samples containing composite structure VCC boundary and term alignment annotations;

[0008] Step 2: Clean and format the collected text, extract and annotate VCC sequences and their morphological roles, perform boundary alignment based on manually annotated corpus, and implement standardization by applying character mapping rules.

[0009] Step 3: Perform Unicode encoding standardization, VCC structure verification and standard rearrangement on the character sequence, standardize multi-character word structure, correct illegal or redundant combinations, and unify the encoding of homographs with different codes;

[0010] Step 4: By real-time monitoring of the data compression ratio and effective coverage, and the number of VCC types and the total number of words in the covered text before and after cleaning, the structural standardization coefficient JGX is calculated and compared with the first threshold Q1 to determine whether the VCC cleaning effect is qualified. If it is unqualified, a strategy is given;

[0011] Step 5: Build an enhanced BiLSTM-Attention model that integrates residual connections and multi-head attention mechanisms, model the contextual features of the structured VCC sequence, and calculate the semantic aggregation coefficient YJX through multi-layer attention consistency evaluation. Compare and analyze it with the second threshold Q2 to determine whether the semantic attention consistency of the model is qualified. If it is unqualified, a strategy is given;

[0012] Step 6: Construct a transfer constraint matrix based on Tibetan linguistics. Combined with the language graph prior and penalty mechanism, calculate the structural legitimacy coefficient HFX through path matching and compare it with the third threshold Q3 to determine whether the language path consistency is qualified. If not, a strategy is implemented.

[0013] Step 7: After structural normalization and model training, use Attention-BiLSTM-CRF to predict BMES labels for the VCC sequence, use Viterbi decoding to generate word segmentation results, correct prediction errors, and output the final word segmentation.

[0014] Preferably, step one includes:

[0015] S11. Collect text data of ancient Tibetan documents, including extracting unsegmented Tibetan texts from electronic Tibetan documents, digitized ancient books, and standard corpora; identify and manually proofread document images using OCR tools to preserve the original structural form of the characters; and introduce samples from manually annotated corpora that contain composite structure VCC boundary markers and term alignment annotations.

[0016] Preferably, step 2 includes:

[0017] S21. Clean and format the collected data, remove illegal markers and garbled characters, unify the text encoding format, repair paragraph boundaries, and ensure the integrity of the document structure;

[0018] S22: Extract VCC sequences, decompose the text into character combination structures, preliminarily delineate VCC boundaries, and align them with the terms and boundary information in the manually annotated corpus;

[0019] S23, marking the character formation type, marking each VCC with its role in the character formation structure;

[0020] S24. Calling Tibetan character mapping rules to complete character-level unification and standardization preprocessing.

[0021] Preferably, step three includes:

[0022] S31, perform Unicode encoding standardization, by processing non-standard cleavage symbols and modifiers, unify two-character base words into base characters plus a superscript form, and unify multi-character vowel combinations into a single character representation;

[0023] S32. Verify and correct the VCC structure by detecting invalid VCCs starting with vowels, verifying that each VCC contains only one base character, and handling the case where a single VCC contains multiple vowels, and sorting according to the standard of "base character + superscript + prefix + subscript + suffix + vowel + symbol";

[0024] S33, normalize word structure by forcibly segmenting words with ≥5 VCCs, verify the legitimacy of word combinations using Tibetan grammar rules, and filter out rare word combinations with low frequency;

[0025] S34, processing homographs with different codes is performed by establishing a homograph character mapping table, which includes three types of characters: base characters, vowels, and symbols, and formulating a unified encoding rule for each type of character.

[0026] Preferably, step four includes:

[0027] S41. After the structural cleaning of the VCC is completed, the structural standard coefficient JGX is calculated by real-time monitoring of the data compression ratio and effective coverage rate, combined with the number of VCC types before and after cleaning and the total number of words in the covered text, after dimensionless processing;

[0028] S42, by presetting a first threshold Q1 in advance and comparing and analyzing the structural specification coefficient JGX with the first threshold Q1, obtaining a first evaluation result includes:

[0029] When the structural specification coefficient JGX ≥ the first threshold Q1, it indicates that the cleaning effect of VCC is qualified and is used as model training data for continuous monitoring;

[0030] When the structural norm coefficient JGX is less than the first threshold Q1, it indicates that the VCC cleaning effect is unsatisfactory, triggering the first warning instruction and generating the first strategy: count the VCC structures with a frequency ≥ L before cleaning, and the missing parts after cleaning, setting L = 3, to construct a candidate set of high-frequency missing combinations; based on the candidate set, roll back and import from the structure set before cleaning, without participating in sorting and reconstruction, only restore the original combination, and mark it as a soft rollback structure; lower the word structure filtering threshold, and eliminate the low-frequency combinations of the original rule, i.e., <3, and change it to eliminate the low-frequency combinations with a frequency of <2 and not appearing in the first 10,000 words of the corpus; and automatically discover the structures that are lost due to cleaning but contribute to the coverage rate to make up for the information loss; recalculate until the structural norm coefficient JGX is greater than or equal to the first threshold Q1.

[0031] Preferably, step five includes:

[0032] S51, Enhanced BiLSTM-Attention model construction, by embedding the structured VCC sequence into the vector space, building a multi-layer bidirectional LSTM network with residual connections, and introducing a multi-head attention mechanism to extract contextual features;

[0033] S52. During the model training process, a multi-layer representation consistency evaluation mechanism is used to monitor the degree of attention mechanism's attention to the same semantic unit at different levels; based on the consistency of the attention score matrix and the aggregation entropy of the context representation, the semantic aggregation coefficient YJX is calculated after dimensionless processing.

[0034] Preferably, step five further includes:

[0035] S53, by presetting a second threshold Q2 in advance and comparing and analyzing the semantic aggregation coefficient YJX with the second threshold Q2, obtaining a second evaluation result includes:

[0036] When the semantic aggregation coefficient YJX ≤ the second threshold Q2, it means that the semantic attention consistency of the model is qualified and continuous monitoring is required;

[0037] When the semantic aggregation coefficient YJX> the second threshold Q2, it means that the semantic attention consistency of the model is unqualified, triggering the second warning instruction and generating the second strategy: reducing the number of attention heads H from 8 to 4 to reduce the source of disagreement; applying a semantic regularization term to each attention head to constrain its attention center to not deviate from the average distribution Attention is guided to focus on manually annotated semantic boundaries in the training data, such as syntactic key nodes; external part-of-speech tags or syntactic dependency information are used to guide different attention heads to focus on different language roles, such as base words and modifiers; a KL divergence regularization term is introduced to the Attention distribution to promote consistency between layers; and the semantic aggregation coefficient YJX is recalculated until it is ≤ the second threshold Q2.

[0038] Preferably, step six includes:

[0039] S61. A transfer constraint matrix is ​​constructed based on Tibetan linguistic knowledge. The language graph prior probability and illegal transfer penalty mechanism are introduced. Based on the path overlap between the actual annotated path and the legal path generated in the language knowledge base, dynamic path matching and prior edge weight comparison are performed. After dimensionless processing, the structural legitimacy coefficient HFX is calculated.

[0040] Preferably, step six further includes:

[0041] S61: Presetting a third threshold Q3 in advance and comparing and analyzing the structural legitimacy coefficient HFX with the third threshold Q3 to obtain a third evaluation result includes:

[0042] When the structural legitimacy coefficient HFX ≥ the third threshold Q3, it means that the language path consistency is qualified and continuous monitoring is required;

[0043] When the structural legitimacy coefficient HFX is less than the third threshold Q3, it indicates that the language path consistency is unqualified, triggering the third warning instruction and generating the third strategy: In the CRF transfer matrix, negative edge weights are introduced to illegal paths, such as setting them to -∞ or significantly less than 0, to inhibit the model from learning illegal transfers; increase the structural loss term in the training stage, Penalize the gradient for transfer; add legal paths with high frequency but not included in the corpus into the graph; integrate language rules from multiple sources to construct a fusion prior path graph, such as combining the "Handbook of Ancient Tibetan Grammar" and the "Guide to Tibetan Morphological Structure"; add fuzzy boundary transition paths; sample and train on unqualified sample sets to strengthen the recognition of boundary labels; add Dropout and LayerNorm to the CRF layer to improve its generalization ability for legal paths; retrain the model and recalculate HFX after each epoch, and continue to optimize until the structural legitimacy coefficient HFX ≥ the third threshold Q3.

[0044] Preferably, step seven includes:

[0045] S71. After structural normalization and model training, the trained enhanced Attention-BiLSTM-CRF network is used to predict BMES labels for the input structured VCC sequence. The Viterbi dynamic programming algorithm is used on the transition probability graph output by the CRF layer to decode the globally optimal label path. The label path is the Tibetan character unit division method that the model believes best conforms to linguistic rules.

[0046] S72. Introduce the BMES tag system, where B represents the starting character of a word structure, M represents the middle character within a word, E represents the ending character, and S represents a single-character structure that forms an independent word. Map the predicted BMES tag sequence to the corresponding word segmentation point to generate preliminary word segmentation results. Correct common prediction errors, including: merging multiple consecutive S tags; performing fallback corrections when illegal M start or E isolated tags appear; replacing structures that violate Tibetan word formation logic with alternative paths in the language map; and formulating post-processing rules based on annotation statistics and language rules.

[0047] S73. Outputting the optimized and corrected ancient Tibetan character segmentation result as the final result of the character segmentation method.

[0048] This invention provides an intelligent ancient Tibetan character segmentation method based on character structure and AttentionBiLSTM. It has the following beneficial effects:

[0049] (1) This intelligent ancient Tibetan character segmentation method based on character group structure and Attention BiLSTM introduces a structure cleaning and VCC standardization process to unify the character combination structure, eliminate redundancy, and correct illegal structures, significantly improving the accuracy and consistency of structural expression. Combined with the structural standardization coefficient JGX evaluation mechanism, it effectively ensures the coverage integrity of high-frequency character-building units, making the character segmentation model stable and applicable to a wider range of corpora.

[0050] (2) This is an intelligent word segmentation method for ancient Tibetan based on word group structure and AttentionBiLSTM. It adopts an enhanced BiLSTM model that integrates residual connection and multi-head attention mechanism to perform deep context modeling on structured VCC sequences; combined with the semantic aggregation coefficient YJX and consistency tuning strategy, it effectively guides the model to focus on semantic key structures and improves the recognition and processing capabilities of complex language phenomena such as phrase nesting and morphological changes.

[0051] (3) This intelligent ancient Tibetan word segmentation method based on word group structure and AttentionBiLSTM uses language graph and transfer constraint mechanism, combined with the structural legitimacy coefficient HFX evaluation standard, to conduct rationality review and structural penalty optimization on the model prediction path, suppress the generation of illegal paths from the source, improve the consistency of model output with ancient Tibetan language norms, and significantly reduce invalid segmentation and semantic deviation.

[0052] (4) This intelligent ancient Tibetan character segmentation method based on character group structure and AttentionBiLSTM combines the BMES labeling system and Viterbi decoding strategy to post-process and correct errors in the model prediction results, automatically identify and correct common label conflicts and illegal character formation patterns; integrate language rules and statistical features to optimize and refine the final output results, ensuring the high accuracy and practical value of the ancient Tibetan character segmentation results. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] Figure 1 This is a schematic diagram of the steps of the ancient Tibetan intelligent word segmentation method based on word group structure and AttentionBiLSTM in the present invention. DETAILED DESCRIPTION

[0054] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0055] Example 1

[0056] See also Figure 1 The present invention provides an ancient Tibetan intelligent word segmentation method based on word group structure and AttentionBiLSTM, comprising the following steps:

[0057] Step 1: Collect ancient Tibetan document data, including extracting unclassified text from electronic documents, digitized ancient books, and standard corpora; perform OCR recognition and manual proofreading on image documents, preserving the original character structure; and introduce manually annotated samples containing composite structure VCC boundary and term alignment annotations;

[0058] Step 2: Clean and format the collected text, extract and annotate VCC sequences and their morphological roles, perform boundary alignment based on manually annotated corpus, and implement standardization by applying character mapping rules.

[0059] Step 3: Perform Unicode encoding standardization, VCC structure verification and standard rearrangement on the character sequence, standardize multi-character word structure, correct illegal or redundant combinations, and unify the encoding of homographs with different codes;

[0060] Step 4: By real-time monitoring of the data compression ratio and effective coverage, and the number of VCC types and the total number of words in the covered text before and after cleaning, the structural standardization coefficient JGX is calculated and compared with the first threshold Q1 to determine whether the VCC cleaning effect is qualified. If it is unqualified, a strategy is given;

[0061] Step 5: Build an enhanced BiLSTM-Attention model that integrates residual connections and multi-head attention mechanisms, model the contextual features of the structured VCC sequence, and calculate the semantic aggregation coefficient YJX through multi-layer attention consistency evaluation. Compare and analyze it with the second threshold Q2 to determine whether the semantic attention consistency of the model is qualified. If it is unqualified, a strategy is given;

[0062] Step 6: Construct a transfer constraint matrix based on Tibetan linguistics. Combined with the language graph prior and penalty mechanism, calculate the structural legitimacy coefficient HFX through path matching and compare it with the third threshold Q3 to determine whether the language path consistency is qualified. If not, a strategy is implemented.

[0063] Step 7: After structural normalization and model training, use Attention-BiLSTM-CRF to predict BMES labels for the VCC sequence, use Viterbi decoding to generate word segmentation results, correct prediction errors, and output the final word segmentation.

[0064] In this embodiment, by introducing the triple evaluation indicators of structural norm coefficient JGX, semantic aggregation coefficient YJX and structural legitimacy coefficient HFX, and setting corresponding thresholds and error correction strategies respectively, a full-process quality control mechanism covering data cleaning, semantic modeling and language path legitimacy is constructed, which can realize real-time evaluation and dynamic optimization in the process of automated training and prediction, and significantly improve the accuracy, stability and language compliance of the ancient Tibetan character segmentation results.

[0065] Example 2

[0066] This embodiment is explained in Example 1, please refer to Figure 1 ,Specifically, step one includes:

[0067] S11. Collect text data of ancient Tibetan documents, including extracting unsegmented Tibetan texts from electronic Tibetan documents, digitized ancient books, and standard corpora; identify and manually proofread document images using OCR tools to preserve the original structural form of the characters; and introduce samples from manually annotated corpora that contain composite structure VCC boundary markers and term alignment annotations.

[0068] In this embodiment, by combining electronic text collection and image OCR recognition, and introducing manually annotated samples containing VCC boundary and term alignment information, the present invention can comprehensively cover various data forms of ancient Tibetan documents, ensure the integrity and accuracy of word structure information, and provide high-quality, structurally faithful original corpus support for subsequent structured processing and model training.

[0069] Example 3

[0070] This embodiment is explained in Example 2, please refer to Figure 1 ,Specifically, step 2 includes:

[0071] S21. Clean and format the collected data, remove illegal markers and garbled characters, unify the text encoding format, repair paragraph boundaries, and ensure the integrity of the document structure;

[0072] S22: Extract VCC sequences, decompose the text into character combination structures, preliminarily delineate VCC boundaries, and align them with the terms and boundary information in the manually annotated corpus;

[0073] S23, marking the character formation type, marking each VCC with its role in the character formation structure;

[0074] S24. Calling Tibetan characters to complete character-level unification and standardization preprocessing.

[0075] In this embodiment, by cleaning and regularizing the collected data, unifying the shape annotation and character mapping, it is possible to effectively eliminate coding ambiguity and character noise, clarify the functional role of VCC in the character structure, significantly improve the structural consistency and computability of ancient Tibetan texts, and provide a semantically clear and formatted input basis for the character segmentation model.

[0076] Example 4

[0077] This embodiment is explained in Example 8, please refer to Figure 1 ,Specifically, step three includes:

[0078] S31, perform Unicode encoding standardization, by processing non-standard cleavage symbols and modifiers, unify two-character base words into base characters plus a superscript form, and unify multi-character vowel combinations into a single character representation;

[0079] S32. Verify and correct the VCC structure by detecting invalid VCCs starting with vowels, verifying that each VCC contains only one base character, and handling the case where a single VCC contains multiple vowels, and sorting according to the standard of "base character + superscript + prefix + subscript + suffix + vowel + symbol";

[0080] S33, normalize word structure by forcibly segmenting words with ≥5 VCCs, verify the legitimacy of word combinations using Tibetan grammar rules, and filter out rare word combinations with low frequency;

[0081] S34, processing homographs with different codes is performed by establishing a homograph character mapping table, which includes three types of characters: base characters, vowels, and symbols, and formulating a unified encoding rule for each type of character.

[0082] In this embodiment, by performing Unicode normalization, structural verification, and unified processing of homographs and different codes on character sequences, illegal or redundant character combinations can be systematically corrected, the legitimacy and encoding consistency of the VCC structure can be improved, semantic interference and structural ambiguity in model training can be effectively reduced, and the accuracy and stability of ancient Tibetan character segmentation can be enhanced.

[0083] Example 5

[0084] This embodiment is explained in Example 4. Please refer to Figure 1 ,Specifically, step four includes:

[0085] S41. After the VCC is cleaned, the structural normalization coefficient JGX is calculated by real-time monitoring of the data compression ratio and effective coverage rate, combined with the number of VCC types and the total number of words in the covered text before and after cleaning, and dimensionless processing. The formula is as follows:

[0086]

[0087] Where Npre represents the number of VCC types before structural cleaning, Npost represents the number of VCC types after structural cleaning, Cpre represents the total number of Tibetan terms covered before cleaning, and Cpost represents the total number of Tibetan terms covered after cleaning.

[0088] S42, by presetting a first threshold Q1 in advance and comparing and analyzing the structural specification coefficient JGX with the first threshold Q1, obtaining a first evaluation result includes:

[0089] When the structural specification coefficient JGX ≥ the first threshold Q1, it indicates that the cleaning effect of VCC is qualified and is used as model training data for continuous monitoring;

[0090] When the structural norm coefficient JGX is less than the first threshold Q1, it indicates that the VCC cleaning effect is unsatisfactory, triggering the first warning instruction and generating the first strategy: count the VCC structures with a frequency ≥ L before cleaning, and the missing parts after cleaning, setting L = 3, to construct a candidate set of high-frequency missing combinations; based on the candidate set, roll back and import from the structure set before cleaning, without participating in sorting and reconstruction, only restore the original combination, and mark it as a soft rollback structure; lower the word structure filtering threshold, and eliminate the low-frequency combinations of the original rule, i.e., <3, and change it to eliminate the low-frequency combinations with a frequency of <2 and not appearing in the first 10,000 words of the corpus; and automatically discover the structures that are lost due to cleaning but contribute to the coverage rate to make up for the information loss; recalculate until the structural norm coefficient JGX is greater than or equal to the first threshold Q1.

[0091] In this embodiment, by introducing the structural normalization coefficient JGX and its comparative evaluation mechanism with the first threshold Q1, quantitative judgment and dynamic feedback of the VCC cleaning effect are achieved, which can effectively identify and repair high-frequency combination structures that are mistakenly deleted during the cleaning process, ensure the corpus coverage and data integrity, and thus improve the data quality and word segmentation accuracy of subsequent modeling.

[0092] Example 6

[0093] This embodiment is explained in Example 5, please refer to Figure 1 ,Specifically, step five includes:

[0094] S51, Enhanced BiLSTM-Attention model construction, by embedding the structured VCC sequence into the vector space, building a multi-layer bidirectional LSTM network with residual connections, and introducing a multi-head attention mechanism to extract contextual features;

[0095] S52. During model training, a multi-layer representation consistency evaluation mechanism is used to monitor the degree of attention paid by the attention mechanism to the same semantic unit at different levels. Based on the consistency of the attention score matrix and the aggregation entropy of the context representation, the semantic aggregation coefficient YJX is calculated after dimensionless processing. The formula is as follows:

[0096]

[0097] Where T represents the time step, H represents the number of attention heads, It is represented as the attention distribution of the h-th head at time t, It is represented by the average distribution of all attention heads, and JS represents the Jensen-Shannon divergence, which is a measure of the degree of semantic aggregation.

[0098] In this embodiment, by introducing the semantic aggregation coefficient YJX and the multi-head attention consistency evaluation mechanism, the model's focus consistency on VCC semantic units at different levels can be accurately measured, and the model's ability to understand complex word structures can be improved, thereby enhancing the semantic sensitivity and structural analysis accuracy of the word segmentation model, and significantly improving the accuracy of intelligent word segmentation of ancient Tibetan texts.

[0099] Example 7

[0100] This embodiment is explained in Example 6, please refer to Figure 1 Specifically, step five also includes:

[0101] S53, by presetting a second threshold Q2 in advance and comparing and analyzing the semantic aggregation coefficient YJX with the second threshold Q2, obtaining a second evaluation result includes:

[0102] When the semantic aggregation coefficient YJX ≤ the second threshold Q2, it means that the semantic attention consistency of the model is qualified and continuous monitoring is required;

[0103] When the semantic aggregation coefficient YJX> the second threshold Q2, it means that the semantic attention consistency of the model is unqualified, triggering the second warning instruction and generating the second strategy: reducing the number of attention heads H from 8 to 4 to reduce the source of disagreement; applying a semantic regularization term to each attention head to constrain its attention center to not deviate from the average distribution Attention is guided to focus on manually annotated semantic boundaries in the training data, such as syntactic key nodes; external part-of-speech tags or syntactic dependency information are used to guide different attention heads to focus on different language roles, such as base words and modifiers; a KL divergence regularization term is introduced to the Attention distribution to promote consistency between layers; and the semantic aggregation coefficient YJX is recalculated until it is ≤ the second threshold Q2.

[0104] In this embodiment, by introducing the second threshold Q2 to dynamically evaluate the semantic aggregation coefficient YJX and executing the optimization strategy when the attention consistency is unqualified, the information divergence in the multi-head attention mechanism can be effectively reduced, and the model's focusing ability and stability on the semantic structure of ancient Tibetan can be improved.

[0105] Example 8

[0106] This embodiment is explained in Example 7, please refer to Figure 1 ,Specifically, step six includes:

[0107] S61. A transfer constraint matrix is ​​constructed based on Tibetan linguistic knowledge. The language graph prior probability and illegal transfer penalty mechanism are introduced. Based on the path overlap between the actual annotated path and the legal path generated in the language knowledge base, dynamic path matching and prior edge weight comparison are performed. After dimensionless processing, the structural legitimacy coefficient HFX is calculated. The formula is as follows:

[0108]

[0109] Where N represents the sequence length, Y i represents the i-th predicted label, Plegal represents the set of legal transfer paths defined by the language graph, and α(.) represents the indicator function.

[0110] In this embodiment, by introducing the transfer constraint matrix constructed by Tibetan linguistics and calculating the structural legitimacy coefficient HFX, the label sequence output by the model can be effectively compared and corrected with the legal structure in the language graph, thereby improving the linguistic compliance and interpretability of the model's word segmentation results, avoiding the generation of illegal structural combinations, and enhancing the system's inherent constraint ability on ancient Tibetan grammatical rules.

[0111] Example 9

[0112] This embodiment is explained in Example 8, please refer to Figure 1 Specifically, step six also includes:

[0113] S61: Presetting a third threshold Q3 in advance and comparing and analyzing the structural legitimacy coefficient HFX with the third threshold Q3 to obtain a third evaluation result includes:

[0114] When the structural legitimacy coefficient HFX ≥ the third threshold Q3, it means that the language path consistency is qualified and continuous monitoring is required;

[0115] When the structural legitimacy coefficient HFX is less than the third threshold Q3, it indicates that the language path consistency is unqualified, triggering the third warning instruction and generating the third strategy: In the CRF transfer matrix, negative edge weights are introduced to illegal paths, such as setting them to -∞ or significantly less than 0, to inhibit the model from learning illegal transfers; increase the structural loss term in the training stage, Penalize the gradient for transfer; add legal paths with high frequency but not included in the corpus into the graph; integrate language rules from multiple sources to construct a fusion prior path graph, such as combining the "Handbook of Ancient Tibetan Grammar" and the "Guide to Tibetan Morphological Structure"; add fuzzy boundary transition paths; sample and train on unqualified sample sets to strengthen the recognition of boundary labels; add Dropout and LayerNorm to the CRF layer to improve its generalization ability for legal paths; retrain the model and recalculate HFX after each epoch, and continue to optimize until the structural legitimacy coefficient HFX ≥ the third threshold Q3.

[0116] In this embodiment, by setting a comparative analysis between the structural legitimacy coefficient HFX and the preset threshold Q3, and dynamically triggering the third strategy to adjust the model structure and language map when it fails to meet the requirements, the learning tendency of illegal paths can be effectively suppressed, the legal but missed language transfer relationship can be completed, and the model's generalization ability of the ancient Tibetan morphological structure and the consistency of the language path can be improved, thereby ensuring the logical rationality and standardization of the word segmentation results in the linguistic context.

[0117] Example 10

[0118] This embodiment is explained in Example 9, please refer to Figure 1 ,Specifically, step seven includes:

[0119] S71. After structural normalization and model training, the trained enhanced Attention-BiLSTM-CRF network is used to predict BMES labels for the input structured VCC sequence. The Viterbi dynamic programming algorithm is used on the transition probability graph output by the CRF layer to decode the globally optimal label path. The label path is the Tibetan character unit division method that the model believes best conforms to linguistic rules.

[0120] S72. Introduce the BMES tag system, where B represents the starting character of a word structure, M represents the middle character within a word, E represents the ending character, and S represents a single-character structure that forms an independent word. Map the predicted BMES tag sequence to the corresponding word segmentation point to generate preliminary word segmentation results. Correct common prediction errors, including: merging multiple consecutive S tags; performing fallback corrections when illegal M start or E isolated tags appear; replacing structures that violate Tibetan word formation logic with alternative paths in the language map; and formulating post-processing rules based on annotation statistics and language rules.

[0121] S73. Outputting the optimized and corrected ancient Tibetan character segmentation result as the final result of the character segmentation method.

[0122] In this embodiment, an enhanced Attention-BiLSTM-CRF model is adopted in combination with the BMES labeling system and Viterbi decoding to achieve accurate word segmentation prediction of structured VCC sequences. Assisted by automatic correction of common errors and reasonable replacement of language graphs, the accuracy and language consistency of ancient Tibetan word segmentation are effectively improved, ensuring that the word segmentation results are both compliant with linguistic norms and have high practical value.

[0123] The threshold value is set to facilitate comparison. The size of the threshold depends on the amount of sample data and the number of bases set by technicians in this field for each set of sample data; as long as it does not affect the proportional relationship between the parameter and the quantized value.

Claims

1. An intelligent ancient Tibetan character segmentation method based on character group structure and AttentionBiLSTM, characterized in that: The following steps are involved: Step 1: Collect ancient Tibetan document data, including extracting unclassified text from electronic documents, digitized ancient books, and standard corpora; perform OCR recognition and manual proofreading on image documents, preserving the original character structure; and introduce manually annotated samples containing composite structure VCC boundary and term alignment annotations; Step 2: Clean and format the collected text, extract and annotate VCC sequences and their morphological roles, perform boundary alignment based on manually annotated corpus, and implement standardization by applying character mapping rules. Step 3: Perform Unicode encoding standardization, VCC structure verification and standard rearrangement on the character sequence, standardize multi-character word structure, correct illegal or redundant combinations, and unify the encoding of homographs with different codes; Step 4: By real-time monitoring of the data compression ratio and effective coverage, and the number of VCC types and the total number of words in the covered text before and after cleaning, the structural standardization coefficient JGX is calculated and compared with the first threshold Q1 to determine whether the VCC cleaning effect is qualified. If it is unqualified, a strategy is given; Step 5: Build an enhanced BiLSTM-Attention model that integrates residual connections and multi-head attention mechanisms, model the contextual features of the structured VCC sequence, and calculate the semantic aggregation coefficient YJX through multi-layer attention consistency evaluation. Compare and analyze it with the second threshold Q2 to determine whether the semantic attention consistency of the model is qualified. If it is unqualified, a strategy is given; Step 6: Construct a transfer constraint matrix based on Tibetan linguistics. Combined with the language graph prior and penalty mechanism, calculate the structural legitimacy coefficient HFX through path matching and compare it with the third threshold Q3 to determine whether the language path consistency is qualified. If not, a strategy is implemented. Step 7: After structural normalization and model training, use Attention-BiLSTM-CRF to predict BMES labels for the VCC sequence, use Viterbi decoding to generate word segmentation results, correct prediction errors, and output the final word segmentation.

2. The ancient Tibetan intelligent word segmentation method based on word group structure and AttentionBiLSTM according to claim 1 is characterized in that: Step one includes: S11. Collect text data of ancient Tibetan documents, including extracting unsegmented Tibetan text from electronic Tibetan documents, digitized ancient books, and standard corpora; identify and manually proofread document images using OCR tools, preserving the original structural form of the characters; and introduce samples from manually annotated corpora that contain composite structure VCC boundary markers and term alignment annotations.

3. The ancient Tibetan intelligent word segmentation method based on word group structure and AttentionBiLSTM according to claim 2 is characterized in that: Step 2 includes: S21. Clean and format the collected data, remove illegal markers and garbled characters, unify the text encoding format, repair paragraph boundaries, and ensure the integrity of the document structure; S22: Extract VCC sequences, decompose the text into character combination structures, preliminarily delineate VCC boundaries, and align them with the terms and boundary information in the manually annotated corpus; S23, marking the character formation type, marking each VCC with its role in the character formation structure; S24. Calling Tibetan character mapping rules to complete character-level unification and standardization preprocessing.

4. The ancient Tibetan intelligent word segmentation method based on word group structure and AttentionBiLSTM according to claim 3 is characterized in that: Step three includes: S31, perform Unicode encoding standardization, by processing non-standard cleavage symbols and modifiers, unify two-character base words into base characters plus a superscript form, and unify multi-character vowel combinations into a single character representation; S32. Verify and correct the VCC structure by detecting invalid VCCs starting with vowels, verifying that each VCC contains only one base character, and handling the case where a single VCC contains multiple vowels, and sorting according to the standard of "base character + superscript + prefix + subscript + suffix + vowel + symbol"; S33, normalize word structure by forcibly segmenting words with ≥5 VCCs, verify the legitimacy of word combinations using Tibetan grammar rules, and filter out rare word combinations with low frequency; S34, processing homographs with different codes is performed by establishing a homograph character mapping table, which includes three types of characters: base characters, vowels, and symbols, and formulating a unified encoding rule for each type of character.

5. The ancient Tibetan intelligent word segmentation method based on word group structure and AttentionBiLSTM according to claim 4 is characterized in that: Step 4 includes: S41. After the structural cleaning of the VCC is completed, the structural standard coefficient JGX is calculated by real-time monitoring of the data compression ratio and effective coverage rate, combined with the number of VCC types before and after cleaning and the total number of words in the covered text, after dimensionless processing; S42, by presetting a first threshold Q1 in advance and comparing and analyzing the structural specification coefficient JGX with the first threshold Q1, obtaining a first evaluation result includes: When the structural specification coefficient JGX ≥ the first threshold Q1, it indicates that the cleaning effect of VCC is qualified and is used as model training data for continuous monitoring; When the structural norm coefficient JGX is less than the first threshold Q1, it indicates that the VCC cleaning effect is unsatisfactory, triggering the first warning instruction and generating the first strategy: count the VCC structures with a frequency ≥ L before cleaning, and the missing parts after cleaning, setting L=3, to construct a candidate set of high-frequency missing combinations; based on the candidate set, roll back and import from the structure set before cleaning, without participating in sorting and reconstruction, only restore the original combination, and mark it as a soft rollback structure; lower the word structure filtering threshold, and eliminate the low-frequency combinations of the original rule, that is, <3, and change it to eliminate the low-frequency combinations with a frequency of <2 and not appearing in the first 10,000 words of the corpus; and automatically discover the structures that are lost due to cleaning but contribute to the coverage rate to make up for the information loss; recalculate until the structural norm coefficient JGX is greater than or equal to the first threshold Q1.

6. The ancient Tibetan intelligent word segmentation method based on word group structure and AttentionBiLSTM according to claim 5, characterized in that: Step five includes: S51, Enhanced BiLSTM-Attention model construction, by embedding the structured VCC sequence into the vector space, building a multi-layer bidirectional LSTM network with residual connections, and introducing a multi-head attention mechanism to extract contextual features; S52. During the model training process, a multi-layer representation consistency evaluation mechanism is used to monitor the degree of attention mechanism's attention to the same semantic unit at different levels; based on the consistency of the attention score matrix and the aggregation entropy of the context representation, the semantic aggregation coefficient YJX is calculated after dimensionless processing.

7. The ancient Tibetan intelligent word segmentation method based on word group structure and AttentionBiLSTM according to claim 6 is characterized in that: Step five also includes: S53, by presetting a second threshold Q2 in advance and comparing and analyzing the semantic aggregation coefficient YJX with the second threshold Q2, obtaining a second evaluation result includes: When the semantic aggregation coefficient YJX ≤ the second threshold Q2, it means that the semantic attention consistency of the model is qualified and continuous monitoring is required; When the semantic aggregation coefficient YJX> the second threshold Q2, it means that the semantic attention consistency of the model is unqualified, triggering the second warning instruction and generating the second strategy: reducing the number of attention heads H from 8 to 4 to reduce the source of disagreement; applying a semantic regularization term to each attention head to constrain its attention center to not deviate from the average distribution ; Guide attention to focus on manually marked semantic boundaries in the training data, such as syntactic key nodes; Use external part-of-speech tagging or syntactic dependency information to guide different attention heads to focus on different language roles such as base words and modifiers; Introduce the KL divergence regularization term to the Attention distribution to promote consistency between layers; Recalculate until the semantic aggregation coefficient YJX ≤ the second threshold Q2.

8. The ancient Tibetan intelligent character segmentation method based on character group structure and AttentionBiLSTM according to claim 7 is characterized in that: Step six includes: S61. A transfer constraint matrix is ​​constructed based on Tibetan linguistic knowledge. The language graph prior probability and illegal transfer penalty mechanism are introduced. Based on the path overlap between the actual annotated path and the legal path generated in the language knowledge base, dynamic path matching and prior edge weight comparison are performed. After dimensionless processing, the structural legitimacy coefficient HFX is calculated.

9. The ancient Tibetan intelligent character segmentation method based on character group structure and AttentionBiLSTM according to claim 8, characterized in that: Step six also includes: S61: Presetting a third threshold Q3 in advance and comparing and analyzing the structural legitimacy coefficient HFX with the third threshold Q3 to obtain a third evaluation result includes: When the structural legitimacy coefficient HFX ≥ the third threshold Q3, it means that the language path consistency is qualified and continuous monitoring is required; When the structural legitimacy coefficient HFX is less than the third threshold Q3, it indicates that the language path consistency is unqualified, triggering the third warning instruction and generating the third strategy: In the CRF transfer matrix, negative edge weights are introduced to illegal paths, such as setting them to -∞ or significantly less than 0, to inhibit the model from learning illegal transfers; increase the structural loss term in the training stage, Penalize the gradient for transfer; add legal paths with high frequency but not included in the corpus into the graph; integrate language rules from multiple sources to build a fusion prior path graph, such as combining the "Handbook of Ancient Tibetan Grammar" and the "Guide to Tibetan Morphological Structure"; add fuzzy boundary transition paths; sample and train on unqualified sample sets to strengthen the recognition of boundary labels; add Dropout and LayerNorm to the CRF layer to improve its generalization ability for legal paths; retrain the model and recalculate after each epoch , and continue optimizing until the structural legitimacy coefficient HFX ≥ the third threshold Q3.

10. The ancient Tibetan intelligent character segmentation method based on character group structure and AttentionBiLSTM according to claim 9, characterized in that: Step seven includes: S71. After structural normalization and model training, the trained enhanced Attention-BiLSTM-CRF network is used to predict BMES labels for the input structured VCC sequence. The Viterbi dynamic programming algorithm is used on the transition probability graph output by the CRF layer to decode the globally optimal label path. The label path is the Tibetan character unit division method that the model believes best conforms to linguistic rules. S72. Introduce the BMES tag system, where B represents the starting character of a word structure, M represents the middle character within a word, E represents the ending character, and S represents a single-character structure that forms an independent word. Map the predicted BMES tag sequence to the corresponding word segmentation point to generate preliminary word segmentation results. Correct common prediction errors, including: merging multiple consecutive S tags; performing fallback corrections when illegal M start or E isolated tags appear; replacing structures that violate Tibetan word formation logic with alternative paths in the language map; and formulating post-processing rules based on annotation statistics and language rules. S73. Outputting the optimized and corrected ancient Tibetan character segmentation result as the final result of the character segmentation method.

Images