Tibetan multi-granularity sub-word segmentation method and device, medium and computer equipment

Abstract

The invention discloses a Tibetan multi-granularity sub-word segmentation method and device, a medium and computer equipment, and relates to the technical field of natural language processing. The method provides a word segmentation device for Tibetan language characteristic optimization, aims to solve the problem of granularity and semantic imbalance of a general word segmentation device in Tibetan language processing, and comprises the following steps: in a Tibetan language large model sub-word segmentation device, carrying out initial word segmentation on a Tibetan language to be segmented through the Tibetan language large model sub-word segmentation device to obtain an initial word segmentation result, performing hierarchical matching on the initial word segmentation result and the multi-level fusion word list according to a top-down hierarchical dimension reduction strategy to obtain words, syllables, characters and characters which are segmented layer by layer in the Tibetan to be segmented; the multi-level fusion word list comprises a word-level word list, a syllable-level word list, a character-D-level word list and a character-level word list. According to the method, the segmentation accuracy is ensured in the Tibetan large model sub-word segmentation process, and meanwhile, the specific language structure and semantic integrity of the Tibetan are reserved to the maximum extent.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of Tibetan natural language processing, and in particular to a Tibetan multi-granularity subword segmentation method, device, medium and computer equipment. BACKGROUND

[0002] At present, in the field of Tibetan natural language processing, subword segmentation is the first step of language model architecture, and its design advantages and disadvantages have a decisive influence on the performance, training efficiency, cross-language processing ability, compression rate and out-of-vocabulary processing of the language model. An excellent subword segmentation method can effectively control the sequence length to reduce the computational complexity and improve the training speed, and can enhance the model generalization ability by sharing subword units, and can significantly alleviate the out-of-vocabulary problem.

[0003] However, the current general subword segmentation method represented by BPE, WordPiece and Unigram language model is mainly derived from high-resource languages such as English, and these languages usually have stable grammatical structures. When processing languages with unique morphological structures such as Tibetan, there are serious adaptability problems. Tibetan takes syllable as the basic writing unit, and its rich morphological changes (such as affixation and inflection of roots and affixes) and strict case marking system are the key to carrying grammatical and semantic information. The existing method lacks explicit modeling of Tibetan language characteristics, and in the processing process, the integrity of the syllable is easily destroyed, the roots and affixes are incorrectly segmented, and the morphological features and semantic information of the vocabulary are lost. SUMMARY

[0004] Therefore, it is necessary to provide a Tibetan multi-granularity subword segmentation method, device, medium and computer equipment, which can ensure the accuracy of the segmentation while maximizing the preservation of the unique language structure and semantic integrity of Tibetan in the process of Tibetan large model subword segmentation.

[0005] The technical scheme adopted in the present specification is as follows: The present specification provides a Tibetan multi-granularity subword segmentation method, comprising: obtaining a Tibetan large model subword segmenter; the Tibetan large model subword segmenter comprises a hierarchical dimension reduction strategy and a multi-level fusion vocabulary; the multi-level fusion vocabulary comprises a word-level vocabulary, a syllable-level vocabulary, a character-level vocabulary and a character-level vocabulary; the hierarchical dimension reduction strategy comprises descending priority of word, syllable, character and character; inputting the Tibetan to be segmented into the Tibetan large model subword segmenter, and performing initial word segmentation on the Tibetan to be segmented by the Tibetan large model subword segmenter to obtain an initial word segmentation result; the initial word segmentation result comprises a plurality of words; In the Tibetan large model sub-word segmenter, a plurality of words are hierarchically matched with a word-level vocabulary table to obtain matched words and unmatched words; syllable boundary recognition is performed on the unmatched words to complete syllable segmentation; syllable-level vocabulary table matching verification is performed on the segmented syllables through a syllable-level vocabulary table to obtain matched syllables and unmatched syllables; the unmatched syllables are structurally decomposed to obtain a plurality of character radicals, and the plurality of character radicals are verified through a character radical-level vocabulary table to obtain matched character radicals and unmatched character radicals; and character-level segmentation is performed on the unmatched character radicals through a character-level vocabulary table to obtain segmented characters. The matched words, syllables, character radicals, and segmented characters are taken as Tibetan multi-granularity sub-word segmentation results.

[0006] Optionally, the word-level vocabulary table is a first level of a multi-level fusion vocabulary table, and a construction process of the word-level vocabulary table includes: Through quantitative analysis and frequency-driven screening mechanism of a preset scale corpus, a plurality of high-frequency multi-syllable words are selected, and a data-driven optimization strategy is adopted to obtain the word-level vocabulary table by eliminating single-syllable interference items and dynamically thresholding based on word frequency distribution characteristics.

[0007] Optionally, the syllable-level vocabulary table is a second level of the multi-level fusion vocabulary table, and a construction process of the syllable-level vocabulary table includes: The syllable-level vocabulary table is constructed by integrating the Tibetan Standard Syllable Frequency Dictionary and the Tibetan Standard Hierarchical Syllable Library.

[0008] Optionally, the character radical-level vocabulary table is a third level of the multi-level fusion vocabulary table, and includes a plurality of Tibetan character radicals. The character radical is any form that can be segmented on a linear arrangement of Tibetan visible symbols. The character-level vocabulary table is a fourth level of the multi-level fusion vocabulary table, and is a character-level vocabulary table of a Tibetan-Chinese-English mixed language system. The character-level vocabulary table includes a plurality of basic character units.

[0009] Optionally, the method includes: In the process of segmenting the Tibetan text to be segmented, if it is detected that there is a Tibetan text in the contracted form, a constraint mechanism based on "#" mark is used to restore the complete structure of the Tibetan text.

[0010] Optionally, the Tibetan text to be segmented is subjected to initial word segmentation to obtain an initial word segmentation result, including: The Tibetan text to be segmented is subjected to initial word segmentation through a dictionary to obtain an initial word segmentation result.

[0011] The present specification provides a Tibetan multi-granularity sub-word segmentation device, including: The acquisition module is configured to acquire a Tibetan large model subword segmenter; the Tibetan large model subword segmenter comprises a hierarchical dimension reduction strategy and a multi-level fusion vocabulary; the multi-level fusion vocabulary comprises a word-level vocabulary, a syllable-level vocabulary, a character-level vocabulary, and a word-level vocabulary; the hierarchical dimension reduction strategy comprises descending priorities of words, syllables, character-level words, and characters; The segmentation module is configured to input the Tibetan text to be segmented into the Tibetan large model subword segmenter, perform initial segmentation on the Tibetan text to be segmented by the Tibetan large model subword segmenter, and obtain an initial segmentation result; the initial segmentation result comprises a plurality of words; in the Tibetan large model subword segmenter, the plurality of words are matched with the word-level vocabulary respectively to obtain matched words and unmatched words; syllable boundary recognition is performed on the unmatched words to complete syllable segmentation; syllable-level vocabulary matching verification is performed on the segmented syllables by the syllable-level vocabulary to obtain matched syllables and unmatched syllables; structural decomposition is performed on the unmatched syllables to obtain a plurality of character-level words, and the plurality of character-level words are verified by the character-level vocabulary to obtain matched character-level words and unmatched character-level words; character-level segmentation is performed on the unmatched character-level words by the word-level vocabulary to obtain segmented characters; and the matched words, syllables, character-level words, and segmented characters are taken as Tibetan multi-granularity subword segmentation results.

[0012] The present specification provides a computer readable storage medium, which stores a computer program, and the computer program is executed by a processor to implement the Tibetan multi-granularity subword segmentation method.

[0013] The present specification provides a computer device, which comprises a memory, a processor, and a computer program stored in the memory and executable on the processor, and the processor implements the Tibetan multi-granularity subword segmentation method when executing the program.

[0014] The above technical solutions adopted by the present specification can achieve the following beneficial effects: This invention provides a Tibetan language large-scale model word segmenter, which includes a hierarchical dimensionality reduction strategy and a multi-level fusion vocabulary. The multi-level fusion vocabulary includes multiple levels: word-level vocabulary, syllable-level vocabulary, character-level vocabulary, and character-level vocabulary. During the Tibetan language large-scale model word segmentation process, the Tibetan language large-scale model word segmenter performs initial word segmentation on the Tibetan text to be segmented, obtaining initial segmentation results. Then, following a top-down hierarchical dimensionality reduction strategy, the initial segmentation results are hierarchically matched with the multi-level fusion vocabulary to obtain words, syllables, characters, and sub-characters segmented layer by layer from the Tibetan text to be segmented. The hierarchical dimensionality reduction strategy includes descending priority for words, syllables, characters, and sub-characters. In the process of Tibetan language large model word segmentation, the Tibetan language large model word segmenter breaks through the semantic fragmentation limitation of traditional word segmenters with single-granularity segmentation. With a multi-level fused vocabulary as knowledge-driven, it achieves fine segmentation of complex Tibetan language units through a top-down hierarchical dimensionality reduction strategy. It can preserve the unique language structure and semantic integrity of Tibetan to the maximum extent while ensuring text coverage. Attached Figure Description

[0016] The accompanying drawings, which are included to provide a further understanding of this application and form part of this application, illustrate exemplary embodiments and are used to explain this application, but do not constitute an undue limitation of this application. In the drawings:

[0017] Figure 1 This document provides a flowchart illustrating a Tibetan multi-granularity word segmentation method. Figure 2 This specification provides a detailed distribution diagram of a multi-level fusion vocabulary. Figure 3 This is a schematic diagram of an adaptive degradation splitting path provided in this specification; Figure 4 This is a schematic diagram of a computer device used to implement a multi-granularity word segmentation method for Tibetan, as provided in this specification. Detailed Implementation

[0018] To make the objectives, technical solutions, and advantages of this specification clearer, the technical solutions of this application will be clearly and completely described below in conjunction with specific embodiments and corresponding drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of them. All other embodiments obtained by those skilled in the art based on the embodiments in this specification without creative effort are within the scope of protection of this application.

[0019] Biased segmentation directly leads to semantic distortion of the model's input sequence. It disrupts the high coupling between Tibetan syntax and semantics, making it difficult for downstream models to learn the correspondence between case markers and semantic roles. The fundamental flaws in the preprocessing stage of general word segmentation methods when dealing with Tibetan not only fail to leverage the theoretical advantages of word segmentation in improving model efficiency and generalization ability, but also become a source of error propagation and performance bottlenecks, severely restricting the development and practical application of Tibetan large language models.

[0020] In Tibetan natural language processing tasks, word segmentation is a core step in text preprocessing, and its quality directly impacts the performance of large language models. Currently, existing solutions mainly rely on several common word segmentation methods, including Byte-Pair Encoding (BPE) and its variants based on frequency statistics, the WordPiece algorithm based on language model probabilities, and the Unigram language model based on a probabilistic framework. While these methods perform well in high-resource languages ​​such as English, they exhibit significant limitations when processing Tibetan because they fail to fully consider the unique morphological structure and spelling rules of the language.

[0021] Specifically, existing technical solutions can be mainly divided into three categories, each with its own inherent defects: The first type of scheme employs a frequency-based byte-pair encoding method. The core of this method is to construct a vocabulary by iteratively merging high-frequency character pairs. Its advantage lies in its simplicity and lack of language-specific knowledge. However, this mechanism, which relies purely on statistical frequency, completely ignores the linguistic characteristics of Tibetan. Tibetan uses syllables as the basic writing unit, with strict spelling rules and rich morphological variations. During the merging process, the BPE method is prone to disrupting the integrity of syllables, forcibly breaking down syllables that should be a unified semantic unit, or incorrectly segmenting affixes and stems with specific grammatical functions. This segmentation violates the word formation rules of Tibetan, resulting in generated sub-word units lacking clear semantic information, directly affecting the representation learning quality of subsequent models.

[0022] The second approach uses the WordPiece algorithm, which is based on language model probabilities. This method selects merging targets by calculating the increase in likelihood of the merging operation on the training data, theoretically producing more linguistically meaningful subwords. However, the performance of this algorithm is highly dependent on the quality of its underlying statistical language model. In the low-resource scenario of Tibetan, where high-quality labeled data is severely scarce, the trained language model itself is unreliable, and the probability estimation is biased, leading to merging decision errors. As a result, WordPiece not only struggles to consistently identify the correct semantic units in Tibetan, but also has extremely limited ability to model complex morphological changes in Tibetan (such as squaring).

[0023] The third approach relies on probabilistic segmentation based on a unigram language model. This method employs a top-down strategy, providing multiple possible segmentation schemes and their probabilities for the input, theoretically making it more suitable for handling segmentation ambiguity. However, in Tibetan, a language with complex morphology and blurred lexical boundaries, this method faces significant challenges: firstly, the quality of its initial vocabulary construction is difficult to guarantee under low-resource conditions; secondly, given the rich morphological variations in Tibetan, its probabilistic model is also constrained by data sparsity, easily generating multiple segmentation schemes with similar probabilities but inconsistent semantics, failing to stably and accurately output segmentation results that conform to linguistic principles.

[0024] Current mainstream general word segmentation methods face pressing technical challenges when applied to Tibetan, as they fail to adequately consider the language's unique morphological structure and hierarchical features. These methods, primarily designed for high-resource languages ​​like English, suffer from a severe mismatch between their statistical frequency or probability-based segmentation mechanisms and Tibetan's unique multi-level "character-syllable-word" structure. This leads to frequent disruption of syllable integrity during segmentation, incorrect splitting of root and affix combinations, and an inability to effectively identify and preserve key grammatical units such as case markers, resulting in significant semantic information loss. Particularly in the context of Tibetan, a typical low-resource language, frequency-based BPE methods tend to produce segmentation results that do not conform to linguistic principles, while the probabilistic WordPiece method, relying on language model probability, suffers from decreased decision reliability due to insufficient training data. Unigram language models, on the other hand, exhibit severe word segmentation ambiguity in the face of Tibetan's complex morphological variations. Furthermore, existing methods lack mechanisms for handling special linguistic phenomena such as Tibetan contraction, and cannot achieve the best balance between segmentation granularity and semantic preservation. They cannot effectively protect the superimposed writing features of Tibetan, nor can they adaptively select the optimal segmentation granularity according to the specific context. This results in either increasing sequence length and reducing processing efficiency due to overly fine segmentation, or losing key semantic information due to overly coarse segmentation.

[0025] In summary, the fundamental flaw of existing technical solutions lies in the fact that they are all general statistical models designed for high-resource languages, driven primarily by data frequency rather than linguistic knowledge. Therefore, they cannot effectively incorporate key linguistic constraints unique to Tibetan, such as syllable structure and morphological composition, during the segmentation process. This systematic disconnect from the characteristics of the target language leads to inherent biases in the generated word representations, not only failing to effectively alleviate the problem of out-of-vocabulary words in Tibetan but also becoming a bottleneck restricting the performance improvement of language models. This invention is proposed precisely to fundamentally overcome these shortcomings.

[0026] To address the issue of poor performance of general word segmentation methods in Tibetan, this invention proposes a multi-granularity word segmentation method for Tibetan, based on a thorough examination of the morphological characteristics of Tibetan. Tibetan vocabulary exhibits a distinct hierarchical structure, specifically: first, Tibetan characters combine according to specific rules to form words; these words further combine to form syllables; and finally, one or more syllables constitute a word. This hierarchical structure determines the multi-granularity of Tibetan semantic expression. Traditional word segmentation methods often ignore this hierarchical structure, leading to the segmentation of semantically related words, syllables, or words, resulting in information loss. The multi-granularity word segmentation method proposed in this invention, from a linguistic perspective, comprehensively considers different granularity linguistic units such as words, syllables, syllables, and words in Tibetan, establishing a multi-granularity segmentation system that conforms to the linguistic rules of Tibetan, and developing an adaptive granularity selection mechanism. While ensuring segmentation accuracy, it maximizes the preservation of the unique linguistic structure and semantic integrity of Tibetan, thus providing a high-quality text representation foundation for Tibetan natural language processing tasks.

[0027] The technical solutions provided by the various embodiments of this application are described in detail below with reference to the accompanying drawings.

[0028] Figure 1 This is a flowchart illustrating a Tibetan multi-granularity word segmentation method described in this specification, which specifically includes the following steps: S101, Obtain the Tibetan language large model sub-word segmenter; the Tibetan language large model sub-word segmenter includes a hierarchical dimensionality reduction strategy and a multi-level fusion vocabulary; the multi-level fusion vocabulary includes a word-level vocabulary, a syllable-level vocabulary, a character-level vocabulary, and a character-level vocabulary; the hierarchical dimensionality reduction strategy includes descending priority of words, syllables, characters, and words.

[0029] In one embodiment, this invention studies the unique and complex morphological structure of Tibetan, meticulously constructs a character database and a character set database, and establishes them as the fundamental components of the entire vocabulary architecture to solve the problem of out-of-vocabulary words in sub-word segmentation; it uses statistical methods to obtain the usage distribution of syllables and words in a large-scale Tibetan corpus, and expands the vocabulary accordingly, ultimately constructing a multi-level fused vocabulary containing four levels: characters, character sets, syllables, and words. Figure 2 As shown, Figure 2 This is a detailed distribution diagram of a multi-level fusion vocabulary. This multi-level fusion vocabulary fully considers the morphological characteristics of Tibetan, laying a resource foundation for realizing a more accurate multi-granularity sub-word segmentation method for Tibetan.

[0030] In one embodiment, the word-level vocabulary is the first level of the multi-level fused vocabulary. The construction process of the word-level vocabulary includes: selecting multiple high-frequency polysyllabic words through quantitative analysis and frequency-driven screening mechanisms using a pre-defined corpus; and using a data-driven optimization strategy to obtain the word-level vocabulary by removing monosyllabic interference items and performing dynamic thresholding based on word frequency distribution characteristics. Specifically, after obtaining multiple high-frequency polysyllabic words, monosyllabic words are removed, and the multiple polysyllabic words with the highest frequency are used as words in the word-level vocabulary.

[0031] Specifically, the Tibetan word-level vocabulary was selected through quantitative analysis of a large-scale corpus (12.13 GB) and a frequency-driven screening mechanism, resulting in 12,000 high-frequency multisyllabic words. The vocabulary construction adopted a data-driven optimization strategy, which eliminated monosyllabic interference items and performed dynamic thresholding based on word frequency distribution characteristics, thus providing a foundation for efficient representation and coarse-grained segmentation of the Tibetan vocabulary system.

[0032] In one embodiment, the syllable-level vocabulary is the second level of the multi-level fusion vocabulary. The process of constructing the syllable-level vocabulary includes: constructing the syllable-level vocabulary by integrating the Tibetan Standard Syllable Frequency Dictionary with the Tibetan Standard Hierarchical Syllable Library.

[0033] Specifically, syllables are the basic units of Tibetan vocabulary, and the syllable-level vocabulary system includes 9,784 Tibetan syllable units with linguistic representation functions. During its construction, by integrating the currently comprehensive "Standard Tibetan Syllable Frequency Dictionary" and the standard Tibetan hierarchical syllable database, a syllable vocabulary that is both standardized and timely was created, improving the vocabulary's coverage and representational capabilities.

[0034] In one embodiment, the character-level vocabulary is the third level of the multi-level fused vocabulary, which includes multiple Tibetan characters, where each character is any divisible form of the linear arrangement of Tibetan visual symbols.

[0035] Specifically, a "ding" is any segmentable form of the linear arrangement of Tibetan visual symbols, which can include single-layered consonant letters and all vertically stacked letters. The ding-level vocabulary contains 562 Tibetan dings, fully preserving the unique vertical stacking writing characteristics of Tibetan, which helps to understand the morphological changes of Tibetan at a finer granular level.

[0036] In one embodiment, the character-level vocabulary is the fourth level of the multi-level fused vocabulary, and the character-level vocabulary is a character-level vocabulary of the Tibetan-Chinese-English mixed language family. The character-level vocabulary includes multiple basic character units.

[0037] Specifically, the character-level vocabulary, as the core of the multi-level fusion vocabulary, directly determines the ability of the sub-word segmentation method to process Tibetan characters and other common characters. This invention constructs a character-level vocabulary for a Tibetan-Chinese-English mixed language system, containing 4319 basic character units. Among them, the Tibetan character set covers 142 key characters, including base characters, non-positional characters, punctuation marks, and numbers. Based on existing research results, 3500 high-frequency Chinese characters have been selected. In addition, 672 cross-language universal symbols are included, including English letters, Arabic numerals, commonly used Chinese and English symbols, and control characters, to ensure the universality of the vocabulary.

[0038] S102, input the Tibetan text to be segmented into the Tibetan language large model word segmenter, and perform initial word segmentation on the Tibetan text to be segmented through the Tibetan language large model word segmenter to obtain the initial word segmentation result; the initial word segmentation result includes multiple words.

[0039] In one embodiment, the initial segmentation of the Tibetan text to be segmented to obtain the initial segmentation result includes: performing initial segmentation of the Tibetan text to be segmented using a dictionary to obtain the initial segmentation result.

[0040] The first processing stage of the Tibetan multi-granularity word segmentation method is the dictionary-based initial word segmentation process. This step aims to segment the Tibetan text to be segmented into complete words as much as possible to retain the largest semantic units. Then, it attempts to match the segmented words with words in the word-level vocabulary and select the optimal segmentation path.

[0041] S103, in the Tibetan language large-scale sub-word segmenter, multiple words are matched hierarchically with word-level vocabularies to obtain successfully matched words and unmatched words; syllable boundary recognition is performed on unmatched words to complete syllable segmentation; syllable-level vocabulary matching verification is performed on the segmented syllables through the syllable-level vocabulary to obtain successfully matched syllables and unmatched syllables; the unmatched syllables are structurally decomposed to obtain multiple characters, and these characters are verified through a character-level vocabulary to obtain successfully matched characters and unmatched characters; character-level segmentation is performed on the unmatched characters through a character-level vocabulary to obtain segmented characters; the successfully matched words, syllables, characters, and segmented characters are used as the Tibetan multi-granularity sub-word segmentation results.

[0042] After obtaining the initial word segmentation results, a top-down hierarchical dimensionality reduction strategy can be used to perform hierarchical matching between the initial word segmentation results and the multi-level fused vocabulary to obtain the words, syllables, characters, and sub-characters of the Tibetan text to be segmented layer by layer, specifically including: S201, perform hierarchical matching of multiple words with the word-level vocabulary to obtain successfully matched words and unmatched words.

[0043] This process involves word-level vocabulary matching and downgrading: After initial word segmentation, the initial segmentation results are hierarchically matched with the word-level vocabulary in the pre-constructed multi-level fusion vocabulary. For language units (words) that successfully match the word-level vocabulary, their structural integrity is maintained; for out-of-vocabulary words (words that do not match successfully), a hierarchical downgrading mechanism is implemented, transferring them to the syllable-level analysis process, i.e., S202.

[0044] S202, perform syllable boundary recognition on words that fail to match, and complete syllable segmentation; perform syllable-level word list matching verification on the segmented syllables through a syllable-level word list to obtain the successfully matched syllables and the unmatched syllables.

[0045] Syllable matching and downgrading: For downgraded text units (words that did not match successfully), a hybrid strategy combining syllable-level features based on a multi-level fused vocabulary and a Tibetan syllable segmentation algorithm is used for syllable boundary recognition. After syllable segmentation, a second syllable-level vocabulary matching verification is performed. Successfully matched syllable units are retained; unmatched syllable units are downgraded to the character-level granularity processing layer.

[0046] Optionally, the Tibetan syllable segmentation algorithm is based on the inherent characteristics of the Tibetan writing system and uses a character type-driven state transition mechanism to achieve automatic syllable recognition. The core principle of the algorithm is as follows: by scanning the input text character by character, the character type is determined using the Unicode character encoding range, and characters are divided into three categories according to Tibetan linguistic rules: syllable-forming characters, boundary markers, and illegal characters; a dynamic syllable buffer is maintained. When Tibetan letters (Unicode range U+0F40-U+0FBC and specific additional symbols) are recognized, they are accumulated in the buffer; when syllable separators (་), punctuation marks (།, etc.) or spaces are encountered, the boundary judgment logic is triggered, the contents of the buffer are output as complete syllables, and the boundary markers are retained as independent units. Then the buffer is cleared and the recognition state of the next syllable is entered; for non-Tibetan characters, they are marked as out-of-vocabulary (OOV) characters and output independently to ensure that they are not confused with normal syllables; when the text scanning ends, if there are still residual characters in the buffer, they are output as the last syllable. This algorithm does not require a pre-built dictionary; it can accurately locate syllable boundaries by relying solely on the encoding attributes and positional relationships of the characters themselves. This enables precise parsing of the Tibetan text structure and provides a reliable basic unit segmentation capability for subsequent language processing tasks.

[0047] S203: The unmatched syllables are structurally decomposed to obtain multiple characters. After the multiple characters are verified by the character-level vocabulary, the successfully matched characters and the unmatched characters are obtained.

[0048] Character-level parsing strategy: In the character processing stage, a character segmentation algorithm unique to Tibetan is applied for structural decomposition. After the segmentation results are verified by a character-level vocabulary, entries that match the vocabulary entries maintain their existing structure, while unmatched entries are further downgraded to the character granularity.

[0049] S204: The unmatched characters are segmented using a character-level vocabulary to obtain the segmented characters.

[0050] The character-level vocabulary segments unmatched characters into Tibetan and non-Tibetan characters, such as Chinese and English. For characters that ultimately cannot be matched, forced character-level segmentation is performed to ensure system integrity.

[0051] Multi-granularity segmentation result integration: The segmentation results at four levels—words, syllables, characters, and sub-characters—are systematically integrated to obtain the layer-by-layer segmentation of words, syllables, characters, and sub-characters from the Tibetan text to be segmented. This method preserves language structure information at different granularities, providing richer input for the language model.

[0052] In one embodiment, if a compressed Tibetan text is detected during the process of segmenting Tibetan text, a constraint mechanism based on the "#" symbol is used to restore the complete structure of the Tibetan text.

[0053] This invention proposes a Tibetan multi-granularity word segmentation method that constructs a collaborative Tibetan word segmentation framework encompassing "word-syllable-character-signature" dimensions through a top-down hierarchical dimensionality reduction strategy. Based on dynamic threshold matching and a progressive granularity reduction mechanism, and driven by a multi-level fused vocabulary, this method achieves refined segmentation of complex Tibetan language units through a hierarchical recursive matching-reduction-rematching paradigm. Its core advantage lies in overcoming the semantic fragmentation limitations of traditional single-granularity segmentation by maximizing semantic integrity while ensuring text coverage through an adaptive degradation segmentation path.

[0054] like Figure 3 As shown, the adaptive degradation segmentation path specifically includes: first, segmenting the Tibetan text to be segmented using a dictionary to obtain initial segmentation results; then, determining whether the words segmented in the initial segmentation results are in the word-level vocabulary, and outputting the words in the word-level vocabulary from the initial segmentation results; segmenting the words in the initial segmentation results that are not in the word-level vocabulary into syllables, determining whether the segmented syllables are in the syllable-level vocabulary, and outputting the syllables in the syllable-level vocabulary; segmenting the syllables that are not in the syllable-level vocabulary into characters, determining whether the segmented characters are in the character-level vocabulary, and outputting the characters in the character-level vocabulary; and segmenting the characters that are not in the character-level vocabulary into characters, and outputting them. The output words, syllables, characters, and characters are the results of segmenting the Tibetan text to be segmented.

[0055] The top-down Tibetan multi-granularity word segmentation method proposed in this invention is an effective strategy that comprehensively considers the morphological characteristics of Tibetan. This method employs a top-down strategy, gradually downgrading from coarse-grained word level to fine-grained character level, which better preserves semantic information and avoids over-segmentation. Based on a multi-level fused vocabulary, the algorithm fully utilizes linguistic information at different granularities, such as words, syllables, and individual characters, improving the accuracy and robustness of segmentation.

[0056] In one embodiment, to comprehensively evaluate the reliability, effectiveness, and advantages of the Tibetan multi-granularity word segmentation method, this embodiment compares and analyzes the method with the word segmentation modules used in current mainstream large language models and BERT pre-trained models from two key dimensions: compression rate and semantic integrity. Through quantitative metrics and experimental results, the performance of this method in Tibetan language processing is objectively demonstrated.

[0057] (1) Comparison of compression ratios Compression ratio is a key metric for measuring the efficiency of word segmentation methods. It is defined as the average number of bytes per token after segmentation, measured in bytes / token. The compression ratio directly reflects the number of tokens generated by the segmentation method and the amount of original text information represented by each token. A higher compression ratio means fewer tokens after segmentation, thus reducing the computational load of subsequent model processing and improving processing efficiency. This invention samples 300MB of text from pre-training data resources, processes it using the word segmentation modules of various mainstream models, and calculates the compression ratio. Specific comparison results are shown in Table 1.

[0058] Table 1. Comparison of compression rates of word segmentation methods Table 1 provides a systematic evaluation of the multi-granularity Tibetan word segmentation method from the perspective of compression ratio, achieving a compression ratio of 11.88, which is 0.76% higher than the traditional syllable segmentation method (11.21), indicating that the multi-granularity segmentation strategy can effectively reduce the number of primitives. At the same time, compared with existing pre-trained models such as CINO, the multi-granularity Tibetan word segmentation method significantly improves the compression ratio.

[0059] In the monolingual model, compared with the BPE (11.71) and Unigram (11.12) models trained on a 4GB large-scale monolingual corpus in the present invention and the Tibetan multi-granularity subword segmentation method, the multi-granularity segmentation strategy shows better performance and performs better in balancing the number of primitives and semantic representation. Mainstream multilingual large models (such as GPT4, Llama3, etc.) generally show extremely low compression rates (1.01 - 4.30) in Tibetan processing, reflecting the insufficient adaptability of their shared vocabulary mechanism to low-resource languages. It can be seen that in order to cover the characteristics of multiple languages, multilingual models usually adopt a fine-granularity segmentation strategy, resulting in the problem that Tibetan primitives are over-segmented into atomic units. Although their language generalization ability has been improved, the computational complexity has increased significantly.

[0060] The results of the comparative experiment show that the segmentation method designed for the morphological characteristics of Tibetan has significant advantages in compression efficiency. By integrating multi-granularity information such as characters, sub-characters, syllables, and words, this method ensures that the segmentation results conform to the characteristics of Tibetan agglutinative language while maintaining a reasonable number of primitives. Compared with directly applying general segmentation algorithms or subword segmentation methods of multilingual models, this subword segmentation method that fully combines language characteristics has a better compression effect on Tibetan texts.

[0061] (2)Semantic integrity comparison Semantic integrity aims to examine the degree to which primitives retain the semantic information of the original text after subword segmentation. It is difficult to directly quantify semantic integrity. In view of this, the present invention uses manual judgment to determine whether semantic integrity is damaged. Since semantics has certain complexity and subjectivity, the present invention observes the structural integrity of words, syllables, and sub-characters. The situation where the root and affix are wrongly segmented, or the internal characters of syllables and sub-characters are split, resulting in structural defects, is regarded as damage to semantic integrity. Take "བརྩོན་པ་ཆུ་བོའི་རྒྱུན་བཞིན་སྟེན་ཅིག་ཨང་།།" (Constantly strive like a flowing river) as an example to compare the differences in semantic integrity under different subword segmentation methods. The results are shown in Table 2.

[0062] Based on the semantic integrity evaluation system, the present invention conducts a systematic comparative analysis of seven subword segmentation algorithms. The empirical results show that the proposed multi-granularity Tibetan subword segmentation method of the present invention shows significant performance advantages in the dimension of semantic integrity. This algorithm innovatively integrates more language constraint conditions and achieves a high semantic retention rate at the two language levels of words and syllables. For the unique contraction case in Tibetan, this method designs a constraint mechanism based on the "#" mark (such as "#འི"), and effectively avoids the problem of structural damage through predefined grammar rules.

[0063] Traditional Tibetan syllable segmentation methods (Syllable) maintain the semantic integrity of basic syllables, but their single-layer analysis framework leads to a decrease in the parsing accuracy of high-order semantic features. The segmentation methods (CINO, TBERT, TiBERT) of mainstream pre-training models all have varying degrees of semantic loss. Among them, CINO and TBERT improperly split the syllable "སྟེན", and the processing result of TiBERT even has multiple unknown tokens ([UNK]), seriously affecting the preservation of the original semantics.

[0064] Table 2 Comparison of semantic integrity of subword segmentation methods The general subword segmentation methods BPE and Unigram also have semantic integrity problems. BPE has an improper split like "འི་" when processing characters, and Unigram, similar to CINO and TBERT, wrongly splits the syllable "སྟེན" into two tokens. The byte-level processing method adopted by current mainstream large language models is not included in this comparison because it cannot effectively maintain the semantic integrity of Tibetan.

[0065] From the perspective of the number of tokens, the multi-granularity Tibetan subword segmentation method (9 tokens) has a higher information density compared to other methods, indicating that this method achieves a better compression effect while maintaining semantic integrity. In contrast, TiBERT generates 20 tokens, significantly increasing the sequence length and affecting the processing efficiency of subsequent models.

[0066] In summary, the experimental results show that the multi-granularity Tibetan subword segmentation method proposed in this invention has obvious advantages in terms of compression rate and maintaining semantic integrity. While capturing the language features of Tibetan, this method avoids improper structural splits and provides a reliable data preprocessing technology for the training of Tibetan large language models.

[0067] Technical advantages and beneficial effects of the method provided by this invention (1) Achieved a breakthrough improvement in semantic integrity Compared with the closest prior art, this invention has achieved a qualitative leap in retaining the semantic integrity of Tibetan. Compared with general subword segmentation algorithms (BPE, Unigram) and mainstream multilingual models (such as CINO, TiBERT), this method uses multi-granularity fusion and an adaptive degradation mechanism to minimize the impairment rate of semantic integrity caused by incorrect segmentation. Taking the test of typical examples as an example, this method correctly segments all 9 semantic units, while the comparative methods have serious problems such as wrongly splitting the key syllable "སྟེན" and generating unrecognizable [UNK] symbols, fully demonstrating the powerful parsing ability of this invention for complex Tibetan language phenomena.

[0068] (2) Achieving the best balance between processing efficiency and model performance This invention demonstrates a significant advantage in processing efficiency through a multi-granularity collaborative segmentation strategy. Experimental data shows that the compression ratio of this method reaches 11.88 bytes / token. This metric not only outperforms traditional single-syllable segmentation methods (11.21 bytes / token) but also far surpasses mainstream multilingual large-scale models (such as GPT-4's 1.44 bytes / token). A higher compression ratio means that after segmenting text of the same length, the generated token sequence is shorter, directly reducing the computational overhead and memory usage of downstream pre-trained models by approximately 10%-15%, laying a solid foundation for the efficient training and deployment of large-scale Tibetan language models.

[0069] (3) Deeply adapted to the unique morphological characteristics of Tibetan script The core technical advantage of this invention lies in its deep modeling of the hierarchical structure of Tibetan characters (characters-syllables-words). By constructing a multi-level fused vocabulary covering 4319 characters, 562 syllables, 9784 syllables, and 12000 words, and combining this with a top-down dimensionality reduction segmentation path, it systematically solves the limitations of traditional methods in handling unique phenomena such as Tibetan contracted cases and overlapping syllables. For example, the constraint rules designed for the genitive case marker "འི" effectively avoid structural damage to its preceding root words—a technical detail that general algorithms cannot achieve.

[0070] The economic benefits and application prospects of the method provided by this invention: This invention provides an effective technical solution to address the bottleneck problem of low-resource languages ​​in natural language processing, offering broad economic benefits and application prospects. First, this method significantly reduces the cloud computing costs of training and inference for large Tibetan language models because it generates shorter token sequences, directly saving computing resources. Second, as a core text preprocessing module, this method can be widely applied to Tibetan machine translation, intelligent search, content generation, and educational software, greatly improving the accuracy and user experience of these applications and accelerating the industrialization of Tibetan language informatization. Finally, the multi-granularity, high-fidelity segmentation results provided by this invention offer key technical support for building a high-quality pre-training corpus for large Tibetan language models, which is of strategic significance for promoting the maturity and development of the Tibetan language artificial intelligence ecosystem.

[0071] When applying the Tibetan multi-granularity sub-word segmentation method provided in this manual, it is not necessary to follow the... Figure 1 The steps shown are executed in sequence. The specific execution order of each step can be determined as needed, and this manual does not impose any restrictions on it.

[0072] The above describes one or more embodiments of the Tibetan multi-granularity word segmentation method provided in this specification. Based on the same idea, this specification also provides a corresponding Tibetan multi-granularity word segmentation device, which includes: The acquisition module is used to acquire the Tibetan language model; the Tibetan language model includes a multi-level fused vocabulary and a word segmenter; the multi-level fused vocabulary includes a word-level vocabulary, a syllable-level vocabulary, a character-level vocabulary, and a character-level vocabulary; The word segmentation module is used to input the Tibetan text to be segmented into the large language model. The word segmenter performs initial word segmentation on the Tibetan text to be segmented, and obtains the initial word segmentation result. The initial word segmentation result includes multiple words. In the Tibetan large language model sub-word segmenter, multiple words are matched hierarchically with word-level vocabulary, and words that are matched successfully and words that are not matched successfully are obtained. Syllable boundary recognition is performed on the words that are not matched successfully to complete syllable segmentation. Syllable-level vocabulary matching verification is performed on the segmented syllables through syllable-level vocabulary, and words that are matched successfully and words that are not matched successfully are obtained. The unmatched syllables are structurally decomposed to obtain multiple characters. After the characters are verified by character-level vocabulary, words that are matched successfully and words that are not matched successfully are obtained. Character-level segmentation is performed on the unmatched characters through character-level vocabulary, and the segmented characters are obtained. The matched words, syllables, characters, and segmented characters are used as the Tibetan multi-granularity sub-word segmentation result.

[0073] Specific limitations regarding the Tibetan multi-granularity word segmentation device can be found in the limitations of the Tibetan multi-granularity word segmentation method described above, and will not be repeated here. Each module in the aforementioned Tibetan multi-granularity word segmentation device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device, or stored in the memory of a computer device as software, so that the processor can call and execute the corresponding operations of each module.

[0074] This specification also provides a computer-readable storage medium storing a computer program that can be used to execute the above-described... Figure 1 The provided method for multi-granularity word segmentation in Tibetan.

[0075] This instruction manual also provides Figure 4 The schematic diagram of the computer device shown is as follows: Figure 4 At the hardware level, the computer device includes a processor, internal bus, network interface, memory, and non-volatile memory, and may also include other hardware required for business operations. The processor reads the corresponding computer program from the non-volatile memory into memory and then runs it to achieve the above-mentioned functions. Figure 1 The provided method for multi-granularity word segmentation in Tibetan.

[0076] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, storage, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, or optical storage, etc. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM), etc.

[0077] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

Claims

1. A Tibetan multi-granularity word segmentation method, characterized in that, include: Obtain a Tibetan language large-scale model word segmenter; The Tibetan language large-scale model word segmenter includes a hierarchical dimensionality reduction strategy and a multi-level fused vocabulary; The multi-level fusion vocabulary includes word-level, syllable-level, character-level, and character-level vocabulary; the hierarchical dimensionality reduction strategy includes descending priority for words, syllables, characters, and words. The Tibetan text to be segmented is input into the Tibetan language large model word segmenter. The Tibetan language large model word segmenter performs initial word segmentation on the Tibetan text to be segmented, and obtains the initial word segmentation result; the initial word segmentation result includes multiple words. In the Tibetan language large-scale sub-word segmenter, multiple words are matched hierarchically with word-level vocabulary to obtain successfully matched words and unmatched words; syllable boundary recognition is performed on the unmatched words to complete syllable segmentation; The segmented syllables are matched and verified using a syllable-level vocabulary to obtain successfully matched syllables and unmatched syllables. The unmatched syllables are then structurally decomposed to obtain multiple characters, which are then verified using a character-level vocabulary to obtain successfully matched characters and unmatched characters. The unmatched characters are segmented using a character-level vocabulary to obtain the segmented characters. The successfully matched words, syllables, pinyin, and segmented characters are used as the Tibetan multi-granularity sub-word segmentation results.

2. The method as described in claim 1, characterized in that, The word-level vocabulary is the first level of the multi-level fused vocabulary. The construction process of the word-level vocabulary includes: Through quantitative analysis of a pre-set corpus and a frequency-driven screening mechanism, multiple high-frequency multisyllabic words are selected to obtain a word-level vocabulary.

3. The method as described in claim 1, characterized in that, The syllable-level vocabulary is the second level of the multi-level fused vocabulary. The construction process of the syllable-level vocabulary includes: By integrating the "Tibetan Standard Syllable Frequency Dictionary" with the Tibetan Standard Graded Syllable Database, a syllable-level word list is constructed.

4. The method as described in claim 1, characterized in that, The character-level vocabulary is the third level of the multi-level fused vocabulary, which includes multiple Tibetan character-level words. A character-level word is any form that can be divided in the linear arrangement of Tibetan visual symbols. The character-level vocabulary is the fourth level of the multi-level fusion vocabulary. It is a character-level vocabulary for Tibetan-Chinese-English mixed language families and includes multiple basic character units.

5. The method as described in claim 1, characterized in that, The method includes: If a compressed Tibetan character is detected during the process of segmenting Tibetan text, a constraint mechanism based on the "#" symbol is used to restore the complete structure of the Tibetan text.

6. The method as described in claim 1, characterized in that, The Tibetan text to be segmented is initially segmented to obtain the initial segmentation results, including: The initial segmentation results are obtained by using a dictionary to perform initial segmentation of the Tibetan text to be segmented.

7. A Tibetan multi-granularity word segmentation device, characterized in that, include: The acquisition module is used to acquire the Tibetan language large model word segmenter; The Tibetan language large-scale model word segmenter includes a hierarchical dimensionality reduction strategy and a multi-level fused vocabulary; The multi-level fusion vocabulary includes word-level, syllable-level, character-level, and character-level vocabulary; the hierarchical dimensionality reduction strategy includes descending priority for words, syllables, characters, and words. The word segmentation module is used to input the Tibetan text to be segmented into the Tibetan language large model word segmenter. The Tibetan language large model word segmenter performs initial word segmentation on the Tibetan text to be segmented, and obtains the initial word segmentation result, which includes multiple words. In the Tibetan language large model word segmenter, the multiple words are matched hierarchically with the word-level vocabulary, and the words that are matched successfully and those that are not matched successfully are obtained. Syllable boundary recognition is performed on the words that are not matched successfully to complete syllable segmentation. The segmented syllables are matched and verified using a syllable-level vocabulary to obtain successfully matched and unmatched syllables. The unmatched syllables are structurally decomposed to obtain multiple characters, which are then verified using a character-level vocabulary to obtain successfully matched and unmatched characters. The unmatched characters are then segmented at the character level using a character-level vocabulary to obtain segmented characters. The successfully matched words, syllables, characters, and segmented characters are used as the Tibetan multi-granularity sub-word segmentation results.

8. A computer-readable storage medium, characterized in that, The storage medium stores a computer program, which, when executed by a processor, implements the method described in any one of claims 1 to 6.

9. A computer device, characterized in that, It includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor, when executing the program, implements the method as described in any one of claims 1 to 6.

Images