A Tibetan machine translation system and a Tibetan text automatic segmentation method
By extracting the sentence representation sequence and keyword vectors of Tibetan text, calculating the transfer characteristics between words and paragraph boundary characteristics, the adaptive segmentation of Tibetan text paragraphs is realized, and the problem of accuracy of Tibetan text paragraph division is solved, and the improvement of Logical and semantic relationships between paragraphs.
Patent Information
- Application Number
- CN202411243510.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-09-05
- Publication Date
- 2024-12-03
- Estimated Expiration
- 2044-09-05
AI Technical Summary
The classification of paragraphs in Tibetan texts is difficult to accurately determine because there is a lack of clear indentation or blank line representation, which depends on changes in semantic logical structure, resulting in low accuracy of paragraph division.
By obtaining Tibetan text information, extracting sentence representation sequences, determining statement granularity and semantic theme clusters, extracting part-of-speech vectors of keywords, calculating inter-word transfer characteristics and paragraph boundary characteristics, and using adaptive segmentation methods to enhance semantic content and paragraphs logical structure association between.
The accuracy of the partitioning of Tibetan texts is improved, and by quantifying the transfer degree between words and semantic theme clusters, the logical and semantic connections between paragraphs are identified to ensure the consistency and consistency of text.
Smart Images

Figure CN118761389B_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the technical field of Tibetan machine translation, and more specifically, to a Tibetan machine translation system and a Tibetan text automatic segmentation method. Background Art
[0002] Tibetan machine translation is an important direction in natural language processing, aiming to achieve automatic translation between Tibetan and other languages (such as Chinese, English, etc.). The unique grammar and text characteristics of Tibetan make its machine translation somewhat challenging. In recent years, with the advancement of neural network machine translation (NMT) technology, Tibetan machine translation has made significant progress. Deep learning models such as long short-term memory networks (LSTM), convolutional neural networks (CNN) and attention-based Transformer models are used to capture the complex semantic and syntactic relationships between Tibetan and the target language. The application of pre-trained language models (such as BERT and GPT) in Tibetan translation has also improved the translation quality. The research on Tibetan machine translation involves corpus construction, vocabulary expansion, parallel corpus alignment, and training and optimization of translation models. The development of Tibetan machine translation technology has not only promoted the informatization application of Tibetan, but also provided important technical support for multilingual translation systems and promoted cultural exchanges and information sharing.
[0003] Automatic segmentation of Tibetan text usually starts with preliminary segmentation using spaces and punctuation marks, taking into account the phrase and compound word structures of Tibetan, using a vocabulary or rules to identify these structures, and then combining Tibetan grammatical rules and expression habits, such as verbs, nouns and other keywords to determine paragraph boundaries. Finally, machine learning technology is used to train the model to divide paragraphs based on a large number of segmented text samples. However, in Tibetan text, paragraph separation is usually not represented by clear indents or blank lines like in Western languages. Paragraph division relies more on semantic logical structural changes, such as topic switching or changes in content focus. Therefore, how to enhance the association between Tibetan semantic content and logical structural changes between paragraphs, thereby improving the accuracy of Tibetan text paragraph division has become a difficult problem faced by the industry. Summary of the invention
[0004] The present application provides a Tibetan machine translation system and a Tibetan text automatic segmentation method, which can enhance the association between the semantic content of Tibetan and the logical structure changes between paragraphs, thereby improving the accuracy of Tibetan text paragraph division.
[0005] In a first aspect, the present application provides a method for automatic segmentation of Tibetan text, comprising the following steps:
[0006] Acquire Tibetan text information, extract sentence representations of all Tibetan sentences in the Tibetan text information, and obtain multiple Tibetan sentence representation sequences;
[0007] Determining the sentence granularity of each Tibetan sentence in the Tibetan text segmentation process by the position span of each Tibetan character in the Tibetan text information in the context and the semantic dependency relationship between each Tibetan sentence representation sequence, and then subject-dividing all Tibetan sentences in the Tibetan text information according to each sentence granularity to obtain multiple semantic subject clusters;
[0008] Extracting part-of-speech vectors of different Tibetan keywords from the Tibetan text information, determining inter-word transfer features of each Tibetan keyword in each Tibetan sentence through inter-word relationship features of each part-of-speech vector, and then determining inter-word transfer degrees of different Tibetan keywords in the Tibetan text segmentation process according to all inter-word transfer features;
[0009] The paragraph boundary features of the Tibetan text information are determined by all inter-word transfer degrees and all semantic topic clusters, and then the Tibetan text information is adaptively segmented based on the paragraph boundary features.
[0010] In some embodiments, extracting sentence representations from all Tibetan sentences in the Tibetan text information to obtain multiple Tibetan sentence representation sequences specifically includes:
[0011] Converting all Tibetan sentences in the Tibetan text information into Tibetan sentence vector space;
[0012] Extracting a plurality of Tibetan sentence representation information from the Tibetan sentence vector space based on a pre-trained long short-term memory neural network;
[0013] The representation information of each Tibetan sentence is subjected to segmented pooling operation to obtain multiple Tibetan sentence representation sequences.
[0014] In some embodiments, determining the sentence granularity of each Tibetan sentence in the Tibetan text segmentation process by the position span of each Tibetan character in the Tibetan text information in the context and the semantic dependency relationship between each Tibetan sentence representation sequence specifically includes:
[0015] Determine the semantic differences of each Tibetan sentence based on all Tibetan sentence representation sequences;
[0016] Determine the semantic dependencies between the representation sequences of each Tibetan sentence;
[0017] The sentence granularity of each Tibetan sentence in the Tibetan text segmentation process is determined by the position span of each Tibetan character in the Tibetan text information in the context, all semantic dependencies and all semantic differences.
[0018] In some embodiments, all Tibetan sentences in the Tibetan text information are divided into topics according to the granularity of each sentence, and multiple semantic topic clusters are obtained, specifically including:
[0019] Perform linear fitting on all sentence granularities to obtain sentence granularity curve;
[0020] Determining a plurality of semantic topic centers according to the sentence granularity curve;
[0021] All Tibetan sentences in the Tibetan text information are divided according to all semantic topic centers and all sentence granularities to obtain multiple semantic topic clusters.
[0022] In some embodiments, extracting part-of-speech vectors of different Tibetan keywords from the Tibetan text information specifically includes:
[0023] Converting the Tibetan text information into a Tibetan text sequence;
[0024] Determining the part-of-speech co-occurrence degree of each Tibetan character in the Tibetan text information according to the Tibetan text sequence and a preset co-occurrence window;
[0025] The part-of-speech vectors of different Tibetan keywords are determined according to the part-of-speech co-occurrence of all Tibetan characters.
[0026] In some embodiments, determining the inter-word transition degree of different Tibetan keywords in the Tibetan text segmentation process according to all inter-word transition features specifically includes:
[0027] Determine the forward transfer degree and backward transfer degree of each Tibetan keyword based on all inter-word transfer features;
[0028] The inter-word transfer degrees of different Tibetan keywords in the Tibetan text segmentation process are determined through all forward transfer degrees and all backward transfer degrees.
[0029] In some embodiments, determining the paragraph boundary features of the Tibetan text information through all inter-word transfer degrees and all semantic topic clusters specifically includes:
[0030] Selecting a semantic topic cluster as a selected semantic topic cluster;
[0031] Determining multiple inter-segment topic change degrees of the selected semantic topic cluster according to all inter-word transfer degrees and each adjacent Tibetan sentence in the selected semantic topic cluster;
[0032] Continue to determine multiple inter-segment topic variation degrees of the remaining semantic topic clusters;
[0033] The paragraph boundary features of the Tibetan text information are determined according to all inter-segment topic variation degrees of each semantic topic cluster.
[0034] In a second aspect, the present application provides a Tibetan machine translation system, the Tibetan machine translation system comprising a Tibetan text automatic segmentation unit, the Tibetan text automatic segmentation unit comprising:
[0035] An acquisition module, used to acquire Tibetan text information, extract sentence representations of all Tibetan sentences in the Tibetan text information, and obtain multiple Tibetan sentence representation sequences;
[0036] A processing module, used to determine the sentence granularity of each Tibetan sentence in the Tibetan text segmentation process through the position span of each Tibetan character in the Tibetan text information in the context and the semantic dependency relationship between each Tibetan sentence representation sequence, and then divide all the Tibetan sentences in the Tibetan text information into topics according to each sentence granularity to obtain multiple semantic topic clusters;
[0037] The processing module is further used to extract part-of-speech vectors of different Tibetan keywords from the Tibetan text information, determine the inter-word transfer features of each Tibetan keyword in each Tibetan sentence through the inter-word relationship features of each part-of-speech vector, and then determine the inter-word transfer degrees of different Tibetan keywords in the Tibetan text segmentation process according to all the inter-word transfer features;
[0038] The execution module is used to determine the paragraph boundary features of the Tibetan text information through all the inter-word transfer degrees and all the semantic topic clusters, and then adaptively segment the Tibetan text information based on the paragraph boundary features.
[0039] In a third aspect, the present application provides a computer device, comprising a memory and a processor, wherein the memory stores codes, and the processor is configured to obtain the codes and execute the above-mentioned Tibetan text automatic segmentation method.
[0040] In a fourth aspect, the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the above-mentioned Tibetan text automatic segmentation method is implemented.
[0041] The technical solution provided by the embodiments disclosed in this application has the following beneficial effects:
[0042] In the Tibetan machine translation system and the Tibetan text automatic segmentation method provided by the present application, Tibetan text information is first obtained, sentence representations of all Tibetan sentences in the Tibetan text information are extracted to obtain multiple Tibetan sentence representation sequences; the sentence granularity of each Tibetan sentence in the Tibetan text segmentation process is determined by the position span of each Tibetan character in the Tibetan text information in the context and the semantic dependency relationship between each Tibetan sentence representation sequence, and then all Tibetan sentences in the Tibetan text information are subject-divided according to each sentence granularity to obtain multiple semantic topic clusters; part-of-speech vectors of different Tibetan keywords are extracted from the Tibetan text information, the inter-word transfer features of each Tibetan keyword in each Tibetan sentence are determined according to the inter-word relationship features of each part-of-speech vector, and then the inter-word transfer degrees of different Tibetan keywords in the Tibetan text segmentation process are determined according to all the inter-word transfer features; the paragraph boundary features of the Tibetan text information are determined according to all the inter-word transfer degrees and all the semantic topic clusters, and then the Tibetan text information is adaptively segmented based on the paragraph boundary features.
[0043] It can be seen that in the present application, the paragraph boundary features of the Tibetan text information can be determined based on all inter-word transfer degrees and all semantic topic clusters; wherein, firstly, the inter-word transfer features used to describe the transition relationship between the current Tibetan keyword in the Tibetan text information and the next same Tibetan keyword are obtained according to the inter-word relationship features of each part-of-speech vector. The inter-word transfer features can help understand the orderly connection and mutual dependence of Tibetan vocabulary in the Tibetan text information in the logical connection between paragraphs, and then all inter-word transfer features are quantified to obtain the inter-word transfer degrees that describe the transfer strength of different Tibetan keywords in the text segmentation process. The inter-word transfer degrees can be used to quantify the importance and frequency of Tibetan keywords when transferring between paragraphs, and help identify the logical connection and semantic connection between Tibetan keywords; secondly, the sentence granularity that measures the degree of refinement of the Tibetan sentence representation corresponding to each Tibetan sentence representation sequence is determined, which can be used as a reference when dividing different Tibetan sentences in the Tibetan text information, which helps to better capture the semantics between paragraphs. The content and logical relationship of the Tibetan text information are determined to ensure the coherence and consistency of the Tibetan text information, and then all the Tibetan sentences in the Tibetan text information are divided into topics according to the granularity of each sentence, and a semantic topic cluster composed of multiple Tibetan sentences with semantic relevance is obtained. Different semantic topic clusters reflect the semantic structure and content organization between different Tibetan sentences, and the same semantic topic cluster jointly expresses a specific theme or topic; then, all inter-word transfer degrees and all semantic topic clusters are quantified to obtain paragraph boundary features describing the Tibetan text information, and the paragraph boundary features represent feature indicators for determining the boundary points between paragraphs in the Tibetan text information, and measure the logical and semantic changes between paragraphs in the Tibetan text information, especially in terms of topic transfer and semantic connection between different paragraphs; finally, the Tibetan text information is adaptively segmented based on the paragraph boundary features; in summary, the scheme of the present application can enhance the association between the semantic content of Tibetan and the logical structure changes between paragraphs, thereby improving the accuracy of Tibetan text paragraph division. BRIEF DESCRIPTION OF THE DRAWINGS
[0044] Figure 1 is an exemplary flow chart of a Tibetan text automatic segmentation method according to some embodiments of the present application;
[0045] Figure 2 is a schematic diagram of a process for determining a Tibetan sentence representation sequence in some embodiments of the present application;
[0046] Figure 3 is a schematic diagram of a process for determining paragraph boundary features in some embodiments of the present application;
[0047] Figure 4 is a schematic diagram of exemplary hardware and / or software of a Tibetan text automatic segmentation unit according to some embodiments of the present application;
[0048] Figure 5 It is a structural schematic diagram of a computer device for implementing a Tibetan text automatic segmentation method according to some embodiments of the present application. DETAILED DESCRIPTION
[0049] The core of this application is to first obtain Tibetan text information, extract sentence representations of all Tibetan sentences in the Tibetan text information, and obtain multiple Tibetan sentence representation sequences; determine the sentence granularity of each Tibetan sentence in the Tibetan text segmentation process through the position span of each Tibetan character in the Tibetan text information in the context and the semantic dependency between each Tibetan sentence representation sequence, and then divide all Tibetan sentences in the Tibetan text information into topics according to each sentence granularity to obtain multiple semantic topic clusters; extract the part-of-speech vectors of different Tibetan keywords from the Tibetan text information, determine the inter-word transfer features of each Tibetan keyword in each Tibetan sentence through the inter-word relationship features of each part-of-speech vector, and then determine the inter-word transfer degrees of different Tibetan keywords in the Tibetan text segmentation process according to all inter-word transfer features; determine the paragraph boundary features of the Tibetan text information through all inter-word transfer degrees and all semantic topic clusters, and then adaptively segment the Tibetan text information based on the paragraph boundary features. The scheme of this application can enhance the association between the semantic content of Tibetan and the logical structure changes between paragraphs, thereby improving the accuracy of Tibetan text paragraph division.
[0050] In order to better understand the above technical solution, the above technical solution will be described in detail below in conjunction with the accompanying drawings and specific implementation methods. Figure 1 , which is an exemplary flow chart of a Tibetan text automatic segmentation method according to some embodiments of the present application. The Tibetan text automatic segmentation method 100 mainly includes the following steps:
[0051] In step 101, Tibetan text information is obtained, and sentence representations are extracted for all Tibetan sentences in the Tibetan text information to obtain a plurality of Tibetan sentence representation sequences.
[0052] In specific implementation, for example, after starting the Tibetan machine translation system, Tibetan text information can be obtained from the database of the Tibetan machine translation system. It should be noted that the Tibetan text information described in the present application is composed of multiple Tibetan characters, and these Tibetan characters include basic Tibetan alphabetic characters, Tibetan consonant characters, Tibetan vowel characters, Tibetan diacritical marks and Tibetan punctuation marks.
[0053] In some embodiments, reference Figure 2As shown, this figure is a schematic diagram of the process of determining the Tibetan sentence representation sequence in some embodiments of the present application. In this embodiment, sentence representations of all Tibetan sentences in the Tibetan text information are extracted to obtain multiple Tibetan sentence representation sequences, which can be achieved by the following steps:
[0054] First, in step 1011, all Tibetan characters in the Tibetan text information are converted into a Tibetan sentence vector space;
[0055] Secondly, in step 1012, a plurality of Tibetan sentence representation information is extracted from the Tibetan sentence vector space based on the pre-trained long short-term memory neural network;
[0056] Then, in step 1013, a segmented pooling operation is performed on each Tibetan sentence representation information to obtain a plurality of Tibetan sentence representation sequences.
[0057] In specific implementation, converting all Tibetan characters in the Tibetan text information into Tibetan sentence vector space can be implemented in the following manner, namely: first, using the word segmenter in the TibetanNLP library to divide all Tibetan characters in the Tibetan text information into multiple different Tibetan sentences, wherein there is a sequence between different Tibetan sentences, and then, using a pre-trained word vector model (such as Word2Vec, GloVe or FastText, which is not limited here) to convert each Tibetan character in each Tibetan sentence into a word vector, and taking the set composed of all word vectors as the Tibetan sentence vector of the corresponding Tibetan sentence, and finally, taking the set composed of all Tibetan sentence vectors as the Tibetan sentence vector space, and in other embodiments, other methods can also be used for implementation, which will not be repeated here.
[0058] It should be noted that the pre-trained long short-term memory neural network in this application refers to a long short-term memory neural network model that has been pre-trained on a large-scale data set. This model is usually trained for a long time on a large amount of text data to learn the grammatical structure, semantic information and contextual relationships of the language.
[0059] In a specific implementation, extracting multiple Tibetan sentence representation information from the Tibetan sentence vector space based on a pre-trained long short-term memory neural network can be implemented in the following manner, namely: using a pre-trained long short-term memory neural network to perform feature extraction on each Tibetan sentence vector in the Tibetan sentence vector space to obtain Tibetan sentence representation information of each Tibetan sentence vector. In other embodiments, other methods can also be used for implementation, which is not limited here.
[0060] It should be noted that the Tibetan sentence representation information described in the present application represents a vector of semantic features and grammatical features of a Tibetan sentence. The Tibetan sentence representation information captures the relationship and contextual information between Tibetan characters in a Tibetan sentence, thereby converting complex natural language text into a form that can be understood and processed by a computer, wherein each Tibetan sentence representation information corresponds to a Tibetan sentence.
[0061] In specific implementation, the segmented pooling operation is performed on each Tibetan sentence representation information to obtain multiple Tibetan sentence representation sequences, which can be implemented in the following manner, namely: select a Tibetan sentence representation information as the selected Tibetan sentence representation information, use the square of the total number of Tibetan characters in the Tibetan sentence corresponding to the selected Tibetan sentence representation information as the number of divided segments, use the number of divided segments to evenly divide the Tibetan sentence representation information into multiple Tibetan sentence representation segments, then perform a maximum pooling operation on each Tibetan sentence representation segment, use all values obtained after the maximum pooling operation as Tibetan sentence representation values, and then sort all Tibetan sentence representation values according to their positions in the selected Tibetan sentence representation information, and use the sorted sequence as the Tibetan sentence representation sequence of the selected Tibetan sentence representation information, and continue to determine the Tibetan sentence representation sequences of the remaining Tibetan sentence representation information, so as to obtain multiple Tibetan sentence representation sequences. In other embodiments, other methods can also be used for implementation, which will not be repeated here.
[0062] It should be noted that the Tibetan sentence representation sequence described in the present application represents an ordered sequence composed of multiple Tibetan sentence representation values, wherein the Tibetan sentence representation values in the Tibetan sentence representation sequence represent the parameter values of the most significant logical and semantic relationships between the Tibetan characters in the Tibetan sentence, and each Tibetan sentence representation sequence corresponds to a Tibetan sentence, that is, the Tibetan sentence representation sequence has an arrangement order in the entire Tibetan text information.
[0063] In step 102, the sentence granularity of each Tibetan sentence in the Tibetan text segmentation process is determined by the position span of each Tibetan character in the Tibetan text information in the context and the semantic dependency relationship between each Tibetan sentence representation sequence, and then all the Tibetan sentences in the Tibetan text information are divided into topics according to each sentence granularity to obtain multiple semantic topic clusters.
[0064] In some embodiments, determining the sentence granularity of each Tibetan sentence in the Tibetan text segmentation process by the position span of each Tibetan character in the Tibetan text information in the context and the semantic dependency between each Tibetan sentence representation sequence can be implemented by the following steps, namely:
[0065] Determine the semantic differences of each Tibetan sentence based on all Tibetan sentence representation sequences;
[0066] Determine the semantic dependencies between the representation sequences of each Tibetan sentence;
[0067] The sentence granularity of each Tibetan sentence in the Tibetan text segmentation process is determined by the position span of each Tibetan character in the Tibetan text information in the context, all semantic dependencies and all semantic differences.
[0068] In a specific implementation, determining the semantic difference of each Tibetan sentence according to all Tibetan sentence representation sequences can be implemented in the following manner, namely: selecting a Tibetan sentence as a selected Tibetan sentence, then selecting two Tibetan sentences adjacent to the selected Tibetan sentence, and taking the two adjacent Tibetan sentences as adjacent Tibetan sentences, and then calculating the cosine similarity between the Tibetan sentence representation sequence corresponding to the selected Tibetan sentence and the Tibetan sentence representation sequences corresponding to the two adjacent Tibetan sentences, and taking the minimum cosine similarity as the semantic difference of the selected Tibetan sentence, and continuing to determine the semantic differences of the remaining Tibetan sentences. In other embodiments, other methods can also be used for implementation, which are not limited here.
[0069] It should be noted that in the present application, there is only one adjacent Tibetan sentence between the first Tibetan sentence and the last Tibetan sentence in the Tibetan text information, which are the second Tibetan sentence and the penultimate Tibetan sentence respectively. When calculating the semantic difference, it is only necessary to calculate the cosine similarity between the adjacent Tibetan sentences, and use the obtained cosine similarity as the semantic difference of the first or last Tibetan sentence.
[0070] In specific implementation, determining the semantic dependency relationship between each Tibetan sentence representation sequence can be implemented in the following manner, namely: selecting a Tibetan sentence representation sequence as a selected Tibetan sentence representation sequence, calculating the Euclidean distance between the selected Tibetan sentence representation sequence and the remaining Tibetan sentence representation sequences, taking all Tibetan sentence representation sequences as nodes, and at the same time, taking the selected Tibetan sentence representation sequence as a starting node, and taking the Tibetan sentence representation sequence with the largest Euclidean distance from the selected Tibetan sentence representation sequence as a terminating node, and at the same time, taking the Euclidean distance between the selected Tibetan sentence representation sequence and the remaining Tibetan sentence representation sequences as edge weights, thereby constructing a graph structure according to the weights of all nodes and edges, and then using the Dijkstra algorithm to calculate the shortest path between the starting node and the terminating node, and then summing the weights of all edges on the shortest path, and taking the summed value as the semantic dependency relationship between the selected Tibetan sentence representation sequence and other Tibetan sentence representation sequences, and continuing to determine the semantic dependency relationship between the remaining Tibetan sentence representation sequences and other Tibetan sentence representation sequences. In other embodiments, other methods can also be used for implementation, which are not limited here.
[0071] It should be noted that the semantic dependency relationship described in the present application represents the dependency relationship between Tibetan sentences in Tibetan text information in terms of semantic content. The semantic dependency relationship reflects the information flow between Tibetan sentences in Tibetan text information. By analyzing the dependency relationship, the overall content of the Tibetan text information composed of various Tibetan sentences can be understood, and key elements such as themes, arguments and conclusions can be identified. Among them, each Tibetan sentence representation sequence corresponds to a semantic dependency relationship.
[0072] In addition, it should be noted that the position span described in the present application represents the distance between the first and last appearance positions of a Tibetan character in Tibetan text information, wherein each Tibetan character occupies one position.
[0073] In a specific implementation, the sentence granularity of each Tibetan sentence in the Tibetan text segmentation process is determined by the position span of each Tibetan character in the context, all semantic dependencies and all semantic differences in the Tibetan text information, which can be implemented in the following manner, namely: first, a Tibetan sentence representation sequence is selected as a selected Tibetan sentence representation sequence, the position spans of each Tibetan character in the Tibetan sentence corresponding to the selected Tibetan sentence representation sequence are added, and the added value is used as the sentence character span value of the selected Tibetan sentence representation sequence; secondly, the semantic difference of the Tibetan sentence corresponding to the selected Tibetan sentence representation sequence is divided by the sentence character span value, and the sentence character span value is used as the sentence character span value of the selected Tibetan sentence representation sequence; The value obtained by division is used as the semantic difference span of the selected Tibetan sentence representation sequence. Then, the reciprocal of the exponential function with the natural logarithm as the base is taken for the semantic dependency relationship corresponding to the selected Tibetan sentence representation sequence, and the value obtained by taking the reciprocal of the exponential function with the natural logarithm as the base is used as the semantic dependency adjustment coefficient. The semantic dependency adjustment coefficient is multiplied by the semantic difference span, and the multiplied value is used as the sentence granularity of the Tibetan sentence corresponding to the selected Tibetan sentence representation sequence in the Tibetan text segmentation process. The sentence granularity of the Tibetan sentence corresponding to the remaining Tibetan sentence representation sequences in the Tibetan text segmentation process is continued to be determined. In other embodiments, other methods can also be used to achieve this, which will not be repeated here.
[0074] It should be noted that the sentence granularity in the present application represents the degree of refinement of the representation of the Tibetan sentence corresponding to the Tibetan sentence representation sequence, which can be used as a reference for dividing different Tibetan sentences in Tibetan text information. The sentence granularity can better capture the semantic content and logical relationship between paragraphs, and ensure the coherence and consistency of the Tibetan text information. Each sentence granularity corresponds to a Tibetan sentence representation sequence, and each Tibetan sentence representation sequence corresponds to a Tibetan sentence, that is, each sentence granularity corresponds to a Tibetan sentence.
[0075] In some embodiments, all Tibetan sentences in the Tibetan text information are divided into topics according to the granularity of each sentence to obtain multiple semantic topic clusters, which can be achieved by the following steps:
[0076] Perform linear fitting on all sentence granularities to obtain sentence granularity curve;
[0077] Determining a plurality of semantic topic centers according to the sentence granularity curve;
[0078] All Tibetan sentences in the Tibetan text information are divided according to all semantic topic centers and all sentence granularities to obtain multiple semantic topic clusters.
[0079] In the specific implementation, linear fitting is performed on all statement granularities to obtain a statement granularity curve, which can be achieved in the following manner, namely: all statement granularities are sorted in ascending order, and the sorted sequence is used as the statement granularity sequence, the statement granularity sequence is fit by the least squares support vector machine algorithm in the prior art, and the fitted sequence is used as the statement granularity curve. Other fitting algorithms can also be used in other embodiments, which are not limited here.
[0080] In specific implementation, determining multiple semantic topic centers according to the sentence granularity curve can be achieved in the following manner, namely: obtaining the derivative of the position corresponding to each sentence granularity on the sentence granularity curve, and taking each derivative as the sentence granularity derivative, wherein each sentence granularity derivative corresponds to a sentence granularity, and then clustering all the sentence granularity derivatives using a hierarchical clustering algorithm to obtain multiple cluster centers, wherein each cluster center corresponds to a sentence granularity, and taking the sentence granularity corresponding to each cluster center as the semantic topic center, thereby obtaining multiple semantic topic centers. Other methods can also be used in other embodiments, which will not be repeated here.
[0081] It should be noted that the semantic topic center described in this application represents the key points representing the semantic content in the Tibetan text information. Each semantic topic center corresponds to a specific semantic topic, and the semantic topic center reflects the core ideas and main contents of different paragraphs in the Tibetan text information.
[0082] In specific implementation, all Tibetan sentences in the Tibetan text information are divided by all semantic topic centers and all sentence granularities to obtain multiple semantic topic clusters, which can be implemented in the following manner: first, each semantic topic center is used as the initial cluster center of the K-means clustering algorithm, and all sentence granularities are clustered using K-means clustering to obtain multiple different clusters, and the obtained clusters are all used as sentence granularity clusters, and then, a set of Tibetan sentences corresponding to all sentence granularities in the same sentence granularity cluster is used as a semantic topic cluster, thereby obtaining multiple semantic topic clusters. In other embodiments, other methods can also be used for implementation, which are not limited here.
[0083] It should be noted that the semantic topic cluster in the present application represents a set of Tibetan sentences with semantic relevance in the Tibetan text information. The Tibetan sentences in the semantic topic cluster jointly express a specific theme or topic, reflecting the semantic structure and content organization of the Tibetan text information. In addition, it should be noted that the Tibetan sentences in the semantic topic cluster in the present application have a sequence, that is, each Tibetan sentence has a relative position in the Tibetan text information.
[0084] In step 103, part-of-speech vectors of different Tibetan keywords are extracted from the Tibetan text information, and inter-word transfer features of each Tibetan keyword in each Tibetan sentence are determined based on the inter-word relationship features of each part-of-speech vector, and then the inter-word transfer degrees of different Tibetan keywords in the Tibetan text segmentation process are determined based on all the inter-word transfer features.
[0085] In some embodiments, extracting part-of-speech vectors of different Tibetan keywords from the Tibetan text information may be achieved by using the following steps:
[0086] Converting the Tibetan text information into a Tibetan text sequence;
[0087] Determining the part-of-speech co-occurrence degree of each Tibetan character in the Tibetan text information according to the Tibetan text sequence and a preset co-occurrence window;
[0088] The part-of-speech vectors of different Tibetan keywords are determined according to the part-of-speech co-occurrence of all Tibetan characters.
[0089] In specific implementation, the Tibetan text information can be converted into a Tibetan text sequence by the following method: converting the Tibetan text information into a sequence according to the order of each Tibetan character in the Tibetan text information, and using the obtained sequence as the Tibetan text sequence.
[0090] It should be noted that the co-occurrence window described in the present application can be set using a fixed-length sequence, wherein the fixed length can be set using the value obtained by rounding down the square root of the total number of Tibetan characters in the Tibetan text sequence. In addition, the value obtained by rounding down the square root of the total number of Tibetan characters in the Tibetan text sequence is used as the keyword frequency.
[0091] In specific implementation, the part-of-speech co-occurrence degree of each Tibetan character in the Tibetan text information is determined according to the Tibetan text sequence and the preset co-occurrence window, which can be implemented in the following manner, namely: first, all Tibetan characters in the Tibetan text sequence are converted into vectors using the Word2Vec word embedding model, and the obtained vectors are used as Tibetan character vectors, wherein each Tibetan character corresponds to a Tibetan character vector; secondly, the co-occurrence window is aligned with the first Tibetan character in the Tibetan text sequence, and the mutual trust between the Tibetan character vector corresponding to the first Tibetan character and the Tibetan character vector corresponding to the Tibetan character covered by the co-occurrence window is calculated. information, and using the maximum mutual information as the part-of-speech co-occurrence degree of the first Tibetan character in the Tibetan text information, sliding the co-occurrence window, aligning the co-occurrence window with the second Tibetan character in the Tibetan text sequence, calculating the mutual information between the Tibetan character vector corresponding to the second Tibetan character and the Tibetan character vector corresponding to the Tibetan character covered by the co-occurrence window, and using the maximum mutual information as the part-of-speech co-occurrence degree of the first Tibetan character in the Tibetan text information, repeating the above steps, and sequentially determining the part-of-speech co-occurrence degrees of the remaining Tibetan characters in the Tibetan text information. In other embodiments, other methods may also be used to achieve this, which will not be repeated here.
[0092] It should be noted that the part-of-speech co-occurrence degree described in this application represents the probability of Tibetan characters with different part-of-speech tags co-occurring locally in Tibetan text information, wherein each part-of-speech co-occurrence degree corresponds to a Tibetan character, and each Tibetan character corresponds to a Tibetan character vector.
[0093] In specific implementation, determining the part-of-speech vectors of different Tibetan keywords based on the part-of-speech co-occurrence of all Tibetan characters can be implemented in the following manner, namely: sorting the part-of-speech co-occurrence of all Tibetan characters in descending order, using the obtained sequence as the part-of-speech co-occurrence sequence, sequentially selecting part-of-speech co-occurrences with the same number as the keyword frequency value from the part-of-speech co-occurrence sequence, and using the Tibetan characters corresponding to each selected part-of-speech co-occurrence as the Tibetan keywords, and then using the Tibetan character vectors corresponding to each Tibetan keyword as the part-of-speech vector, thereby obtaining the part-of-speech vectors of different Tibetan keywords. In other embodiments, other methods can also be used for implementation, which are not limited here.
[0094] It should be noted that the part-of-speech vector described in the present application represents the part-of-speech information of Tibetan characters encoded in a vector form. The part-of-speech vector not only contains the meaning information of the word, but also contains its functional information in the grammatical structure. In addition, it should be noted that the Tibetan keywords in the present application have a sequence in the entire Tibetan text information.
[0095] In some embodiments, determining the inter-word transfer features of each Tibetan keyword in each Tibetan sentence through the inter-word relationship features of each part-of-speech vector can be implemented by the following steps:
[0096] Determine the inter-word relationship characteristics of each part-of-speech vector;
[0097] Selecting a Tibetan sentence as the selected Tibetan sentence;
[0098] Determine the number of subsequent occurrences of each Tibetan keyword in the selected Tibetan sentence;
[0099] Determining inter-word transfer features of each Tibetan keyword in the selected Tibetan sentence according to the subsequent occurrence number of each Tibetan keyword in the selected Tibetan sentence and the inter-word relationship features of the part-of-speech vectors corresponding to each Tibetan keyword;
[0100] Continue to determine the inter-word transfer features of each Tibetan keyword in the remaining Tibetan sentences.
[0101] In specific implementation, determining the inter-word relationship features of each part-of-speech vector can be achieved in the following manner, namely: selecting a part-of-speech vector as the selected part-of-speech vector, calculating the Pearson correlation coefficient between the selected part-of-speech vector and all remaining part-of-speech vectors, and taking the largest Pearson correlation coefficient as the maximum Pearson correlation coefficient, then calculating the total number of Tibetan keywords corresponding to the selected part-of-speech vector in the entire Tibetan text information, and taking the obtained total number as the total number of Tibetan keywords, then multiplying the maximum Pearson correlation coefficient by the total number of Tibetan keywords, and taking the multiplied value as the inter-word relationship feature of the selected part-of-speech vector, and continuing to determine the inter-word relationship features of the remaining part-of-speech vectors. In other embodiments, other methods can also be used for implementation, which are not limited here.
[0102] It should be noted that the inter-word relationship feature described in this application represents the degree of linear association between the Tibetan keyword and other Tibetan keywords in the Tibetan text information. The larger the inter-word relationship feature, the higher the degree of linear association between the Tibetan keyword and other Tibetan keywords in the Tibetan text information, and vice versa.
[0103] In specific implementation, determining the subsequent occurrence number of each Tibetan keyword in the selected Tibetan sentence can be implemented in the following manner, namely: calculating the total number of subsequent occurrences of each Tibetan keyword in the selected Tibetan sentence in the Tibetan text information, and using the total number of subsequent occurrences of each Tibetan keyword as the subsequent occurrence number of the corresponding Tibetan keyword. In other embodiments, other methods can also be used for implementation, which will not be repeated here.
[0104] It should be noted that the inter-word transfer feature described in the present application represents the transfer relationship represented by the transition from the current Tibetan keyword to the next same Tibetan keyword in the Tibetan text information. The inter-word transfer feature can help understand the orderly connection and mutual dependence of Tibetan vocabulary in the Tibetan text information in the logical connection between paragraphs. As a preferred embodiment, the inter-word transfer feature of each Tibetan keyword in the selected Tibetan sentence is determined according to the subsequent occurrence number of each Tibetan keyword in the selected Tibetan sentence and the inter-word relationship feature of the part-of-speech vector corresponding to each Tibetan keyword. It can be implemented in the following way, that is, a Tibetan keyword is selected from the selected Tibetan sentence as the selected Tibetan keyword, the subsequent occurrence number of the selected Tibetan keyword and the inter-word relationship feature of the part-of-speech vector corresponding to the selected Tibetan keyword are multiplied, and the multiplied value is used as the inter-word transfer feature of the selected Tibetan keyword, and the inter-word transfer features of the remaining Tibetan keywords in the selected Tibetan sentence are continued to be determined. In other embodiments, other methods can also be used to implement, which will not be repeated here.
[0105] In some embodiments, determining the inter-word transition degree of different Tibetan keywords in the Tibetan text segmentation process according to all inter-word transition features can be implemented by the following steps:
[0106] Determine the forward transfer degree and backward transfer degree of each Tibetan keyword based on all inter-word transfer features;
[0107] The inter-word transfer degrees of different Tibetan keywords in the Tibetan text segmentation process are determined through all forward transfer degrees and all backward transfer degrees.
[0108] In specific implementation, the forward transfer degree and backward transfer degree of each Tibetan keyword are determined according to all the inter-word transfer features, which can be implemented in the following manner, namely: a Tibetan keyword is selected from all the Tibetan keywords as the selected Tibetan keyword, the inter-word transfer features of all the Tibetan keywords before the selected Tibetan keyword are added, and the added value is used as the forward transfer degree of the selected Tibetan keyword, the inter-word transfer features of all the Tibetan keywords after the selected Tibetan keyword are added, and the added value is used as the backward transfer degree of the selected Tibetan keyword, and then the forward transfer degree is divided by the backward transfer degree, and the divided value is used as the inter-word transfer degree of the selected Tibetan keyword in the Tibetan text segmentation process, and the inter-word transfer degrees of the remaining Tibetan keywords in the Tibetan text segmentation process are continuously determined. In other embodiments, other methods can also be used for implementation, which are not limited here.
[0109] In addition, it should be noted that the forward transfer degree described in the present application represents the semantic and logical connection strength between the Tibetan keyword and all previous Tibetan keywords in the Tibetan text information, and the backward transfer degree represents the semantic and logical connection strength between the Tibetan keyword and all subsequent Tibetan keywords in the Tibetan text information. In addition, each Tibetan keyword corresponds to a forward transfer degree and a backward transfer degree.
[0110] It should be noted that the inter-word transfer degree in the present application represents the transfer strength of different Tibetan keywords in the text segmentation process. The inter-word transfer degree can be used to quantify the importance and frequency of Tibetan keywords when they are transferred between paragraphs, and help identify the logical connection and semantic connection between Tibetan keywords. As a preferred embodiment, determining the inter-word transfer degrees of different Tibetan keywords in the Tibetan text segmentation process through all forward transfer degrees and all backward transfer degrees can be implemented in the following manner, namely: selecting a Tibetan keyword from all Tibetan keywords as the selected Tibetan keyword, subtracting the backward transfer degree of the selected Tibetan keyword from the forward transfer degree of the selected Tibetan keyword, and then dividing the subtracted value by the backward transfer degree of the selected Tibetan keyword, and using the divided value as the inter-word transfer degree of the selected Tibetan keyword in the Tibetan text segmentation process, and continuing to determine the inter-word transfer degrees of the remaining Tibetan keywords in the Tibetan text segmentation process. In other embodiments, other methods can also be used, which are not limited here.
[0111] In addition, it should be noted that in the present application, the inter-word transition degree of the first Tibetan keyword and the inter-word transition degree of the last Tibetan keyword are both set to 1. By setting the inter-word transition degree of the first Tibetan keyword and the inter-word transition degree of the last Tibetan keyword to 1, the boundary conditions can be normalized to ensure that there are clear start and end marks when processing Tibetan text information, which helps prevent the inter-word transition degree of the boundary Tibetan keywords from being underestimated or ignored.
[0112] In step 104, the paragraph boundary features of the Tibetan text information are determined through all the inter-word transfer degrees and all the semantic topic clusters, and then the Tibetan text information is adaptively segmented based on the paragraph boundary features.
[0113] In some embodiments, reference Figure 3 As shown, this figure is a schematic diagram of the process of determining paragraph boundary features in some embodiments of the present application. In this embodiment, the paragraph boundary features of the Tibetan text information are determined by all inter-word transfer degrees and all semantic topic clusters, which can be achieved by the following steps:
[0114] First, in step 1041, a semantic topic cluster is selected as a selected semantic topic cluster;
[0115] Next, in step 1042, multiple inter-segment topic change degrees of the selected semantic topic cluster are determined based on all inter-word transfer degrees and each adjacent Tibetan sentence in the selected semantic topic cluster;
[0116] Then, in step 1043, continue to determine multiple inter-segment topic change degrees of the remaining semantic topic clusters;
[0117] Finally, in step 1044, the paragraph boundary features of the Tibetan text information are determined according to all inter-segment topic variation degrees of each semantic topic cluster.
[0118] In specific implementation, determining multiple inter-segment topic change degrees of the selected semantic topic cluster according to all inter-word transition degrees and each adjacent Tibetan sentence in the selected semantic topic cluster can be implemented in the following manner, namely: first, calculating the mean of all inter-word transition degrees, and using the obtained mean as the inter-word transition balance degree; second, selecting a Tibetan sentence from the selected semantic topic cluster as the selected Tibetan sentence, summing the inter-word transition degrees of each Tibetan keyword in the selected Tibetan sentence, and using the summed value as the inter-word transition transition degree; then summing the inter-word transition degrees of each Tibetan keyword in the next adjacent Tibetan sentence of the selected Tibetan sentence in the Tibetan text information, and using the summed value as the adjacent inter-word transition degree; further subtracting the adjacent inter-word transition degree from the inter-word transition transition degree, and then dividing the subtracted value by the inter-word transition balance degree, and using the divided value as an inter-segment topic change degree of the selected semantic topic cluster, and continuing to determine the remaining inter-segment topic change degrees in the selected semantic topic cluster; other methods can also be used in other embodiments, which will not be repeated here.
[0119] It should be noted that the inter-segment topic variation degree described in the present application represents the degree of topic variation between adjacent Tibetan sentences in a semantic topic cluster. The greater the inter-segment topic variation degree, the greater the degree of topic variation between adjacent Tibetan sentences in the semantic topic cluster, and vice versa. In addition, it should be noted that each inter-segment topic variation degree in the present application corresponds to a Tibetan sentence.
[0120] In specific implementation, determining the paragraph boundary feature of the Tibetan text information based on all inter-segment topic variation degrees of each semantic topic cluster can be achieved in the following manner, namely: selecting the largest inter-segment topic variation degree and the smallest inter-segment topic variation degree from all inter-segment topic variation degrees of each semantic topic cluster, and sorting all the largest inter-segment topic variation degrees and all the smallest inter-segment topic variation degrees according to the order in which the corresponding Tibetan sentences appear in the Tibetan text information, and using the sorted sequence as the paragraph boundary feature of the Tibetan text information. In other embodiments, other methods can also be used for implementation, which is not limited here.
[0121] It should be noted that the paragraph boundary features described in this application represent characteristic indicators for determining the boundary points between paragraphs in Tibetan text information, and measure the logical and semantic changes between paragraphs in Tibetan text information, especially in terms of topic transfer and semantic connection between different paragraphs.
[0122] In a specific implementation, the adaptive segmentation of the Tibetan text information based on the paragraph boundary feature can be implemented in the following manner, namely: selecting a first inter-segment topic change degree from the paragraph boundary feature, and using a paragraph symbol " " after the Tibetan sentence corresponding to the first inter-segment topic change degree appears in the Tibetan text information. ", then select a second inter-segment topic change degree from the paragraph boundary feature, and use a paragraph symbol " after the Tibetan sentence corresponding to the second inter-segment topic change degree appears in the Tibetan text information " is used to mark the remaining Tibetan sentences corresponding to the subject variation degree in the paragraph boundary feature after the position where they appear in the Tibetan text information using the paragraph symbol " " is marked to complete the adaptive segmentation of the entire Tibetan text information. Other methods may also be used in other embodiments, which are not limited here.
[0123] It should be noted that, in the present application, adaptive segmentation can intelligently adjust the segmentation position according to the changes in the text content and semantic features to maintain the coherence and logic of the Tibetan text information.
[0124] In addition, in another aspect of the present application, in some embodiments, the present application provides a Tibetan machine translation system, the Tibetan machine translation system includes a Tibetan text automatic segmentation unit, referring to Figure 4 , which is a schematic diagram of exemplary hardware and / or software of a Tibetan text automatic segmentation unit according to some embodiments of the present application. The Tibetan text automatic segmentation unit 400 includes: an acquisition module 401, a processing module 402 and an execution module 403, which are described as follows:
[0125] Acquisition module 401, in this application, acquisition module 401 is mainly used to acquire Tibetan text information, extract sentence representations of all Tibetan sentences in the Tibetan text information, and obtain multiple Tibetan sentence representation sequences;
[0126] Processing module 402, in the present application, is used to determine the sentence granularity of each Tibetan sentence in the Tibetan text segmentation process through the position span of each Tibetan character in the Tibetan text information in the context and the semantic dependency relationship between each Tibetan sentence representation sequence, and then divide all Tibetan sentences in the Tibetan text information into topics according to each sentence granularity to obtain multiple semantic topic clusters;
[0127] It should be noted that the processing module 402 in the present application is also used to extract part-of-speech vectors of different Tibetan keywords from the Tibetan text information, determine the inter-word transfer features of each Tibetan keyword in each Tibetan sentence through the inter-word relationship features of each part-of-speech vector, and then determine the inter-word transfer degree of different Tibetan keywords in the Tibetan text segmentation process according to all the inter-word transfer features;
[0128] Execution module 403, in the present application, execution module 403 is mainly used to determine the paragraph boundary features of the Tibetan text information through all inter-word transfer degrees and all semantic topic clusters, and then adaptively segment the Tibetan text information based on the paragraph boundary features.
[0129] In addition, the present application also provides a computer device, which includes a memory and a processor, wherein the memory stores codes, and the processor is configured to obtain the codes and execute the above-mentioned Tibetan text automatic segmentation method.
[0130] In some embodiments, reference Figure 5 , which is a schematic diagram of the structure of a computer device using the automatic segmentation method of Tibetan text according to some embodiments of the present application. The automatic segmentation method of Tibetan text in the above embodiment can be Figure 5 The computer device 500 shown in the figure is implemented, and the computer device 500 includes at least one processor 501, a communication bus 502, a memory 503 and at least one communication interface 504.
[0131] The processor 501 may be a general-purpose central processing unit (CPU), an application-specific integrated circuit (ASIC), or one or more processors for controlling the execution of the Tibetan text automatic segmentation method in the present application.
[0132] The communication bus 502 may be used to transmit information between the above-mentioned components.
[0133] The memory 503 may be a read-only memory (ROM) or other types of static storage devices that can store static information and instructions, a random access memory (RAM) or other types of dynamic storage devices that can store information and instructions, or an electrically erasable programmable read-only memory (EEPROM), a compact disc read-only memory (CD-ROM) or other optical disc storage, an optical disc storage (including a compressed optical disc, a laser disc, an optical disc, a digital versatile disc, a Blu-ray disc, etc.), a magnetic disk or other magnetic storage device, or any other medium that can be used to carry or store the desired program code in the form of an instruction or data structure and can be accessed by a computer, but is not limited thereto. The memory 503 may exist independently and be connected to the processor 501 via the communication bus 502. The memory 503 may also be integrated with the processor 501.
[0134] The memory 503 is used to store the program code for executing the solution of the present application, and the execution is controlled by the processor 501. The processor 501 is used to execute the program code stored in the memory 503. The program code may include one or more software modules. The method described in the above method embodiment can be implemented by the processor 501 and one or more software modules in the program code in the memory 503.
[0135] The communication interface 504 uses any transceiver or other device for communicating with other devices or communication networks, such as Ethernet, radio access network (RAN), wireless local area networks (WLAN), etc.
[0136] In a specific implementation, as an embodiment, a computer device may include multiple processors, each of which may be a single-CPU processor or a multi-CPU processor. The processor here may refer to one or more devices, circuits, and / or processing cores for processing data (e.g., computer program instructions).
[0137] The above-mentioned computer device may be a general-purpose computer device or a special-purpose computer device. In a specific implementation, the computer device may be a desktop computer, a portable computer, a network server, a personal digital assistant (PDA), a mobile phone, a tablet computer, a wireless terminal device, a communication device or an embedded device. The embodiment of the present application does not limit the type of computer device.
[0138] In addition, the present application also provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, the above-mentioned Tibetan text automatic segmentation method is implemented.
[0139] In summary, in the Tibetan machine translation system and the Tibetan text automatic segmentation method disclosed in the embodiment of the present application, firstly, Tibetan text information is obtained, sentence representations are extracted for all Tibetan sentences in the Tibetan text information, and multiple Tibetan sentence representation sequences are obtained; the sentence granularity of each Tibetan sentence in the Tibetan text segmentation process is determined by the position span of each Tibetan character in the Tibetan text information in the context and the semantic dependency relationship between each Tibetan sentence representation sequence, and then all Tibetan sentences in the Tibetan text information are subject-divided according to each sentence granularity to obtain multiple semantic topic clusters; part-of-speech vectors of different Tibetan keywords are extracted from the Tibetan text information, and the inter-word transfer features of each Tibetan keyword in each Tibetan sentence are determined according to the inter-word relationship features of each part-of-speech vector, and then the inter-word transfer degrees of different Tibetan keywords in the Tibetan text segmentation process are determined according to all the inter-word transfer features; the paragraph boundary features of the Tibetan text information are determined according to all the inter-word transfer degrees and all the semantic topic clusters, and then the Tibetan text information is adaptively segmented based on the paragraph boundary features. The solution of the present application can enhance the association between the semantic content of Tibetan and the logical structure changes between paragraphs, thereby improving the accuracy of Tibetan text paragraph division.
[0140] Although the preferred embodiments of the present application have been described, those skilled in the art may make other changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications falling within the scope of the present application.
[0141] Obviously, those skilled in the art can make various changes and modifications to the present application without departing from the spirit and scope of the present application. Thus, if these modifications and variations of the present application fall within the scope of the claims of the present application and their equivalents, the present application is also intended to include these modifications and variations.
Claims
1. A Tibetan text automatic segmentation method, It is characterized in that The steps include: Acquire Tibetan text information, extract sentence representations of all Tibetan sentences in the Tibetan text information, and obtain multiple Tibetan sentence representation sequences; Determining the sentence granularity of each Tibetan sentence in the Tibetan text segmentation process by the position span of each Tibetan character in the Tibetan text information in the context and the semantic dependency relationship between each Tibetan sentence representation sequence, and then subject-dividing all Tibetan sentences in the Tibetan text information according to each sentence granularity to obtain multiple semantic subject clusters; Extracting part-of-speech vectors of different Tibetan keywords from the Tibetan text information, determining inter-word transfer features of each Tibetan keyword in each Tibetan sentence through inter-word relationship features of each part-of-speech vector, and then determining inter-word transfer degrees of different Tibetan keywords in the Tibetan text segmentation process according to all inter-word transfer features; Determining paragraph boundary features of the Tibetan text information through all inter-word transfer degrees and all semantic topic clusters, and then adaptively segmenting the Tibetan text information based on the paragraph boundary features; The paragraph boundary features of the Tibetan text information are determined by all inter-word transfer degrees and all semantic topic clusters, specifically including: Selecting a semantic topic cluster as a selected semantic topic cluster; Determining multiple inter-segment topic change degrees of the selected semantic topic cluster according to all inter-word transfer degrees and each adjacent Tibetan sentence in the selected semantic topic cluster; Continue to determine multiple inter-segment topic variation degrees of the remaining semantic topic clusters; The paragraph boundary features of the Tibetan text information are determined according to all inter-segment topic variation degrees of each semantic topic cluster.
2. The method according to claim 1, It is characterized in that Sentence representations are extracted from all Tibetan sentences in the Tibetan text information to obtain multiple Tibetan sentence representation sequences, specifically including: Converting all Tibetan sentences in the Tibetan text information into Tibetan sentence vector space; Extracting a plurality of Tibetan sentence representation information from the Tibetan sentence vector space based on a pre-trained long short-term memory neural network; The representation information of each Tibetan sentence is subjected to segmented pooling operation to obtain multiple Tibetan sentence representation sequences.
3. The method according to claim 1, It is characterized in that Determining the sentence granularity of each Tibetan sentence in the Tibetan text segmentation process by the position span of each Tibetan character in the Tibetan text information in the context and the semantic dependency relationship between each Tibetan sentence representation sequence specifically includes: Determine the semantic differences of each Tibetan sentence based on all Tibetan sentence representation sequences; Determine the semantic dependencies between the representation sequences of each Tibetan sentence; The sentence granularity of each Tibetan sentence in the Tibetan text segmentation process is determined by the position span of each Tibetan character in the Tibetan text information in the context, all semantic dependencies and all semantic differences.
4. The method according to claim 1, It is characterized in that All Tibetan sentences in the Tibetan text information are divided into topics according to the granularity of each sentence, and multiple semantic topic clusters are obtained, including: Perform linear fitting on all sentence granularities to obtain sentence granularity curve; Determining a plurality of semantic topic centers according to the sentence granularity curve; All Tibetan sentences in the Tibetan text information are divided according to all semantic topic centers and all sentence granularities to obtain multiple semantic topic clusters.
5. The method according to claim 1, It is characterized in that Extracting part-of-speech vectors of different Tibetan keywords from the Tibetan text information specifically includes: Converting the Tibetan text information into a Tibetan text sequence; Determining the part-of-speech co-occurrence degree of each Tibetan character in the Tibetan text information according to the Tibetan text sequence and a preset co-occurrence window; The part-of-speech vectors of different Tibetan keywords are determined according to the part-of-speech co-occurrence of all Tibetan characters.
6. The method according to claim 1, It is characterized in that The inter-word transfer degree of different Tibetan keywords in the Tibetan text segmentation process is determined based on all inter-word transfer features, including: Determine the forward transfer degree and backward transfer degree of each Tibetan keyword based on all inter-word transfer features; The inter-word transfer degrees of different Tibetan keywords in the Tibetan text segmentation process are determined through all forward transfer degrees and all backward transfer degrees.
7. A Tibetan machine translation system, which uses the method according to any one of claims 1 to 6 to automatically segment Tibetan text, the Tibetan machine translation system comprising a Tibetan text automatic segmentation unit, It is characterized in that The Tibetan text automatic segmentation unit comprises: An acquisition module, used to acquire Tibetan text information, extract sentence representations of all Tibetan sentences in the Tibetan text information, and obtain multiple Tibetan sentence representation sequences; A processing module, used to determine the sentence granularity of each Tibetan sentence in the Tibetan text segmentation process through the position span of each Tibetan character in the Tibetan text information in the context and the semantic dependency relationship between each Tibetan sentence representation sequence, and then divide all the Tibetan sentences in the Tibetan text information into topics according to each sentence granularity to obtain multiple semantic topic clusters; The processing module is further used to extract part-of-speech vectors of different Tibetan keywords from the Tibetan text information, determine the inter-word transfer features of each Tibetan keyword in each Tibetan sentence through the inter-word relationship features of each part-of-speech vector, and then determine the inter-word transfer degrees of different Tibetan keywords in the Tibetan text segmentation process according to all the inter-word transfer features; The execution module is used to determine the paragraph boundary features of the Tibetan text information through all the inter-word transfer degrees and all the semantic topic clusters, and then adaptively segment the Tibetan text information based on the paragraph boundary features.
8. A computer device comprising a memory and a processor, wherein the memory stores a computer program. It is characterized in that When the processor executes the computer program, the Tibetan text automatic segmentation method according to any one of claims 1 to 6 is implemented.
9. A computer-readable storage medium storing a computer program. It is characterized in that When the computer program is executed by a processor, the method for automatic segmentation of Tibetan text according to any one of claims 1 to 6 is implemented.
Citation Information
Cited By
Tibetan written language character-voice conversion method based on voice structure characteristics
CN121789641A
A method for converting Tibetan written language phonetics based on phonetic structure features
CN121789641B
