A Tibetan text automatic annotation method and annotation system
Through semantic features and key feature extraction methods, combined with the attention mechanism, the two-way long and short-term memory network is solved, and the part-of-speech labeling problem of polysynthetics and homonyms in Tibetan text is improved, and the accuracy of Tibetan part-of-speech judgment is improved.
Patent Information
- Application Number
- CN202411243539.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-09-05
- Publication Date
- 2024-12-03
- Estimated Expiration
- 2044-09-05
AI Technical Summary
There are a large number of polysynonyms and homonyms in Tibetan texts, which makes it difficult to achieve the accuracy of part-of-speech labeling and limited flexibility and expansion of traditional methods.
Semantic feature extraction and key feature extraction methods are adopted to obtain semantic representation vectors and part-of-speech vectors through a two-way long and short-term memory network of attention mechanism, and combine semantic similarity and part-of-speech distribution to determine the label granularity and part-of-speech distinction to realize adaptive annotation.
It improves the accuracy of Tibetan part-of-speech judgment, accurately distinguishes the part-of-speech of polysynonyms and homonyms, and reduces labeling ambiguity.
Smart Images

Figure CN118761413B_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the technical field of Tibetan text annotation, and more specifically, to a Tibetan text automatic annotation method and annotation system. Background Art
[0002] Tibetan text annotation refers to the annotation of words, sentences, paragraphs, etc. in Tibetan texts, aiming to improve the readability and processability of the text. In this process, commonly used annotation types include part-of-speech tagging, named entity recognition, syntactic analysis, etc. Part-of-speech tagging refers to assigning corresponding part-of-speech tags to each word in the text. Common methods include statistical-based annotation and machine learning-based annotation. Named entity recognition is to identify proper nouns such as names of people, places, and time in the text, while syntactic analysis focuses on the grammatical relationship between each component in the sentence. In Tibetan text annotation, due to the special grammatical structure and character characteristics of Tibetan, traditional Chinese annotation techniques and tools often need to be properly adjusted and optimized when applied. With the continuous enrichment of the Tibetan corpus and the advancement of technology, Tibetan text annotation technology is developing in a more automated and intelligent direction.
[0003] Automatic Tibetan text annotation mainly relies on machine learning and deep learning models in terms of part-of-speech tagging. Traditional rule-based methods use grammatical rules to perform part-of-speech tagging, but their flexibility and scalability are limited. Machine learning algorithms such as conditional random fields perform part-of-speech tagging by learning patterns from annotated data, and can better handle diverse language phenomena. However, there are a large number of polysemous words and homonyms in Tibetan, that is, the same word may represent different parts of speech in different contexts. For example, a word may be a noun in one sentence and a verb in another sentence. Since there are many such words in Tibetan, it becomes very difficult to accurately determine the part of speech based on the context. Therefore, how to achieve part-of-speech tagging of Tibetan polysemous words and homonyms, thereby improving the accuracy of Tibetan part-of-speech judgment has become a difficult problem faced by the industry. Summary of the invention
[0004] The present application provides a Tibetan text automatic annotation method and annotation system, which can realize the part-of-speech tagging of Tibetan polysemous words and homonyms, thereby improving the accuracy of Tibetan part-of-speech judgment.
[0005] In a first aspect, the present application provides a method for automatically annotating Tibetan text, comprising the following steps:
[0006] Acquire Tibetan text contents in different scenarios, and then combine all acquired Tibetan text contents into Tibetan text information;
[0007] Extracting semantic features from the Tibetan text information to obtain a plurality of semantic representation vectors, and determining a label granularity for marking each Tibetan character in the Tibetan text information according to the semantic similarity between the semantic representation vectors and the part-of-speech distribution of each Tibetan character in all Tibetan text contents in the entire text;
[0008] Determine the part-of-speech tagging features of each Tibetan text content based on all label granularities and all semantic representation vectors;
[0009] Extract key features from the Tibetan text information to obtain part-of-speech vectors of different Tibetan keywords, and determine the part-of-speech distinction of different Tibetan keywords in the annotation process according to the distance dependency relationship of each part-of-speech vector in the full text and the part-of-speech tags of each Tibetan text content;
[0010] The part-of-speech tag vector of the Tibetan text information is determined by all the part-of-speech tag features and all the part-of-speech distinctions, and then the Tibetan text information is adaptively tagged according to the part-of-speech tag vector.
[0011] In some embodiments, performing semantic feature extraction on the Tibetan text information to obtain multiple semantic representation vectors specifically includes:
[0012] Converting all Tibetan text content in the Tibetan text information into a Tibetan text vector set;
[0013] A bidirectional long short-term memory network based on an attention mechanism extracts multiple semantic representation vectors from the Tibetan text vector set.
[0014] In some embodiments, determining the label granularity of each Tibetan character in the Tibetan text information when marking according to the semantic similarity between the semantic representation vectors and the part-of-speech distribution of each Tibetan character in the entire Tibetan text content specifically includes:
[0015] Determine the semantic similarity between each semantic representation vector;
[0016] Determine the semantic block features of each semantic representation vector corresponding to the Tibetan sentence according to all semantic similarities;
[0017] Determine the part-of-speech granularity feature of the corresponding Tibetan characters through the part-of-speech distribution of each Tibetan character in the full text of all Tibetan text contents;
[0018] The label granularity of each Tibetan character in the Tibetan text information when marking is determined according to the semantic block features of each Tibetan sentence and the part-of-speech granularity features of each Tibetan character.
[0019] In some embodiments, determining the part-of-speech tagging features of each Tibetan text content according to all tag granularities and all semantic representation vectors specifically includes:
[0020] Linear fitting is performed on all tag particle sizes to obtain a tag particle size curve;
[0021] Determine the part-of-speech tag vector of each Tibetan text content according to the tag granularity curve and all semantic representation vectors;
[0022] The part-of-speech tagging features of each Tibetan text content are determined through all part-of-speech tag vectors and all tag granularities.
[0023] In some embodiments, extracting key features from the Tibetan text information to obtain part-of-speech vectors of different Tibetan keywords specifically includes:
[0024] Converting all Tibetan text contents in the Tibetan text information into a Tibetan text content sequence;
[0025] Determining the context co-occurrence degree of each Tibetan character in the Tibetan text information according to the Tibetan text content sequence and a preset context window;
[0026] The part-of-speech vectors of different Tibetan keywords are determined through all context co-occurrences.
[0027] In some embodiments, determining the part-of-speech distinction of different Tibetan keywords in the tagging process according to the distance dependency relationship of each part-of-speech vector in the full text and the part-of-speech tags of each Tibetan text content specifically includes:
[0028] Determine the distance dependency of each part-of-speech vector in the full text;
[0029] Select a Tibetan text content as the selected Tibetan text content;
[0030] Determine the number of successive occurrences of all Tibetan keywords in the selected Tibetan text content;
[0031] Determine the part-of-speech distinction of all Tibetan keywords in the selected Tibetan text content during the tagging process according to the part-of-speech tags and all subsequent times of the selected Tibetan text content;
[0032] Continue to determine the part-of-speech distinction of all Tibetan keywords in the remaining Tibetan text content during the annotation process.
[0033] In some embodiments, determining the part-of-speech tag vector of the Tibetan text information by using all part-of-speech tagging features and all part-of-speech distinctions specifically includes:
[0034] Determining the global part-of-speech distinction of the Tibetan text information according to all part-of-speech distinctions;
[0035] Determine the part-of-speech tag group of each part-of-speech distinction through the global part-of-speech distinction, all part-of-speech tagging features and all part-of-speech distinctions;
[0036] The sequence formed by all the part-of-speech tag groups is used as the part-of-speech tag vector of the Tibetan text information.
[0037] In a second aspect, the present application provides a Tibetan text automatic annotation system, comprising:
[0038] An acquisition module is used to acquire Tibetan text contents in different scenarios, and then to form Tibetan text information from all acquired Tibetan text contents;
[0039] A processing module is used to extract semantic features from the Tibetan text information to obtain multiple semantic representation vectors, and determine the label granularity of each Tibetan character in the Tibetan text information when marking according to the semantic similarity between the semantic representation vectors and the part-of-speech distribution of each Tibetan character in all Tibetan text contents in the full text;
[0040] The processing module is further used to determine the part-of-speech tagging features of each Tibetan text content according to all label granularities and all semantic representation vectors;
[0041] The processing module is further used to extract key features from the Tibetan text information to obtain part-of-speech vectors of different Tibetan keywords, and determine the part-of-speech distinction of different Tibetan keywords in the annotation process according to the distance dependency relationship of each part-of-speech vector in the full text and the part-of-speech tags of each Tibetan text content;
[0042] The execution module is used to determine the part-of-speech tag vector of the Tibetan text information through all the part-of-speech tag features and all the part-of-speech distinctions, and then adaptively tag the Tibetan text information according to the part-of-speech tag vector.
[0043] In a third aspect, the present application provides a computer device, comprising a memory and a processor, wherein the memory stores codes, and the processor is configured to obtain the codes and execute the above-mentioned Tibetan text automatic annotation method.
[0044] In a fourth aspect, the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the above-mentioned Tibetan text automatic annotation method is implemented.
[0045] The technical solution provided by the embodiments disclosed in this application has the following beneficial effects:
[0046] The present application provides a Tibetan text automatic annotation method and annotation system. First, Tibetan text contents in different scenarios are obtained, and then all the obtained Tibetan text contents are combined into Tibetan text information; semantic feature extraction is performed on the Tibetan text information to obtain multiple semantic representation vectors; the label granularity of each Tibetan character in the Tibetan text information when annotating is determined according to the semantic similarity between each semantic representation vector and the part-of-speech distribution of each Tibetan character in all Tibetan text contents in the full text, and then the part-of-speech tagging features of each Tibetan text content are determined according to all label granularities and all semantic representation vectors; key feature extraction is performed on the Tibetan text information to obtain part-of-speech vectors of different Tibetan keywords, and the part-of-speech distinction of different Tibetan keywords in the annotation process is determined according to the distance dependency relationship of each part-of-speech vector in the full text and the part-of-speech tag of each Tibetan text content; the part-of-speech tag vector of the Tibetan text information is determined through all the part-of-speech tagging features and all the part-of-speech distinctions, and then the Tibetan text information is adaptively annotated according to the part-of-speech tag vector.
[0047] It can be seen that in the present application, the part-of-speech tag vector of the Tibetan text information can be determined by all part-of-speech tagging features and all part-of-speech distinctions; wherein, firstly, the part-of-speech distribution of the frequency distribution of the part-of-speech categories of each Tibetan character in the full text is used to extract fine-grained features describing the Tibetan character part-of-speech tags, which can reveal the grammatical structure of the Tibetan text content and identify the functions and relationships of Tibetan vocabulary in Tibetan sentences. Then, the fine-grained features of the Tibetan character part-of-speech tags are quantified in combination with the semantic similarity between each semantic representation vector, and the label granularity that measures the degree of refinement of the Tibetan character part-of-speech tags during annotation is obtained. The label granularity provides detailed annotation information in order to more accurately reflect the role and meaning of each Tibetan character in the text, and then all label granularities and all semantic representation vectors are used to obtain the part-of-speech tagging features used to distinguish the parts of speech of polysemous words in the Tibetan text content in different contexts, which helps to reduce annotation ambiguity. , improve the accuracy of annotation; secondly, extract distance features from each part-of-speech vector, obtain the distance dependency relationship of the Tibetan keywords corresponding to the part-of-speech vector and the Tibetan keywords corresponding to other part-of-speech vectors in the position space distribution, the distance dependency relationship is helpful to identify the part-of-speech of the correlation between polysemy and homonymy in the context, further combine the part-of-speech tag marked by the position of each Tibetan character in the Tibetan sentence in the Tibetan text information, and obtain the part-of-speech distinction degree that distinguishes the importance of polysemy and homonymy in the Tibetan sentence. The part-of-speech distinction degree can distinguish the different usages of polysemy and homonymy in the context, and then accurately mark the part-of-speech of polysemy and homonymy; then, quantify all the part-of-speech annotation features and all the part-of-speech distinction degrees to obtain the part-of-speech tag vector of the Tibetan text information; finally, use the part-of-speech tag vector to adaptively annotate the Tibetan text information. In summary, the solution of the present application can realize the part-of-speech tagging of Tibetan polysemous words and homonyms, thereby improving the accuracy of Tibetan part-of-speech judgment. BRIEF DESCRIPTION OF THE DRAWINGS
[0048] Figure 1 is an exemplary flow chart of a Tibetan text automatic annotation method according to some embodiments of the present application;
[0049] Figure 2 is a schematic diagram of a process for determining part-of-speech tagging features in some embodiments of the present application;
[0050] Figure 3 is a schematic diagram of a process for determining a part-of-speech vector in some embodiments of the present application;
[0051] Figure 4 is a structural block diagram of a Tibetan text automatic annotation system in some embodiments of the present application;
[0052] Figure 5It is a structural schematic diagram of a computer device for implementing a method for automatically annotating Tibetan text according to some embodiments of the present application. DETAILED DESCRIPTION
[0053] The core of this application is to first obtain Tibetan text content in different scenarios, and then form Tibetan text information with all the obtained Tibetan text content; extract semantic features of the Tibetan text information to obtain multiple semantic representation vectors; determine the label granularity of each Tibetan character in the Tibetan text information when marking according to the semantic similarity between each semantic representation vector and the part-of-speech distribution of each Tibetan character in all Tibetan text content in the full text, and then determine the part-of-speech tagging features of each Tibetan text content according to all label granularities and all semantic representation vectors; extract key features of the Tibetan text information to obtain the part-of-speech vectors of different Tibetan keywords, and determine the part-of-speech distinction of different Tibetan keywords in the marking process according to the distance dependency of each part-of-speech vector in the full text and the part-of-speech tags of each Tibetan text content; determine the part-of-speech tag vector of the Tibetan text information through all part-of-speech tagging features and all part-of-speech distinctions, and then perform adaptive tagging on the Tibetan text information according to the part-of-speech tag vector. The scheme of this application can realize the part-of-speech tagging of Tibetan polysemous words and homonymous words, thereby improving the accuracy of Tibetan part-of-speech judgment.
[0054] In order to better understand the above technical solution, the above technical solution will be described in detail below in conjunction with the accompanying drawings and specific implementation methods. Figure 1 , which is an exemplary flow chart of a Tibetan text automatic annotation method according to some embodiments of the present application. The Tibetan text automatic annotation method 100 mainly includes the following steps:
[0055] In step 101, Tibetan text contents in different scenarios are obtained, and then all the obtained Tibetan text contents are combined into Tibetan text information.
[0056] In specific implementation, for example, Tibetan text contents in different scenarios may be obtained from a Tibetan database, and a set consisting of all obtained Tibetan text contents may be used as Tibetan text information, which will not be described in detail here.
[0057] It should be noted that the different scenarios in this application refer to different fields such as academic papers, news texts, legal documents, religious classics, etc., and the Tibetan text content is a collection of multiple Tibetan sentences, where the Tibetan sentences are composed of multiple Tibetan characters.
[0058] In step 102, semantic features are extracted from the Tibetan text information to obtain a plurality of semantic representation vectors, and the label granularity of each Tibetan character in the Tibetan text information when annotating is determined according to the semantic similarity between the semantic representation vectors and the part-of-speech distribution of each Tibetan character in all Tibetan text contents in the entire text.
[0059] In some embodiments, the semantic feature extraction of the Tibetan text information to obtain multiple semantic representation vectors can be achieved by using the following steps:
[0060] Converting all Tibetan text content in the Tibetan text information into a Tibetan text vector set;
[0061] A bidirectional long short-term memory network based on an attention mechanism extracts multiple semantic representation vectors from the Tibetan text vector set.
[0062] In a specific implementation, converting all Tibetan text contents in the Tibetan text information into a Tibetan text vector set can be implemented in the following manner, namely: selecting a Tibetan text content from the Tibetan text information as the selected Tibetan text content, using a Tibetan natural language processing tool to segment each Tibetan sentence in the selected Tibetan text content, obtaining multiple Tibetan segmentations of each Tibetan sentence, and taking the sets of multiple Tibetan segmentations of each Tibetan sentence as Tibetan phrases, and then using a pre-trained word vector model (such as Word2Vec, GloVe or FastText not mentioned here) to perform word segmentation on each Tibetan sentence. As a limitation, each Tibetan phrase is converted into a word vector sequence, thereby obtaining multiple word vector sequences, and an ordered sequence composed of all the obtained word vector sequences is used as a word vector sequence group for the selected Tibetan text content, wherein the word vector sequence is composed of multiple word vectors, each word vector corresponds to a Tibetan word segment, and the word vector sequence in the word vector sequence group is arranged in order, and the word vector sequence group of the remaining Tibetan text content is continued to be determined, so that the set composed of all word vector sequence groups is used as a Tibetan text vector set. In other embodiments, other methods can also be used to achieve this, which will not be repeated here.
[0063] In specific implementation, the bidirectional long short-term memory network based on the attention mechanism extracts multiple semantic representation vectors from the Tibetan text vector set, which can be implemented in the following manner, namely: selecting a word vector sequence group from the Tibetan text vector set as the selected word vector sequence group, using the bidirectional long short-term memory network based on the attention mechanism to perform feature extraction on each word vector sequence in the selected word vector sequence group, thereby obtaining multiple vectors, and using each vector as the semantic representation vector of the corresponding word vector sequence, performing feature extraction on each word vector sequence in the remaining word vector sequence group, thereby obtaining multiple semantic representation vectors. Other methods may also be used in other embodiments, which are not limited here.
[0064] It should be noted that the semantic representation vector described in the present application represents a vector composed of the semantic content represented by the word vector sequence corresponding to the Tibetan phrase. The semantic representation vector reflects the grammatical components and structure of the Tibetan sentence, including syntactic relations such as subject, predicate and object. In addition, it should be noted that each semantic representation vector in the present application corresponds to a Tibetan sentence.
[0065] In some embodiments of the present application, determining the label granularity of each Tibetan character in the Tibetan text information when marking according to the semantic similarity between the semantic representation vectors and the part-of-speech distribution of each Tibetan character in the full text of all Tibetan text contents can be implemented by the following steps:
[0066] Determine the semantic similarity between each semantic representation vector;
[0067] Determine the semantic block features of each semantic representation vector corresponding to the Tibetan sentence according to all semantic similarities;
[0068] Determine the part-of-speech granularity feature of the corresponding Tibetan characters through the part-of-speech distribution of each Tibetan character in the full text of all Tibetan text contents;
[0069] The label granularity of each Tibetan character in the Tibetan text information when marking is determined according to the semantic block features of each Tibetan sentence and the part-of-speech granularity features of each Tibetan character.
[0070] In specific implementation, the semantic similarity between each semantic representation vector can be determined in the following manner, namely: select a semantic representation vector as the selected semantic representation vector, calculate the cosine similarity between the selected semantic representation vector and other semantic representation vectors, then calculate the mean of all cosine similarities, use the obtained mean as the semantic similarity between the selected semantic representation vector and other semantic representation vectors, and continue to determine the semantic similarity between the remaining semantic representation vectors and other semantic representation vectors. Other methods can also be used in other embodiments, which are not limited here.
[0071] It should be noted that the semantic similarity described in the present application measures the degree of semantic similarity between Tibetan sentences. The greater the semantic similarity, the higher the degree of semantic similarity between Tibetan sentences, and vice versa. Each semantic similarity corresponds to a semantic representation vector, and each semantic representation vector corresponds to a Tibetan sentence, that is, one semantic similarity corresponds to one Tibetan sentence.
[0072] In specific implementation, the semantic segmentation features of the Tibetan sentences corresponding to each semantic representation vector are determined according to all semantic similarities, which can be implemented in the following manner, namely: all semantic similarities are clustered using hierarchical clustering to obtain multiple clusters, and each of the obtained clusters is used as a semantic similarity cluster, a semantic representation vector is selected as a selected semantic representation vector, the mean of all values in the selected semantic representation vector is calculated, and then the reciprocal of the exponential function with the natural logarithm as the base of the obtained mean is taken, and the value obtained by taking the reciprocal of the exponential function with the natural logarithm as the base is taken as the semantic representation coefficient, and then the mean of all semantic similarities in the semantic similarity cluster where the semantic similarity corresponding to the selected semantic representation vector is located is calculated, and then the obtained mean is multiplied by the semantic representation coefficient, and the multiplied value is used as the semantic segmentation feature of the Tibetan sentence corresponding to the selected semantic representation vector, and the semantic segmentation features of the Tibetan sentences corresponding to the remaining semantic representation vectors are continued to be determined. In other embodiments, other methods can also be used for implementation, which will not be repeated here.
[0073] It should be noted that the semantic chunking feature described in this application reflects the importance of different Tibetan sentences in the overall Tibetan sentence. By introducing the semantic chunking feature, the Tibetan text information can be divided into blocks with similar semantics, providing richer contextual information, so that the role of vocabulary in a specific semantic block can be better understood, thereby improving the accuracy of part-of-speech judgment.
[0074] In addition, it should be noted that the part-of-speech distribution described in this application represents the distribution of the frequency of occurrence of part-of-speech categories (such as nouns, verbs, adjectives, etc.). The BERT model can be trained using Tibetan text information with part-of-speech annotated, and the model obtained after training can be used as a part-of-speech tagging model.
[0075] In specific implementation, the part-of-speech granularity feature of the corresponding Tibetan character is determined by the part-of-speech distribution of each Tibetan character in all Tibetan text contents in the full text, which can be implemented in the following manner, namely: select a Tibetan text content as the selected Tibetan text content, then select a Tibetan character from the selected Tibetan text content as the selected Tibetan character, calculate the distance between the selected Tibetan character and the same Tibetan character in the selected Tibetan text content, and then add all the distances and divide them by the part-of-speech distribution of the Tibetan character in the full text, and use the value obtained by the division as the part-of-speech granularity feature of the selected Tibetan character, continue to determine the part-of-speech granularity features of the remaining selected Tibetan characters in the selected Tibetan text content, repeat the above steps, and perform the same processing on all the remaining Tibetan text contents, so as to obtain the part-of-speech granularity features of different Tibetan characters. In other embodiments, other methods can also be used for implementation, which are not limited here.
[0076] It should be noted that the part-of-speech granularity features described in this application represent the fine-grained features of Tibetan character part-of-speech tags in the context of different Tibetan text contents. The part-of-speech granularity features can reveal the grammatical structure of Tibetan text contents, identify the functions and relationships of Tibetan vocabulary in Tibetan sentences, and improve the accuracy of grammatical analysis.
[0077] In specific implementation, determining the label granularity of each Tibetan character in the Tibetan text information when marking according to the semantic chunking features of each Tibetan sentence and the part-of-speech granularity features of each Tibetan character can be implemented in the following manner, namely: selecting a Tibetan sentence as the selected Tibetan sentence, dividing the part-of-speech granularity features of each Tibetan character in the selected Tibetan sentence by the semantic chunking features of the selected Tibetan sentence, and using the values obtained by the division as the label granularity of the corresponding Tibetan character in the Tibetan text information during the marking process, and continuing to perform the same processing on the remaining Tibetan sentences, so as to obtain the label granularity of each Tibetan character in the Tibetan text information when marking. In other embodiments, other methods can also be used for implementation, which will not be repeated here.
[0078] It should be noted that the label granularity described in this application represents the degree of refinement of assigning a part-of-speech tag to each Tibetan character when annotating Tibetan text information. The label granularity takes into account the context, semantic block, and part-of-speech changes of each Tibetan character, and provides detailed annotation information to more accurately reflect the role and meaning of each Tibetan character in the text.
[0079] In step 103, part-of-speech tagging features of each Tibetan text content are determined according to all label granularities and all semantic representation vectors.
[0080] In some embodiments of the present application, reference Figure 2 As shown, the figure is a schematic diagram of the process of determining part-of-speech tagging features in some embodiments of the present application. In this embodiment, the part-of-speech tagging features of each Tibetan text content are determined according to all label granularities and all semantic representation vectors, which can be implemented by the following steps:
[0081] First, in step 1031, linear fitting is performed on all tag particle sizes to obtain a tag particle size curve;
[0082] Next, in step 1032, the part-of-speech tag vector of each Tibetan text content is determined according to the tag granularity curve and all semantic representation vectors;
[0083] Then, in step 1033, the part-of-speech tagging features of each Tibetan text content are determined through all the part-of-speech tag vectors and all the tag granularities.
[0084] In the specific implementation, linear fitting is performed on all label granularities to obtain a label granularity curve, which can be achieved in the following manner, namely: all label granularities are sorted in ascending order, and the sorted sequence is used as the label granularity sequence, the label granularity sequence is fit by the least squares support vector machine algorithm in the prior art, and the fitted sequence is used as the label granularity curve. In other embodiments, other fitting algorithms can also be used, which are not limited here.
[0085] In specific implementation, the following method is used to determine the part-of-speech tag vector of each Tibetan text content according to the label granularity curve and all semantic representation vectors, namely: first, obtain the derivative of the label granularity curve, select the largest derivative and the smallest derivative from all derivatives, subtract the smallest derivative from the largest derivative, and use the subtracted value as the label derivative extreme value, then, select the semantic representation vector belonging to the same Tibetan text content from all semantic representation vectors, and use the set of semantic representation vectors belonging to the same Tibetan text content as the semantic representation vector space, that is, each semantic representation vector space corresponds to a Tibetan text content, select a semantic representation vector space as the selected semantic representation vector space, select the semantic representation vector with the largest variance and the semantic representation vector with the smallest variance from the selected semantic representation vector space, then subtract all values in the semantic representation vector with the largest variance from the corresponding values in the semantic representation vector with the smallest variance, and use the subtracted value to obtain the label derivative extreme value. The sequence composed of all the values obtained is taken as the semantic representation difference vector, for example: the semantic representation vector A=[1,8,14] with the largest variance and the semantic representation vector B=[1,2,3] with the smallest variance, then the first element 1 in the semantic representation vector A is subtracted from the first element 1 of the semantic representation vector B to obtain 0, and then the second element 8 in the semantic representation vector A is subtracted from the second element 2 of the semantic representation vector B to obtain 6, and then the third element 14 in the semantic representation vector A is subtracted from the third element 3 of the semantic representation vector B to obtain 9, so that the obtained semantic representation difference vector is [0,6,9], and then each value in the semantic representation difference vector is multiplied by the label derivative extreme difference value, and the vector obtained after multiplying by the label derivative extreme difference value is used as the part-of-speech tag vector of the Tibetan text content corresponding to the selected semantic representation vector space, and the part-of-speech tag vectors of the Tibetan text content corresponding to the remaining semantic representation vector space are further determined. In other embodiments, other methods can also be used for implementation, which is not limited here.
[0086] It should be noted that the part-of-speech tag vector described in the present application can identify and distinguish different parts of speech of polysemous words of Tibetan characters in various Tibetan sentences in Tibetan text content in different contexts, thereby improving the accuracy of annotation.
[0087] In specific implementation, the part-of-speech tagging features of each Tibetan text content are determined by all part-of-speech tag vectors and all tag granularities, which can be implemented in the following way, namely: select the tag granularity belonging to the same Tibetan text content from all tag granularities, and use the tag granularity belonging to the same Tibetan text content as a tag granularity cluster, thereby obtaining multiple tag granularity clusters, wherein each tag granularity cluster corresponds to a Tibetan text content, select a tag granularity cluster as the selected tag granularity cluster, subtract the smallest tag granularity from the largest tag granularity in the selected tag granularity cluster, and then divide the subtracted value by the value of all tag granularities in the selected tag granularity cluster. The mean of , and the value obtained by division is used as the standard granularity value of the selected label granularity cluster, and the standard granularity values of the remaining label granularity clusters are continued to be determined, and a part-of-speech tag vector is selected from all the part-of-speech tag vectors as the selected part-of-speech tag vector, and the standard granularity value of the label granularity cluster corresponding to the selected part-of-speech tag vector is multiplied by all values in the selected part-of-speech tag vector, and the selected part-of-speech tag vector after multiplying by the standard granularity value is used as the part-of-speech tagging feature of the Tibetan text content corresponding to the selected part-of-speech tag vector, and the part-of-speech tagging features of the Tibetan text content corresponding to the remaining part-of-speech tag vectors are continued to be determined. In other embodiments, other methods can also be used to achieve this, which will not be repeated here.
[0088] It should be noted that the part-of-speech tagging features described in the present application can accurately distinguish the parts of speech of polysemous words in Tibetan text content in different contexts, and correctly mark the parts of speech of homonyms, which helps to reduce tagging ambiguity and improve the accuracy of tagging.
[0089] In step 104, key features are extracted from the Tibetan text information to obtain part-of-speech vectors of different Tibetan keywords, and the part-of-speech distinction of different Tibetan keywords in the annotation process is determined according to the distance dependency relationship between the part-of-speech vectors in the full text and the part-of-speech tags of each Tibetan text content.
[0090] In some embodiments, reference Figure 3 As shown, this figure is a schematic diagram of the process of determining the part-of-speech vector in some embodiments of the present application. In this embodiment, the key features of the Tibetan text information are extracted to obtain the part-of-speech vectors of different Tibetan keywords, which can be achieved by the following steps:
[0091] First, in step 1041, all Tibetan text contents in the Tibetan text information are converted into a Tibetan text content sequence;
[0092] Secondly, in step 1042, the contextual co-occurrence degree of each Tibetan character in the Tibetan text information is determined according to the Tibetan text content sequence and a preset context window;
[0093] Then, in step 1043, part-of-speech vectors of different Tibetan keywords are determined through all context co-occurrences.
[0094] In specific implementation, converting all Tibetan text contents in the Tibetan text information into a Tibetan text content sequence can be implemented in the following manner, namely: first, calculating the total number of Tibetan characters in each Tibetan text content, and using the obtained total number as the total number of Tibetan characters of the corresponding Tibetan text content; secondly, sorting the arrangement order of each Tibetan text content according to the total number of Tibetan characters of each Tibetan text content, and using the ordered sequence composed of the sorted Tibetan text contents as the Tibetan text content sequence. In other embodiments, other methods can also be used for implementation, which will not be repeated here.
[0095] It should be noted that the context window described in the present application represents a sliding window for performing operations on a Tibetan text content sequence, which can be set using a fixed-length sequence, wherein the fixed length can be set using the value obtained by rounding down the square root of the total number of Tibetan characters in the Tibetan text sequence, and the value obtained by rounding down the square root of the total number of Tibetan characters in the Tibetan text sequence is used as the total number of keywords.
[0096] In a specific implementation, determining the contextual co-occurrence of each Tibetan character in the Tibetan text information according to the Tibetan text content sequence and the preset context window can be implemented in the following manner, namely: first, using the GloVe word embedding model to convert all Tibetan characters in the Tibetan text sequence into vectors, and using all the obtained vectors as Tibetan character vectors, wherein each Tibetan character corresponds to a Tibetan character vector; secondly, aligning the context window with the first Tibetan character in the Tibetan text sequence, and calculating the conditional entropy between the Tibetan character vector corresponding to the first Tibetan character and the Tibetan character vector corresponding to the Tibetan character covered by the context window , and taking the maximum conditional entropy as the context co-occurrence degree of the first Tibetan character in the Tibetan text information, sliding the context window, aligning the context window with the second Tibetan character in the Tibetan text sequence, calculating the conditional entropy between the Tibetan character vector corresponding to the second Tibetan character and the Tibetan character vector corresponding to the Tibetan character covered by the context window, and taking the maximum conditional entropy as the context co-occurrence degree of the second Tibetan character in the Tibetan text information, repeating the above steps, and determining the context co-occurrence degrees of the remaining Tibetan characters in the Tibetan text information in turn. In other embodiments, other methods may also be used to implement, which will not be repeated here.
[0097] It should be noted that the context co-occurrence degree described in the present application represents the uncertainty measure of the local occurrence of other Tibetan characters under the premise that there are Tibetan characters in the context of Tibetan text information, wherein each context co-occurrence degree corresponds to a Tibetan character, and each Tibetan character corresponds to a Tibetan character vector.
[0098] In specific implementation, determining the part-of-speech vectors of different Tibetan keywords through all context co-occurrences can be implemented in the following manner, namely: sorting all context co-occurrences in descending order, taking the obtained sequence as the context co-occurrence sequence, selecting context co-occurrences of the same number as the total number of keywords from the context co-occurrence sequence in turn, and taking the Tibetan characters corresponding to each selected context co-occurrence as the Tibetan keywords, and then taking the Tibetan character vector corresponding to each Tibetan keyword as the part-of-speech vector, thereby obtaining the part-of-speech vectors of different Tibetan keywords. In other embodiments, other methods can also be used for implementation, which are not limited here.
[0099] In some embodiments, determining the part-of-speech distinction of different Tibetan keywords in the tagging process according to the distance dependency relationship of each part-of-speech vector in the full text and the part-of-speech tags of each Tibetan text content can be implemented by the following steps:
[0100] Determine the distance dependency of each part-of-speech vector in the full text;
[0101] Select a Tibetan text content as the selected Tibetan text content;
[0102] Determine the number of successive occurrences of all Tibetan keywords in the selected Tibetan text content;
[0103] Determine the part-of-speech distinction of all Tibetan keywords in the selected Tibetan text content during the tagging process according to the part-of-speech tags and all subsequent times of the selected Tibetan text content;
[0104] Continue to determine the part-of-speech distinction of all Tibetan keywords in the remaining Tibetan text content during the annotation process.
[0105] In specific implementation, determining the distance dependency of each part-of-speech vector in the full text can be achieved in the following manner, namely: selecting a part-of-speech vector as the selected part-of-speech vector, calculating the cosine similarity between the selected part-of-speech vector and the remaining part-of-speech vectors, and taking the maximum cosine similarity as the maximum cosine similarity, and then calculating the distance between the positions where the Tibetan keywords corresponding to the selected part-of-speech vector appear in the entire Tibetan text information, summing all the distances, and then multiplying the summed value by the maximum cosine similarity, and taking the multiplied value as the distance dependency of the selected part-of-speech vector in the full text, and continuing to determine the distance dependency of the remaining part-of-speech vectors in the full text. In other embodiments, other methods can also be used for implementation, which are not limited here.
[0106] It should be noted that the distance dependency relationship described in the present application represents a measure of the semantic dependency relationship between the Tibetan keywords corresponding to the part-of-speech vector and the Tibetan keywords corresponding to other part-of-speech vectors in terms of positional spatial distribution. The distance dependency relationship helps to identify the parts of speech related to polysemous words and homonyms in the context.
[0107] In specific implementation, determining the number of successors of all Tibetan keywords in the selected Tibetan text content can be achieved in the following manner, namely: calculating the total number of subsequent appearances of each Tibetan keyword in the selected Tibetan text content in the selected Tibetan text content, and taking the total number of subsequent appearances of each Tibetan keyword as the number of successors of the corresponding Tibetan keyword. In other embodiments, other methods can also be used for implementation, which will not be repeated here.
[0108] It should be noted that the part-of-speech tags described in the present application represent the part-of-speech position tags of each Tibetan character in the Tibetan text. Common parts of speech include nouns, verbs, adjectives, adverbs, etc., wherein the part-of-speech position of each Tibetan character is marked with the starting position, the middle position, and the ending position. The part-of-speech tags can be used to accurately identify the boundaries and positions of Tibetan characters. In specific implementation, for example, the BERT model can be used to obtain the part-of-speech tags of the Tibetan text content, which will not be repeated here.
[0109] In specific implementation, determining the part-of-speech distinction of all Tibetan keywords in the selected Tibetan text content during the marking process according to the part-of-speech tag of the selected Tibetan text content and all subsequent times can be implemented in the following manner, namely: selecting a Tibetan character from the selected Tibetan text content as the selected Tibetan character, if the selected Tibetan character is not a Tibetan keyword, subtracting the starting position of the selected Tibetan character from the end position of the selected Tibetan character, and using the subtracted value as the part-of-speech distinction of the selected Tibetan character during the marking process; if the selected Tibetan character is a Tibetan keyword, subtracting the middle position of the selected Tibetan character from the end position of the selected Tibetan character, and then dividing the subtracted value by the difference between the middle position of the selected Tibetan character and the starting position of the selected Tibetan character, and then multiplying the divided value by the subsequent times of the Tibetan character, and using the multiplied value as the part-of-speech distinction of the selected Tibetan character during the marking process, and continuing to determine the part-of-speech distinction of the remaining Tibetan keywords in the selected Tibetan text content during the marking process. In other embodiments, other methods can also be used for implementation, which will not be repeated here.
[0110] It should be noted that the part-of-speech distinction degree described in the present application indicates the importance of distinguishing polysemous words and homographs in Tibetan sentences. The part-of-speech distinction degree can be used to distinguish the different usages of polysemous words and homographs in the context, thereby accurately marking the part-of-speech of polysemous words and homographs.
[0111] In step 105, the part-of-speech tag vector of the Tibetan text information is determined by all the part-of-speech tag features and all the part-of-speech distinctions, and then the Tibetan text information is adaptively tagged according to the part-of-speech tag vector.
[0112] In some embodiments, determining the part-of-speech tag vector of the Tibetan text information by using all part-of-speech tagging features and all part-of-speech distinctions can be implemented by the following steps:
[0113] Determining the global part-of-speech distinction of the Tibetan text information according to all part-of-speech distinctions;
[0114] Determine the part-of-speech tag group of each part-of-speech distinction through the global part-of-speech distinction, all part-of-speech tagging features and all part-of-speech distinctions;
[0115] The sequence formed by all the part-of-speech tag groups is used as the part-of-speech tag vector of the Tibetan text information.
[0116] It should be noted that the global part-of-speech discrimination described in the present application represents a description of the overall distribution of part-of-speech discrimination in Tibetan text information. As a preferred embodiment, the global part-of-speech discrimination of the Tibetan text information can be determined based on all part-of-speech discriminations in the following manner, namely, taking the mean of all part-of-speech discriminations and using the obtained mean as the global part-of-speech discrimination of the Tibetan text information.
[0117] In specific implementation, the following method can be used to determine the part-of-speech tag subvector corresponding to the Tibetan text content of each part-of-speech tagging feature through the global part-of-speech discrimination, all part-of-speech tagging features and all part-of-speech discriminations, namely: select a part-of-speech discrimination from all the part-of-speech discriminations as the selected part-of-speech discrimination, and then select the Tibetan text content in which the Tibetan keyword corresponding to the selected part-of-speech discrimination appears in the Tibetan text content from all the Tibetan text contents, so as to form a set of all the obtained Tibetan text contents as a Tibetan text content set, and then multiply all the values in the part-of-speech tagging features of each Tibetan text content in the Tibetan text content set by the selected part-of-speech discrimination, and then sum all the multiplied values, and use the summed value as the part-of-speech recognition amount, wherein the Tibetan keyword corresponding to the part-of-speech recognition amount and the Tibetan keyword corresponding to the selected part-of-speech discrimination are the same, and then the word of each Tibetan text content is multiplied by the selected part-of-speech discrimination. All values in the part-of-speech tagging feature are linearly scaled to obtain multiple tag scaling sequences, wherein the linear scaling can use Min-Max normalization to adjust all values in each tag scaling sequence to between 0 and 10, and the mean of the maximum value and the minimum value in each tag scaling sequence is calculated to obtain multiple means, and then the largest value is selected from the multiple means, and the selected largest value is used as the part-of-speech determination value, and then all values in the part-of-speech tagging feature of each Tibetan text content in the Tibetan text content set are multiplied by the global part-of-speech distinction, and then all multiplied values are summed, and the summed value is used as the part-of-speech distinction threshold, and then the part-of-speech recognition amount, the part-of-speech distinction threshold and the part-of-speech determination value are combined into a triple, and the triple is used as the part-of-speech tag group of the selected part-of-speech distinction, and the part-of-speech tag group of the remaining part-of-speech distinction is continued to be determined. In other embodiments, other methods can also be used to achieve this, which will not be repeated here.
[0118] It should be noted that the arrangement order of each part-of-speech tag group of the part-of-speech tag vector in the present application is based on the order of the Tibetan keywords corresponding to the part-of-speech distinction in the Tibetan text information.
[0119] In addition, it should be noted that there are 10 parts of speech in the Tibetan language in this application, namely nouns, verbs, adjectives, adverbs, pronouns, numerals, prepositions, conjunctions, auxiliary words, and interjections. The value range of nouns is set to [0,1), the value range of verbs is set to [1,2), the value range of adjectives is set to [2,3), the value range of adverbs is set to [3,4), the value range of pronouns is set to [4,5), the value range of numerals is set to [5,6), the value range of prepositions is set to [6,7), the value range of conjunctions is set to [7,8), the value range of auxiliary words is set to [8,9), and the value range of interjections is set to [9,10).
[0120] In a specific implementation, the adaptive annotation of the Tibetan text information according to the part-of-speech tag vector can be implemented in the following manner, namely: a part-of-speech tag group is selected from the part-of-speech tag vector as the selected part-of-speech tag group, if the part-of-speech recognition amount in the selected part-of-speech tag group is less than or equal to the part-of-speech distinction threshold in the selected part-of-speech tag group, the Tibetan characters of the Tibetan keywords corresponding to the part-of-speech recognition amount in the selected part-of-speech tag group in the Tibetan text information are marked as polysemous words, if the part-of-speech recognition amount in the selected part-of-speech tag group is greater than the part-of-speech distinction threshold in the selected part-of-speech tag group, the Tibetan keywords corresponding to the part-of-speech recognition amount in the selected part-of-speech tag group are marked as polysemous words in the Tibetan text information. The Tibetan characters in the Tibetan text information are marked as homographs, and the Tibetan keywords corresponding to the part-of-speech recognition amount in the selected part-of-speech tag group are marked with parts of speech in the Tibetan text information according to the value range of the part-of-speech determination value in the selected part-of-speech tag group. The above steps are repeated to continue marking the Tibetan characters in the Tibetan text information of the Tibetan keywords corresponding to the part-of-speech recognition amount in the remaining part-of-speech tag groups in the part-of-speech tag vector as polysemous words, and marking the parts of speech of the Tibetan keywords corresponding to the part-of-speech recognition amount in the remaining part-of-speech tag groups in the Tibetan text information. In other embodiments, other methods can also be used for implementation, which is not limited here.
[0121] In addition, in another aspect of the present application, in some embodiments, the present application provides a Tibetan text automatic annotation system, referring to Figure 4 , which is a schematic diagram of exemplary hardware and / or software of a Tibetan text automatic annotation system according to some embodiments of the present application. The Tibetan text automatic annotation system 400 includes: an acquisition module 401, a processing module 402 and an execution module 403, which are described as follows:
[0122] Acquisition module 401, in this application, acquisition module 401 is mainly used to acquire Tibetan text content in different scenarios, and then form Tibetan text information from all acquired Tibetan text content;
[0123] Processing module 402, in the present application, is used to extract semantic features from the Tibetan text information to obtain multiple semantic representation vectors, and determine the label granularity of each Tibetan character in the Tibetan text information when annotating according to the semantic similarity between the semantic representation vectors and the part-of-speech distribution of each Tibetan character in all Tibetan text contents in the full text;
[0124] It should be noted that the processing module 402 in the present application is also used to determine the part-of-speech tagging features of each Tibetan text content according to all label granularities and all semantic representation vectors;
[0125] In addition, in a specific implementation, the processing module 402 in the present application is also used to extract key features of the Tibetan text information to obtain part-of-speech vectors of different Tibetan keywords, and determine the part-of-speech distinction of different Tibetan keywords in the annotation process according to the distance dependency relationship of each part-of-speech vector in the full text and the part-of-speech tags of each Tibetan text content;
[0126] Execution module 403, in the present application, execution module 403 is mainly used to determine the part-of-speech tag vector of the Tibetan text information through all part-of-speech tag features and all part-of-speech distinctions, and then adaptively tag the Tibetan text information according to the part-of-speech tag vector.
[0127] In addition, the present application also provides a computer device, which includes a memory and a processor, wherein the memory stores a code, and the processor is configured to obtain the code and execute the above-mentioned Tibetan text automatic annotation method.
[0128] In some embodiments, reference Figure 5 , which is a schematic diagram of the structure of a computer device using the automatic Tibetan text annotation method according to some embodiments of the present application. The automatic Tibetan text annotation method in the above embodiment can be Figure 5 The computer device 500 shown in the figure is implemented, and the computer device 500 includes at least one processor 501, a communication bus 502, a memory 503 and at least one communication interface 504.
[0129] The processor 501 may be a general-purpose central processing unit (CPU), an application-specific integrated circuit (ASIC), or one or more processors for controlling the execution of the automatic Tibetan text annotation method in the present application.
[0130] The communication bus 502 may be used to transmit information between the above-mentioned components.
[0131] The memory 503 may be a read-only memory (ROM) or other types of static storage devices that can store static information and instructions, a random access memory (RAM) or other types of dynamic storage devices that can store information and instructions, or an electrically erasable programmable read-only memory (EEPROM), a compact disc read-only memory (CD-ROM) or other optical disc storage, an optical disc storage (including a compressed optical disc, a laser disc, an optical disc, a digital versatile disc, a Blu-ray disc, etc.), a magnetic disk or other magnetic storage device, or any other medium that can be used to carry or store the desired program code in the form of an instruction or data structure and can be accessed by a computer, but is not limited thereto. The memory 503 may exist independently and be connected to the processor 501 via the communication bus 502. The memory 503 may also be integrated with the processor 501.
[0132] The memory 503 is used to store the program code for executing the solution of the present application, and the execution is controlled by the processor 501. The processor 501 is used to execute the program code stored in the memory 503. The program code may include one or more software modules. The method described in the above method embodiment can be implemented by the processor 501 and one or more software modules in the program code in the memory 503.
[0133] The communication interface 504 uses any transceiver or other device for communicating with other devices or communication networks, such as Ethernet, radio access network (RAN), wireless local area networks (WLAN), etc.
[0134] In a specific implementation, as an embodiment, a computer device may include multiple processors, each of which may be a single-CPU processor or a multi-CPU processor. The processor here may refer to one or more devices, circuits, and / or processing cores for processing data (e.g., computer program instructions).
[0135] The above-mentioned computer device may be a general-purpose computer device or a special-purpose computer device. In a specific implementation, the computer device may be a desktop computer, a portable computer, a network server, a personal digital assistant (PDA), a mobile phone, a tablet computer, a wireless terminal device, a communication device or an embedded device. The embodiment of the present application does not limit the type of computer device.
[0136] In addition, the present application also provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, the above-mentioned Tibetan text automatic annotation method is implemented.
[0137] In summary, in an automatic Tibetan text annotation method and annotation system disclosed in an embodiment of the present application, first, Tibetan text contents in different scenarios are acquired, and then all the acquired Tibetan text contents are combined into Tibetan text information; semantic feature extraction is performed on the Tibetan text information to obtain multiple semantic representation vectors; the label granularity of each Tibetan character in the Tibetan text information when annotating is determined according to the semantic similarity between each semantic representation vector and the part-of-speech distribution of each Tibetan character in all Tibetan text contents in the full text, and then the part-of-speech tagging features of each Tibetan text content are determined according to all label granularities and all semantic representation vectors; key feature extraction is performed on the Tibetan text information to obtain part-of-speech vectors of different Tibetan keywords, and the part-of-speech distinction of different Tibetan keywords in the annotation process is determined according to the distance dependency relationship of each part-of-speech vector in the full text and the part-of-speech tag of each Tibetan text content; the part-of-speech tag vector of the Tibetan text information is determined through all the part-of-speech tagging features and all the part-of-speech distinctions, and then the Tibetan text information is adaptively annotated according to the part-of-speech tag vector. The solution of the present application can realize the part-of-speech tagging of Tibetan polysemous words and homonyms, thereby improving the accuracy of Tibetan part-of-speech judgment.
[0138] Although the preferred embodiments of the present application have been described, those skilled in the art may make other changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications falling within the scope of the present application.
[0139] Obviously, those skilled in the art can make various changes and modifications to the present application without departing from the spirit and scope of the present application. Thus, if these modifications and variations of the present application fall within the scope of the claims of the present application and their equivalents, the present application is also intended to include these modifications and variations.
Claims
1. A method for automatic annotation of Tibetan text, characterized in that: The steps include: Acquire Tibetan text contents in different scenarios, and then combine all acquired Tibetan text contents into Tibetan text information; Extracting semantic features from the Tibetan text information to obtain a plurality of semantic representation vectors, and determining a label granularity for marking each Tibetan character in the Tibetan text information according to the semantic similarity between the semantic representation vectors and the part-of-speech distribution of each Tibetan character in all Tibetan text contents in the entire text; Determine the part-of-speech tagging features of each Tibetan text content based on all label granularities and all semantic representation vectors; Extract key features from the Tibetan text information to obtain part-of-speech vectors of different Tibetan keywords, and determine the part-of-speech distinction of different Tibetan keywords in the annotation process according to the distance dependency relationship of each part-of-speech vector in the full text and the part-of-speech tags of each Tibetan text content; Determining a part-of-speech tag vector of the Tibetan text information by using all part-of-speech tag features and all part-of-speech distinctions, and then adaptively tagging the Tibetan text information according to the part-of-speech tag vector; Among them, the part-of-speech tagging features of each Tibetan text content are determined based on all label granularities and all semantic representation vectors, including: Linear fitting is performed on all tag particle sizes to obtain a tag particle size curve; Determine the part-of-speech tag vector of each Tibetan text content according to the tag granularity curve and all semantic representation vectors; Determine the part-of-speech tagging features of each Tibetan text content through all part-of-speech tag vectors and all tag granularities; The key features of the Tibetan text information are extracted to obtain the part-of-speech vectors of different Tibetan keywords, which specifically include: Converting all Tibetan text contents in the Tibetan text information into a Tibetan text content sequence; Determining the context co-occurrence degree of each Tibetan character in the Tibetan text information according to the Tibetan text content sequence and a preset context window; Determine the part-of-speech vectors of different Tibetan keywords through all contextual co-occurrences; Among them, the part-of-speech distinction of different Tibetan keywords in the annotation process is determined according to the distance dependency relationship of each part-of-speech vector in the full text and the part-of-speech tags of each Tibetan text content, which specifically includes: Determine the distance dependency of each part-of-speech vector in the full text; Select a Tibetan text content as the selected Tibetan text content; Determine the number of successive occurrences of all Tibetan keywords in the selected Tibetan text content; Determine the part-of-speech distinction of all Tibetan keywords in the selected Tibetan text content during the tagging process according to the part-of-speech tags and all subsequent times of the selected Tibetan text content; Continue to determine the part-of-speech distinction of all Tibetan keywords in the remaining Tibetan text content during the annotation process; The method of determining the number of subsequent occurrences of all Tibetan keywords in the selected Tibetan text content specifically includes: The total number of times each Tibetan keyword in the selected Tibetan text content subsequently appears in the selected Tibetan text content is calculated, and the total number of times each Tibetan keyword subsequently appears is used as the subsequent number of the corresponding Tibetan keyword.
2. The method according to claim 1, characterized in that The semantic features of the Tibetan text information are extracted to obtain multiple semantic representation vectors, specifically including: Converting all Tibetan text contents in the Tibetan text information into a Tibetan text vector set; A bidirectional long short-term memory network based on an attention mechanism extracts multiple semantic representation vectors from the Tibetan text vector set.
3. The method according to claim 1, characterized in that Determining the label granularity of each Tibetan character in the Tibetan text information when marking according to the semantic similarity between the semantic representation vectors and the part-of-speech distribution of each Tibetan character in the Tibetan text content specifically includes: Determine the semantic similarity between each semantic representation vector; Determine the semantic block features of each semantic representation vector corresponding to the Tibetan sentence according to all semantic similarities; Determine the part-of-speech granularity feature of the corresponding Tibetan characters through the part-of-speech distribution of each Tibetan character in the full text of all Tibetan text contents; The label granularity of each Tibetan character in the Tibetan text information when marking is determined according to the semantic block features of each Tibetan sentence and the part-of-speech granularity features of each Tibetan character.
4. The method according to claim 1, characterized in that Determining the part-of-speech tag vector of the Tibetan text information by using all the part-of-speech tagging features and all the part-of-speech distinctions specifically includes: Determining the global part-of-speech distinction of the Tibetan text information according to all part-of-speech distinctions; Determine the part-of-speech tag group of each part-of-speech distinction through the global part-of-speech distinction, all part-of-speech tagging features and all part-of-speech distinctions; The sequence formed by all the part-of-speech tag groups is used as the part-of-speech tag vector of the Tibetan text information.
5. A Tibetan text automatic annotation system, characterized in that: The Tibetan text automatic annotation system includes: An acquisition module, used to acquire Tibetan text content in different scenarios, and then combine all acquired Tibetan text content into Tibetan text information; A processing module is used to extract semantic features from the Tibetan text information to obtain multiple semantic representation vectors, and determine the label granularity of each Tibetan character in the Tibetan text information when marking according to the semantic similarity between the semantic representation vectors and the part-of-speech distribution of each Tibetan character in all Tibetan text contents in the full text; The processing module is further used to determine the part-of-speech tagging features of each Tibetan text content according to all label granularities and all semantic representation vectors; The processing module is further used to extract key features from the Tibetan text information to obtain part-of-speech vectors of different Tibetan keywords, and determine the part-of-speech distinction of different Tibetan keywords in the annotation process according to the distance dependency relationship of each part-of-speech vector in the full text and the part-of-speech tags of each Tibetan text content; An execution module, configured to determine a part-of-speech tag vector of the Tibetan text information through all part-of-speech tag features and all part-of-speech distinctions, and then adaptively tag the Tibetan text information according to the part-of-speech tag vector; Among them, the part-of-speech tagging features of each Tibetan text content are determined based on all label granularities and all semantic representation vectors, including: Linear fitting is performed on all tag particle sizes to obtain a tag particle size curve; Determine the part-of-speech tag vector of each Tibetan text content according to the tag granularity curve and all semantic representation vectors; Determine the part-of-speech tagging features of each Tibetan text content through all part-of-speech tag vectors and all tag granularities; The key features of the Tibetan text information are extracted to obtain the part-of-speech vectors of different Tibetan keywords, which specifically include: Converting all Tibetan text contents in the Tibetan text information into a Tibetan text content sequence; Determining the context co-occurrence degree of each Tibetan character in the Tibetan text information according to the Tibetan text content sequence and a preset context window; Determine the part-of-speech vectors of different Tibetan keywords through all contextual co-occurrences; Among them, the part-of-speech distinction of different Tibetan keywords in the annotation process is determined according to the distance dependency relationship of each part-of-speech vector in the full text and the part-of-speech tags of each Tibetan text content, which specifically includes: Determine the distance dependency of each part-of-speech vector in the full text; Select a Tibetan text content as the selected Tibetan text content; Determine the number of successive occurrences of all Tibetan keywords in the selected Tibetan text content; Determine the part-of-speech distinction of all Tibetan keywords in the selected Tibetan text content during the tagging process according to the part-of-speech tags and all subsequent times of the selected Tibetan text content; Continue to determine the part-of-speech distinction of all Tibetan keywords in the remaining Tibetan text content during the annotation process; The method of determining the number of subsequent occurrences of all Tibetan keywords in the selected Tibetan text content specifically includes: The total number of times each Tibetan keyword in the selected Tibetan text content subsequently appears in the selected Tibetan text content is calculated, and the total number of times each Tibetan keyword subsequently appears is used as the subsequent number of the corresponding Tibetan keyword.
6. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that: When the processor executes the computer program, the method for automatic annotation of Tibetan text according to any one of claims 1 to 4 is implemented.
7. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the method for automatic Tibetan text annotation according to any one of claims 1 to 4 is implemented.
