Tibetan language pre-training language model training method, system and device and medium

Through multi-dimensional knowledge enhancement methods, including dynamic multi-head attention mechanism, part-of-word and phrase information, social media data and prompt learning strategies, the Tibetan pre-trained language model is optimized, which solves the problem of insufficient performance of the Tibetan pre-trained language model. Improves its performance and adaptability in natural language processing tasks.

CN119026608APending Publication Date: 2024-11-26TIBET UNIV
View PDF 0 Cites 3 Cited by

Patent Information

Application Number
CN202410922617.2
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-07-10
Publication Date
2024-11-26

AI Technical Summary

Technical Problem

Due to the small data scale, the existing Tibetan pre-trained language model fails to fully learn the unique grammatical structure, vocabulary characteristics and cultural background of Tibetan language, resulting in limited generalization ability and insufficient performance.

Method used

Using multi-dimensional knowledge enhancement methods, including using dynamic multi-head attention mechanisms, introducing part-of-word and phrase information, and training Tibetan pre-trained language models through social media domain data and prompt learning strategies to optimize their in specific downstream tasks. performance.

Benefits of technology

The ability of the Tibetan pre-trained language model to understand the meaning and contextual relationships of Tibetan texts is improved, and its performance in natural language processing tasks is enhanced, especially in social media fields, and it is adapted to different application scenarios.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119026608A_ABST
    Figure CN119026608A_ABST
Patent Text Reader

Abstract

The invention provides a Tibetan language pre-training language model training method, system and device and a medium, and relates to the field of natural language processing. The method comprises the following steps: firstly, pre-training a Tibetan pre-training language model by using data containing Tibetan grammar, vocabularies and cultural background; then, on the data level, enhancing the capturing capability of the model on the sentence core semantics and the internal relation by utilizing a dynamic multi-head attention mechanism; on the language knowledge level, part-of-speech and phrase information is introduced, and the ability of understanding complex contexts and subtle characteristics of Tibetan sentences is enhanced. And finally, optimizing the performance of the model for a specific downstream task through social media data and a prompt learning strategy. According to the scheme, the accuracy and efficiency of the Tibetan language pre-training language model during Tibetan language processing are comprehensively improved, and different context and task requirements are met.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present application relates to the field of natural language processing, and more specifically, to a training method, system, device and medium for a Tibetan pre-trained language model. Background Art

[0002] With the rapid development of natural language processing technology, pre-trained language models have become an important foundation for natural language processing tasks. However, for the specific language of Tibetan, most of the existing pre-trained language models are trained based on small-scale data and fail to fully learn the unique grammatical structure, vocabulary characteristics and cultural background of Tibetan. In other words, if only limited data is relied on, the directly trained Tibetan pre-trained language model may not be able to fully capture the specificity of the Tibetan language, and its generalization ability will also be limited. Therefore, how to improve the performance of the Tibetan pre-trained language model has become an urgent problem to be solved. Summary of the invention

[0003] The purpose of this application is to provide a training method, system, device and medium for a Tibetan pre-trained language model, aiming to improve the performance of the Tibetan pre-trained language model through multi-dimensional knowledge enhancement.

[0004] This application is implemented as follows:

[0005] In a first aspect, the present application provides a method for training a Tibetan pre-trained language model, comprising the following steps: pre-training the Tibetan pre-trained language model using Tibetan training data, wherein the Tibetan training data contains Tibetan-specific grammatical structures, vocabulary features, and cultural background. Performing knowledge enhancement processing on the Tibetan pre-trained language model at the data level, including using a dynamic multi-head attention mechanism to dynamically adjust the number of attention heads according to the length of the sentence to enhance the Tibetan pre-trained language model's ability to capture the core semantics and internal connections of the sentence. Performing knowledge enhancement processing on the Tibetan pre-trained language model at the Tibetan language knowledge level, including enhancing the ability to understand the complex context and subtle features of Tibetan sentences by introducing part of speech and phrase information into the Tibetan pre-trained language model. Performing knowledge enhancement processing on the Tibetan pre-trained language model through social media domain data and prompt learning strategies to optimize the performance of specific downstream tasks.

[0006] Further, based on the aforementioned scheme, the specific downstream task is a Tibetan microblog sentiment classification task; the knowledge enhancement processing of the Tibetan pre-trained language model is performed through social media field data and prompt learning strategies to optimize the performance of the specific downstream task, including: using social media-related Tibetan microblog data to train the Tibetan pre-trained language model; using a sentiment dictionary combined with a TF-IDF weight calculation algorithm to obtain the sentiment polarity of each word in the Tibetan microblog data, and incorporating the sentiment polarity into the training process of the Tibetan pre-trained language model to improve the sentiment classification performance; applying a prompt learning strategy to convert the Tibetan microblog sentiment classification task into a fill-in-the-blank task to match the mechanism of the Tibetan pre-trained language model to process natural language.

[0007] Further, based on the aforementioned scheme, the calculation process of the sentiment polarity includes: translating the Tibetan words in the vocabulary of the Tibetan pre-trained language model into corresponding English words; using an English sentiment dictionary tool to calculate the positive sentiment scores and negative sentiment scores of the translated English words, and accumulating the sentiment scores according to the number of translated English words to obtain the final sentiment polarity score of the Tibetan words; using the TF-IDF weight calculation algorithm to calculate the weight of each word in the Tibetan vocabulary to adjust the comprehensive score of the Tibetan sentiment polarity according to the weight to optimize the accuracy of sentiment classification.

[0008] Further, based on the above scheme, the TF-IDF weight calculation algorithm includes: calculating the frequency TF of Tibetan words in the text, and calculating the importance IDF of the words in the entire prediction library; multiplying the frequency TF and the importance IDF to obtain the TF-IDF weight value of each Tibetan word.

[0009] Furthermore, based on the aforementioned scheme, the dynamic multi-head attention mechanism includes: dynamically determining the number of attention heads according to the length of the input sentence; performing independent attention calculations on the query Q, key K and value V, with each head independently learning and revealing the unique semantic connection in the sentence; merging the results of different heads and transforming them through a weight matrix to form a comprehensive output representation.

[0010] Further, based on the aforementioned scheme, the prompt learning strategy includes: pre-designing an input template, the input template including a placeholder for indicating the focus of the Tibetan pre-trained language model; pre-constructing an answer space mapping to map the prediction results to specific labels or categories; embedding the actual input text into the input template to form a complete input sequence; and sending the input text combined with the input template into the Tibetan language pre-trained model to predict the content of the placeholder position.

[0011] In a second aspect, the present application provides a training system for a Tibetan pre-trained language model, which includes:

[0012] The first training module is configured to: pre-train the Tibetan pre-trained language model using Tibetan training data, wherein the Tibetan training data contains Tibetan-specific grammatical structures, vocabulary features and cultural background. The second training module is configured to: perform knowledge enhancement processing on the Tibetan pre-trained language model at the data level, including using a dynamic multi-head attention mechanism to dynamically adjust the number of attention heads according to the length of the sentence to enhance the Tibetan pre-trained language model's ability to capture the core semantics and internal connections of the sentence. The third training module is configured to: perform knowledge enhancement processing on the Tibetan pre-trained language model at the Tibetan language knowledge level, including enhancing the ability to understand the complex context and subtle features of Tibetan sentences by introducing part of speech and phrase information into the Tibetan pre-trained language model. The fourth training module is configured to: perform knowledge enhancement processing on the Tibetan pre-trained language model through social media domain data and prompt learning strategies to optimize the performance of specific downstream tasks.

[0013] Furthermore, the present application also provides a method for using the system based on the training system of the Tibetan pre-trained language model, wherein the system includes a memory for storing Tibetan training data, and a processor for executing the processing of the first training module, the second training module, the third training module and the fourth training module. The method for using the system includes: loading the Tibetan training data into the memory; executing the pre-training process through the processor, and performing knowledge enhancement processing at the data level and the Tibetan language knowledge level; applying the enhanced Tibetan pre-trained language model to the natural language processing task.

[0014] In a third aspect, the present application provides an electronic device, comprising a memory for storing one or more programs; a processor; when the one or more programs are executed by the processor, a method as described in any one of the first aspects is implemented.

[0015] In a fourth aspect, the present application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements a method as described in any one of the above-mentioned first aspects.

[0016] Compared with the prior art, the present invention has at least the following advantages or beneficial effects:

[0017] This application proposes a training method for a Tibetan pre-trained language model, which, through pre-training and knowledge enhancement processing, can more accurately understand the meaning and contextual relationship of Tibetan text, thereby improving the performance of the Tibetan pre-trained language model in Tibetan natural language processing tasks. Among them, the dynamic multi-head attention mechanism and the processing method of introducing part-of-speech and phrase information make the Tibetan pre-trained language model more flexible in processing sentences of different lengths and different types of texts. In addition, the Tibetan pre-trained language model can also be fine-tuned according to the needs of specific tasks to adapt to different application scenarios. And through the optimization processing of social media field data and prompt learning strategies, the Tibetan pre-trained language model performs well in tasks in the social media field. This helps to meet the growing demand for social media text processing. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the embodiments will be briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present application and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.

[0019] Figure 1 A flowchart of an embodiment of a training method for a Tibetan pre-trained language model of the present application;

[0020] Figure 2 This is a flow chart of performing knowledge enhancement on the Tibetan pre-trained language model in one embodiment of the present application;

[0021] Figure 3 A framework diagram of domain knowledge and prompt enhancement in one embodiment of the present application;

[0022] Figure 4 This is a schematic diagram showing an enhanced architecture in one embodiment of the present application;

[0023] Figure 5 This is a structural block diagram of an embodiment of a training system for a Tibetan pre-trained language model of the present application;

[0024] Figure 6 A structural block diagram of an electronic device provided in an embodiment of the present application.

[0025] Icons: 101, first training module; 102, second training module; 103, third training module; 104, fourth training module; 201, processor; 202, memory; 203, communication interface. DETAILED DESCRIPTION

[0026] In order to make the purpose, technical solution and advantages of the embodiments of the present application clearer, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, rather than all the embodiments. The components of the embodiments of the present application described and shown in the drawings here can be arranged and designed in various different configurations.

[0027] In conjunction with the accompanying drawings, some embodiments of the present application are described in detail below. In the absence of conflict, the following embodiments and features in the embodiments can be combined with each other.

[0028] Example 1

[0029] The embodiment of the present application provides a training method for a Tibetan pre-trained language model, which can improve the performance of the Tibetan pre-trained language model through multi-dimensional knowledge enhancement.

[0030] See also Figure 1 The training method of the Tibetan pre-trained language model comprises the following steps:

[0031] Step S101: pre-training a Tibetan pre-trained language model using Tibetan training data, wherein the Tibetan training data includes Tibetan-specific grammatical structures, vocabulary features and cultural background.

[0032] In step S101, the Tibetan pre-trained language model is first initialized and trained using large-scale Tibetan training data. These training data usually include various text corpora, such as news, novels, web page texts, etc., and these text corpora must have the unique grammatical structure, vocabulary characteristics and cultural background of Tibetan. Through this step, the Tibetan pre-trained language model can learn the basic vocabulary, grammatical rules and cultural background-related knowledge of Tibetan, providing a foundation for subsequent higher-level tasks. That is, step S101 is the basis of the entire training process, which provides the initial Tibetan pre-trained language model parameters for subsequent steps. These parameters will be further adjusted and optimized in subsequent steps to adapt to more complex tasks.

[0033] Specifically, during the pre-training process, the Tibetan pre-trained language model can adopt a self-supervised learning task (such as a masked language model) to train the model by predicting masked words in the text. In this way, the Tibetan pre-trained language model can learn the vocabulary and grammatical rules in the text and generate the corresponding word vector representation. Alternatively, the Tibetan pre-trained language model can learn the basic language structure and knowledge of Tibetan by predicting masked words in the text, predicting the order of text paragraphs or entire sentences, etc. through unsupervised learning.

[0034] Step S102: performing knowledge enhancement processing on the Tibetan pre-trained language model at the data level, including using a dynamic multi-head attention mechanism to dynamically adjust the number of attention heads according to the length of the sentence, so as to improve the Tibetan pre-trained language model's ability to capture the core semantics and internal connections of the sentence.

[0035] Step S102 aims to further enhance the Tibetan pre-trained language model's ability to capture the core semantics and internal connections of sentences. By using a dynamic multi-head attention mechanism, the Tibetan pre-trained language model can dynamically adjust the number of attention heads according to the length of the sentence. In this way, the Tibetan pre-trained language model can focus more flexibly on the key information in the sentence, rather than focusing on a fixed part. The dynamic multi-head attention mechanism allows the Tibetan pre-trained language model to focus on different parts of the input simultaneously in multiple different representation subspaces. For long sentences, the Tibetan pre-trained language model can use more heads to capture richer semantic information; for short sentences, fewer heads can be used to avoid overfitting. This mechanism enables the Tibetan pre-trained language model to handle sentences of different lengths more flexibly.

[0036] That is, step S102 is performed based on the initial Tibetan pre-trained language model provided in step S101, and improves the structure and training mechanism of the Tibetan pre-trained language model so that the Tibetan pre-trained language model has stronger representation ability at the data level. This enhanced representation ability will directly affect the Tibetan pre-trained language model's ability to understand complex contexts and subtle features in subsequent steps.

[0037] Step S103: performing knowledge enhancement processing on the Tibetan pre-trained language model at the Tibetan language knowledge level, including enhancing the ability to understand the complex context and subtle features of Tibetan sentences by introducing part of speech and phrase information into the Tibetan pre-trained language model.

[0038] Step S103 focuses on improving the Tibetan pre-trained language model's ability to understand the complex context and subtle features of Tibetan sentences. By introducing part-of-speech and phrase information into the Tibetan pre-trained language model, the Tibetan pre-trained language model can understand the structure and meaning of sentences more deeply. This information helps the model identify the key components in a sentence and understand the logical relationship between them. During the training process, the Tibetan pre-trained language model will learn how to represent part-of-speech and phrase information, and combine this information with the vocabulary and grammatical rules in the text. In this way, when the Tibetan pre-trained language model processes a sentence, it can not only focus on vocabulary and grammatical rules, but also on the part-of-speech and phrase structure in the sentence. This information helps the Tibetan pre-trained language model understand the meaning and contextual relationship of the sentence more accurately.

[0039] That is, step S103 is performed on the basis of step S102, and it further enriches the representation capability of the Tibetan pre-trained language model. By introducing more language knowledge, the Tibetan pre-trained language model can better process complex Tibetan texts and achieve better performance in subsequent tasks.

[0040] Step S104: Perform knowledge enhancement processing on the Tibetan pre-trained language model through social media domain data and prompt learning strategies to optimize the performance of specific downstream tasks.

[0041] Step S104 focuses on optimizing the performance of the Tibetan pre-trained language model in specific downstream tasks (such as Tibetan microblog sentiment classification). By using social media domain data and prompt learning strategies, the Tibetan pre-trained language model can learn knowledge and representations related to specific tasks. This helps the Tibetan pre-trained language model better adapt to actual application scenarios and achieve better performance. Specifically, in social media domain data, the Tibetan pre-trained language model can learn vocabulary, grammatical rules, and expressions related to social media. These data usually contain a large amount of user-generated content, such as microblogs, comments, etc., which have unique language styles and expressions. By training the Tibetan pre-trained language model to process these data, the Tibetan pre-trained language model can learn knowledge and representations related to social media. At the same time, the prompt learning strategy guides the Tibetan pre-trained language model to train by converting the task into a fill-in-the-blank task. This strategy can help the Tibetan pre-trained language model better understand the nature and requirements of the task and learn specific knowledge and representations related to the task. In this way, the Tibetan pre-trained language model can achieve better performance on specific tasks. In other words, the knowledge enhancement processing in step S104 enables the model to better adapt to the task requirements in the social media field. By learning and applying data and prompt learning strategies in the social media field, the model shows better performance in specific downstream tasks (such as Tibetan microblog sentiment classification). This optimization not only improves the practicality of the model, but also lays the foundation for its application in a wider range of fields.

[0042] That is, step S104 is performed on the basis of step S103, and it further fine-tunes the Tibetan pre-trained language model for specific tasks. By introducing social media domain data and prompt learning strategies, the Tibetan pre-trained language model can learn specific knowledge and representation methods related to the task, thereby improving performance on specific tasks. In other words, considering the uniqueness of texts in the social media field, the above embodiment also uses data and prompt learning strategies in the social media field to further optimize the model. Texts in the social media field often have unique language styles and expressions, which requires special processing methods when processing these texts. By training the Tibetan pre-trained language model to process data in the social media field, and combining the prompt learning strategy, the Tibetan pre-trained language model can learn knowledge and representation methods related to social media, thereby improving the performance of the model in specific downstream tasks.

[0043] In summary, through pre-training and knowledge enhancement processing, the Tibetan pre-trained language model in the above embodiment can more accurately understand the meaning and contextual relationship of Tibetan text, thereby improving the performance of the Tibetan pre-trained language model in Tibetan natural language processing tasks. Among them, the dynamic multi-head attention mechanism and the processing method of introducing part-of-speech and phrase information make the Tibetan pre-trained language model more flexible in processing sentences of different lengths and different types of texts. In addition, the Tibetan pre-trained language model can also be fine-tuned according to the needs of specific tasks to adapt to different application scenarios. And through the optimization processing of social media field data and prompt learning strategies, the Tibetan pre-trained language model performs well in tasks in the social media field. This helps to meet the growing demand for social media text processing.

[0044] Based on the aforementioned scheme, in one implementation of the present application, the specific downstream task is a Tibetan microblog sentiment classification task; the knowledge enhancement processing of the Tibetan pre-trained language model is performed through social media field data and prompt learning strategies to optimize the performance of the specific downstream task, including: using social media-related Tibetan microblog data to train the Tibetan pre-trained language model; using a sentiment dictionary combined with a TF-IDF weight calculation algorithm to obtain the sentiment polarity of each word in the Tibetan microblog data, and incorporating the sentiment polarity into the training process of the Tibetan pre-trained language model to improve the sentiment classification performance; applying a prompt learning strategy to convert the Tibetan microblog sentiment classification task into a fill-in-the-blank task to match the mechanism of the Tibetan pre-trained language model to process natural language.

[0045] The above implementation method proposes a method for knowledge enhancement of the Tibetan pre-trained language model for the Tibetan microblog sentiment classification task. Based on the previous pre-training and knowledge enhancement processing, this implementation method is specially optimized for the Tibetan microblog data in the social media field to improve the performance in the Tibetan microblog sentiment classification task.

[0046] Specifically, in the step of training the Tibetan pre-trained language model using Tibetan Weibo data related to social media, a large amount of Tibetan Weibo data is collected, which should cover a variety of emotional tendencies and expressions. Then, these Tibetan Weibo data are used as new training data to further train the Tibetan pre-trained language model, so that the Tibetan pre-trained language model can adapt to the language style and expression in the social media field.

[0047] In the step of using the sentiment dictionary combined with the TF-IDF weight calculation algorithm to obtain the sentiment polarity of each word in the Tibetan microblog data, first construct or obtain a sentiment dictionary suitable for Tibetan, which should contain common sentiment words in Tibetan and their corresponding sentiment polarities. Then use the TF-IDF (term frequency-inverse document frequency) weight calculation algorithm to calculate the importance of each word in the document in the Tibetan microblog data. Then, combine the sentiment dictionary and TF-IDF weight to assign a sentiment polarity value to each word, which reflects the sentiment tendency expressed by the word in a specific microblog.

[0048] In the process of incorporating sentiment polarity into the training of the Tibetan pre-trained language model, the sentiment polarity values ​​calculated above are used as supervisory information to guide the Tibetan pre-trained language model to learn how to understand and identify sentiment expressions in Tibetan microblogs. In this way, the Tibetan pre-trained language model can more accurately capture the sentiment information in the text, thereby improving the performance in the sentiment classification task.

[0049] In the step of applying the hint learning strategy to convert the Tibetan microblog sentiment classification task into a fill-in-the-blank task, the hint learning strategy is a strategy for converting a natural language processing task into a fill-in-the-blank task. In the Tibetan microblog sentiment classification task, the task can be converted into a form of predicting missing sentiment tags or sentiment keywords in the microblog. In this way, the Tibetan pre-trained language model can process text in a more natural way and understand the sentiment information in the text more accurately.

[0050] In summary, in the above implementation, by using social media-related Tibetan microblog data and sentiment dictionaries for training, the Tibetan pre-trained language model can more accurately understand and identify the emotional expressions in Tibetan microblogs, thereby improving the performance in sentiment classification tasks. Moreover, since the Tibetan pre-trained language model uses a large amount of social media field data during the training process, it can better adapt to the language style and expression methods in the social media field and improve its performance in tasks in this field. In addition, the prompt learning strategy in the above implementation enables the Tibetan pre-trained language model to handle different forms of sentiment classification tasks, improving the flexibility and scalability of the Tibetan pre-trained language model. At the same time, the sentiment dictionary and TF-IDF weight calculation algorithm can also be updated and expanded as needed to adapt to different application scenarios.

[0051] Based on the aforementioned scheme, in one implementation of the present application, the calculation process of the sentiment polarity includes: translating the Tibetan words in the vocabulary of the Tibetan pre-trained language model into corresponding English words; using an English sentiment dictionary tool to calculate the positive sentiment scores and negative sentiment scores of the translated English words, and accumulating the sentiment scores according to the number of translated English words to obtain the final sentiment polarity score of the Tibetan words; using the TF-IDF weight calculation algorithm to calculate the weight of each word in the Tibetan vocabulary to adjust the comprehensive score of the Tibetan sentiment polarity according to the weight to optimize the accuracy of sentiment classification.

[0052] In the above implementation, first, the Tibetan words in the vocabulary of the Tibetan pre-trained language model are translated into corresponding English words. This step can be achieved through existing machine translation tools or bilingual dictionaries to ensure the accuracy of the translation. For each English word translated, an English sentiment dictionary tool (such as VADER, SentiWordNet, etc.) can be used to calculate its positive sentiment score and negative sentiment score. These sentiment dictionary tools are usually trained based on a large number of corpora and can provide relatively accurate sentiment tendency scores. Then, the sentiment scores are accumulated according to the number of English words translated. Specifically, the positive sentiment score of each English word can be added to obtain a total positive sentiment score, and the negative sentiment score of each English word can be added to obtain a total negative sentiment score. Finally, the final sentiment polarity score of the Tibetan word will be obtained, which is usually determined by comparing the size of the positive sentiment score and the negative sentiment score. If the positive sentiment score is higher than the negative sentiment score, it is determined that the Tibetan word has a positive sentiment polarity; otherwise, it has a negative sentiment polarity.

[0053] After obtaining the sentiment polarity scores of Tibetan words, the TF-IDF (term frequency-inverse document frequency) weight calculation algorithm can be used to calculate the weight of each word in the Tibetan vocabulary in a specific Tibetan Weibo document. The TF-IDF weight can reflect the importance of a word in a document, that is, the frequency of its appearance in the document and its rarity in the entire document collection. Then, the comprehensive score of Tibetan sentiment polarity is adjusted according to the calculated TF-IDF weight. Specifically, for each Tibetan word, its sentiment polarity score is multiplied by the corresponding TF-IDF weight, and then the adjusted scores of all words are accumulated to obtain the comprehensive sentiment polarity score of the entire Tibetan Weibo document. In this way, the degree of influence of different words on sentiment polarity in the document can be more accurately reflected, thereby optimizing the accuracy of sentiment classification.

[0054] In summary, in the above implementation, by combining the English sentiment dictionary and the TF-IDF weight calculation algorithm, the sentiment polarity of Tibetan words can be calculated more accurately, and the Tibetan microblog documents can be sentimentally classified based on these sentiment polarities. This method can make full use of the rich resources of the English sentiment dictionary and the document specificity of the TF-IDF weight to improve the accuracy of sentiment classification. In addition, the above implementation is not only applicable to the Tibetan microblog sentiment classification task, but can also be extended to sentiment analysis tasks in other fields. As long as the Tibetan word translation and English sentiment dictionary tools in the corresponding field can be obtained, this method can be used for sentiment analysis. Among them, by calculating the sentiment polarity scores of Tibetan words, it is possible to more intuitively understand which words have an important impact on the sentiment classification results. This helps to enhance the interpretability of the model and make the sentiment classification results more understandable and credible. In addition, the proposal and implementation of the above implementation method helps to promote the development of cross-language sentiment analysis technology. By translating Tibetan words into English and using the English sentiment dictionary tool for calculation, a new idea and method for cross-language sentiment analysis is provided. This has important theoretical and practical significance for dealing with sentiment analysis problems in a multilingual environment.

[0055] Based on the above scheme, in one implementation of the present application, the TF-IDF weight calculation algorithm includes: calculating the frequency TF of Tibetan words in the text, and calculating the importance IDF of the words in the entire expectation library; multiplying the frequency TF and the importance IDF to obtain the TF-IDF weight value of each Tibetan word.

[0056] In the above implementation, the TF-IDF weight value can reflect the importance of a word in a specific text, thereby optimizing the accuracy of sentiment classification. The frequency TF of Tibetan words in a text is referred to as term frequency TF (Term Frequency). Term frequency refers to the number of times a Tibetan word appears in a given text. Specifically, it is necessary to calculate the number of times each Tibetan word appears in the text and divide it by the total number of words in the text to obtain the term frequency TF of the word. The importance IDF of a word in the entire expected library refers to the inverse document frequency IDF (Inverse Document Frequency), which is used to measure the importance of a word in the entire expected library. If a word appears in many documents in the expected library, its IDF value will be relatively low; conversely, if it only appears in a few documents, its IDF value will be relatively high. Among them, when calculating IDF, it is first necessary to count the number of documents in the expected library that contain a certain word, then divide this number by the total number of documents in the expected library, and then take the logarithm to obtain the IDF value of the word. Finally, the term frequency TF and the inverse document frequency IDF are multiplied to obtain the TF-IDF weight value of each Tibetan word. This weight value takes into account both the frequency of occurrence of the word in the text and the importance of the word in the entire prediction library, so it can more accurately reflect the weight of the word in the text.

[0057] In summary, by using the TF-IDF weight calculation algorithm, the weight value of each Tibetan word in the text can be obtained, and the impact of the word on the sentiment classification result can be more accurately evaluated. This helps trainers better identify the keywords that play a decisive role in sentiment polarity, thereby improving the accuracy of sentiment classification. In addition, the TF-IDF weight calculation algorithm can automatically adapt to texts of different lengths. For longer texts, the word frequency TF will be relatively low, but the inverse document frequency IDF may be high; for shorter texts, the word frequency TF may be high, but the inverse document frequency IDF may be low. By comprehensively considering the two factors of TF and IDF, a more accurate weight value can be obtained. Also, because the TF-IDF weight calculation algorithm is based on statistics, it has a certain robustness to noise and outliers in the text. Even if there are some words or spelling errors in the text that are not related to sentiment polarity, it will not have much impact on the calculation of the weight value, which can further improve the performance of the Tibetan pre-trained language model.

[0058] Based on the aforementioned scheme, in one implementation of the present application, the dynamic multi-head attention mechanism includes: dynamically determining the number of attention heads according to the length of the input sentence; performing independent attention calculations on the query Q, key K and value V, with each head independently learning and revealing the unique semantic connections in the sentence; merging the results of different heads and transforming them through a weight matrix to form a comprehensive output representation.

[0059] Typically, the number of heads in a multi-head attention mechanism is fixed, but in some cases, different sentence lengths may require different numbers of heads to better capture semantic connections. To this end, in the above implementation, a mechanism is designed that can dynamically adjust the number of heads based on the length of the sentence. The number of heads can be determined based on the length of the sentence or some complexity measure (such as the lexical diversity in the sentence). For example, shorter sentences may only require fewer heads, while longer or complex sentences may require more heads. For each determined number of heads, attention calculations are performed independently on the query Q (Query), key K (Key), and value V (Value). Each head learns and reveals unique semantic connections in the sentence. In this step, each head calculates an attention score that represents the degree of match between the query Q and the key K. These scores are then used to weight the value V to produce the output representation of each head. After all heads have completed the attention calculation, the results need to be merged into a unified output representation. This is usually achieved by concatenating or averaging the output representations of each head. However, direct merging may result in information redundancy or loss. To solve this problem, a weight matrix conversion step is introduced in the above implementation. Specifically, the above implementation linearly transforms the combined output representation using a learnable weight matrix to produce the final comprehensive output representation. This weight matrix is ​​automatically adjusted during the training process to ensure that the output representation can accurately reflect the overall semantics of the sentence.

[0060] In summary, by dynamically determining the number of attention heads, the mechanism is able to better adapt to sentences of different lengths, thereby more effectively capturing semantic information in sentences. At the same time, each head independently learns and reveals unique semantic connections in sentences, which enables the mechanism to capture richer semantic information and understand the meaning of sentences more accurately. Moreover, through the weight matrix conversion step, the mechanism is able to automatically adjust the output representation to adapt to different input sentences. This makes the Tibetan pre-trained language model more robust in the face of different inputs. In addition, since each head independently learns and reveals unique semantic connections in sentences, trainers can more easily analyze the role of each head and understand how the Tibetan pre-trained language model captures and understands the semantic information in sentences. This helps trainers better understand and improve the Tibetan pre-trained language model.

[0061] Based on the aforementioned scheme, in one implementation of the present application, the prompt learning strategy includes: pre-designing an input template, the input template including a placeholder for indicating the focus of the Tibetan pre-trained language model; pre-constructing an answer space mapping to map the prediction results to specific labels or categories; embedding the actual input text into the input template to form a complete input sequence; and sending the input text combined with the input template into the Tibetan language pre-trained model to predict the content of the placeholder position.

[0062] The above implementation method aims to guide the Tibetan pre-trained language model to focus on a specific focus and accurately map the prediction results to specific labels or categories by designing specific input templates and answer space mapping. Specifically, it is first necessary to pre-design a set of input templates. These templates are used to guide the focus of the Tibetan pre-trained language model when processing text. The template contains placeholders used to indicate the focus of the Tibetan pre-trained language model, which will be replaced by the actual input text in subsequent steps. The design of the input template needs to take into account the characteristics of the Tibetan language and task requirements. For example, in the sentiment analysis task, the template can be designed as "[text placeholder] expresses the sentiment of [emotion placeholder]", where "[text placeholder]" will be replaced by the actual input text, and "[emotion placeholder]" is the part that the Tibetan pre-trained language model needs to predict.

[0063] Next, in order to convert the prediction results of the Tibetan pre-trained language model into specific labels or categories, an answer space mapping needs to be constructed in advance. This mapping defines the possible value range of each placeholder and the correspondence between these values ​​and specific labels or categories. In the sentiment analysis task, the answer space mapping may define the value range of "[sentiment placeholder]" as sentiment labels such as "positive" and "negative", and specify the mapping relationship between each value and the corresponding sentiment label.

[0064] Then, when the Tibetan pre-trained language model is running, the actual input text needs to be embedded into the pre-designed input template, replacing the placeholders in the template to form a complete input sequence. This process ensures that the Tibetan pre-trained language model can focus on the focus preset by the trainer when processing text.

[0065] Finally, the input text combined with the input template is fed into the Tibetan pre-trained language model. The Tibetan pre-trained language model will predict the content of the placeholder position based on the content of the input sequence. The prediction process can be generative (such as generating text) or categorical (such as selecting a value in the answer space mapping). During the prediction process, the Tibetan pre-trained language model will use the language knowledge and contextual information learned in the pre-training stage, combined with the guidance information in the input template, to infer and predict the content of the placeholder position.

[0066] In summary, by designing the input template and answer space mapping, we can purposefully guide the Tibetan pre-trained language model to focus on a specific focus and limit the value range of its prediction results, thereby improving the prediction accuracy of the Tibetan pre-trained language model. In addition, the introduction of input template and answer space mapping makes the prediction process of the Tibetan pre-trained language model more transparent and explainable. Thus, we can clearly see how the Tibetan pre-trained language model makes predictions based on the input text and the guidance information in the template.

[0067] In order to enable those skilled in the art to understand the present application more intuitively, a specific example will be used here to illustrate.

[0068] Due to the limited number of Tibetan pre-training resources, it is difficult to obtain Tibetan training data. Therefore, in order to enable the Tibetan pre-training model to learn more Tibetan knowledge based on limited data, this application enhances the Tibetan language model from multiple dimensions. Figure 2 As shown, after the Tibetan pre-trained language model is pre-trained using Tibetan training data, knowledge enhancement is performed from the data level, language knowledge level and domain level in turn, and prompt learning enhancement is performed to achieve the purpose of multi-dimensional knowledge enhancement of the Tibetan pre-trained language model.

[0069] The first step is to train the Tibetan pre-trained language model and perform knowledge enhancement at the data level. For example, the 4.5GB Tibetan corpus provided by the evaluation task of the 21st China Conference on Computational Linguistics (CCL 2022) can be used for pre-training. This batch of corpus mainly comes from platforms such as Wikipedia and WeChat public accounts.

[0070] Among them, in order to overcome the attention bias problem caused by the fixed number of attention heads in the traditional model when the sentences are of different lengths, the number of attention heads can be dynamically adjusted according to the sentence feature quantity. This method aims to enable the Tibetan pre-trained language model to adaptively adjust the attention according to the length of the sentence, so as to more accurately capture the core semantics and internal connections of the sentence. By implementing the strategy of dynamically adjusting the number of attention heads, the Tibetan pre-trained language model can show a more delicate feature understanding ability when processing sentences with different feature densities. For short sentences with simpler features, a smaller number of attention heads is selected, so that the Tibetan pre-trained language model can focus on key information and avoid being disturbed by irrelevant details; for long sentences with complex features, a larger number of attention heads is selected so that the Tibetan pre-trained language model can interpret more complex semantic structures and capture long-distance dependencies. The specific formula of its dynamic multi-head attention mechanism is shown below.

[0071]

[0072] DMA(Q,k,V)=Concat(head1,...,head h )W h (2)

[0073] if len_token>mh=h i else h=h s (3)

[0074] In formula (1), head i represents the result of the attention mechanism operation corresponding to the i-th head number. The DMA(Q,K,V) function refers to the result of each attention head head i , …, head h Concatenate and combine with the weight matrix W h The result of multiplication, where W h is the weight parameter in the multi-head attention mechanism, and the number of heads h is dynamically determined based on the length of the input sentence len_token. Through this mechanism, the Tibetan pre-trained language model can flexibly adjust the number of multi-head attentions according to the complexity of the internal features of the sentence to better adapt to sentences of different lengths. The dynamic multi-head attention mechanism uses the sentence length information to adaptively select the appropriate number of heads, regardless of whether the number of heads is h. l For longer sentences or h s Corresponding to shorter sentences, the mechanism performs independent attention calculations on the query (Q), key (K), and value (V), and each head learns and reveals unique semantic connections in the sentence. By merging the results of different heads and transforming them through the weight matrix, a comprehensive output representation is finally formed.

[0075] In the dynamic multi-head attention mechanism, the input sentence is first processed by word embedding and converted into three matrices: query (Q), key (K), and value (V). Then, the Tibetan pre-trained language model compares the word length (len_token) of the sentence with the preset threshold m to determine the scale of attention heads to be used. If the sentence length is less than the threshold m, it indicates that the information is relatively concentrated. At this time, the Tibetan pre-trained language model selects a smaller number of heads for processing, that is, a smaller number of heads h is used. s At this time, the Q, K, and V matrices are passed through the left route and pass through h s The linear transformation of the layers is designed to extract the key information of the sentence from different dimensions to obtain a more refined feature representation. After feature extraction, the outputs of each head are concatenated and integrated into the final output through subsequent mapping operations. On the contrary, if the sentence length reaches or exceeds the threshold m, it means that the sentence contains more information, and the model will enable more heads h. lThis configuration is performed via the right route. In addition to using more layers of linear transformations, the processing flow is similar to that of shorter sentences. The output is also finally obtained through concatenation and mapping.

[0076] Please continue reading Figure 2 , and then the second step is to perform knowledge enhancement processing on the Tibetan pre-trained language model at the Tibetan language knowledge level. The first step is to analyze a large amount of general data, that is, to enhance the pre-trained model only at the data level. However, Tibetan is a minority language with unique grammar, vocabulary and language characteristics, and the amount of Tibetan data is relatively small. Enhancement at the data level alone may not be enough to fully capture the complex context and subtle characteristics of its sentences.

[0077] The importance of language knowledge in natural language processing tasks is self-evident. In a variety of application scenarios, the integration of deep language knowledge can significantly improve the model's understanding ability and task performance. For example, the use of part-of-speech tagging can more effectively identify key information in document information extraction; with the help of syntactic analysis, in sentiment classification, you can gain in-depth insights into sentence structure and semantic connections, thereby more accurately evaluating sentiment tendencies; integrating semantic features and labels into the classification model can more comprehensively capture the semantic hierarchy of the text, thereby improving the accuracy of classification. In view of this, the present application further introduces part-of-speech and phrase information into the pre-trained model on the basis of the first step. As shown in Table 1, 1G of data with language knowledge (part-of-speech, phrases) can be used for secondary training. Part-of-speech and phrase information provide important clues about sentence structure, grammatical rules, and semantic relationships for the Tibetan pre-trained language model. Part-of-speech tagging helps the Tibetan pre-trained language model understand the grammatical function and syntactic relationship of words in sentences, while phrase analysis provides deeper semantic information.

[0078]

[0079] Table 1. Table of features of part-of-speech phrases

[0080] By integrating this deep language knowledge, the Tibetan pre-trained language model can more accurately adapt to the characteristics of Tibetan sentiment classification tasks, improve the model's ability to understand and process Tibetan sentences, and thus achieve better performance.

[0081] Please continue reading Figure 3, and then the third step is to perform knowledge enhancement processing on the Tibetan pre-trained language model through social media domain data and prompt learning strategies. The first two steps focus on demonstrating the method of performing knowledge enhancement on the Tibetan pre-trained language model at the data and language levels to improve the Tibetan pre-trained language model's ability to understand Tibetan. It mainly uses large-scale general domain data for training, so that the Tibetan pre-trained language model can learn general data features and language representations. However, when the Tibetan pre-trained language model is used in a specific downstream task, the effect of the model in this field can be enhanced through a unique data enhancement model. For example, the downstream task is Weibo sentiment classification. Since Weibo sentiment classification belongs to the social media field, its text characteristics are different from traditional news articles and written language. Therefore, in order to more accurately process the Tibetan Weibo sentiment classification task, this application will perform knowledge enhancement on the Tibetan pre-trained language model from the domain level based on the Tibetan pre-trained language model in the first two steps. First, we crawl more Tibetan microblog data related to social media from Tibetan microblog websites to retrain the Tibetan pre-trained language model to further optimize the microblog sentiment classification model; secondly, we use the sentiment dictionary to obtain the sentiment polarity of each word in the Tibetan microblog data, and integrate the sentiment polarity into the model training process to deepen the model's ability to learn domain knowledge and improve the performance of downstream sentiment classification; in addition, considering the MLM task mode adopted by the Tibetan pre-trained language model in the training process, we use the prompt learning strategy to transform the traditional Tibetan microblog sentiment classification task into a fill-in-the-blank task to better fit the essential mechanism of the pre-trained language model in processing natural language. The overall architecture diagram is shown in the figure below: Figure 3 As shown in the figure. MLM (Masked Language Model) refers to a pre-training method of the BERT model. It is a mask-based prediction task that aims to enable the model to learn to fill in masked text fragments. During the training process, some words in the input text sequence are randomly selected and masked, and then the model needs to predict these masked words based on contextual information. When predicting masked words, the model can use other words in the context to obtain semantic and grammatical clues.

[0082] exist Figure 3 In Step A, we use SentiWordNet (an English-based sentiment vocabulary ontology) to annotate the sentiment polarity of social media data, and re-pre-train the Tibetan pre-trained language model in the first and second steps. However, SentiWordNet is an English sentiment polarity dictionary and cannot be directly used to calculate the sentiment polarity of Tibetan words. Therefore, in order to obtain the sentiment polarity of Tibetan words, this section uses the following steps and techniques:

[0083] Step 1: Word Translation

[0084] This section will retrain the Tibetan pre-trained language model based on the first and second steps. In order to better utilize the sentiment dictionary, each Tibetan word in the vocabulary of the Tibetan pre-trained language model is translated into the corresponding English word by calling the Microsoft Translator interface. For example, Translation: said proudly).

[0085] Step 2: SentiWordNet calculates sentiment polarity

[0086] After obtaining the English words corresponding to each word in the Tibetan vocabulary through Step 1, the SentiWordNet tool is used to calculate the positive sentiment score (PosScore) and negative sentiment score (NegScore) of each English word translated. However, since the number of words translated from Tibetan words to English is greater than or equal to 1, the sentiment score is calculated according to Formula 4. The polarity of each English word is first calculated and then accumulated. To calculate the sentiment score, first calculate the sentiment polarity comprehensive score of said, then calculate the sentiment polarity comprehensive score of proudly, and finally add the two scores together to get The final sentiment score of the hidden word.

[0087]

[0088] Step 3: TF-IDF weight calculation

[0089] In order to avoid translation errors or inaccurate calculation of English sentiment polarity in Step 1 and Step 2, the TF-IDF (Term Frequency-Inverse Document Frequency) weight calculation method is introduced to adjust the comprehensive score of Tibetan sentiment polarity according to the weight.

[0090] Since the social media texts obtained by crawling the Tibetan Weibo website are used for retraining, the TF-IDF method is used on this dataset to calculate the weight of each word in the Tibetan vocabulary, where TF represents the frequency of the Tibetan word in the text and IDF represents the importance of the word in the entire corpus. TF represents the frequency of the word in the text, that is, the number of times the word appears in the Tibetan text. IDF represents the importance of the word in the entire corpus, that is, how many documents the word appears in. The specific formula is as follows:

[0091]

[0092] tfidf j =tf×id f (7)

[0093] In Formula 5, n represents the number of times a word appears in a document, and N represents the total number of times all words appear in a document. Normalization is used to eliminate the difference in the length of documents. In Formula 6, D represents the total number of documents in the corpus, and d represents the number of documents in which a word appears in the corpus. The 1 in the formula is to prevent the denominator from being 0, and lg is the logarithm with base 10, which has the effect of enhancing the discrimination.

[0094] By using Formula 7, the TF-IDF value of each Tibetan word in the Tibetan text in the social media domain can be calculated.

[0095] Step 4: Tibetan sentiment polarity

[0096] Using the first three steps, we can obtain the comprehensive sentiment polarity score and TF-IDF weight value of each Tibetan word in the Tibetan dictionary. Multiplying the weight value with the comprehensive sentiment polarity score will give us the final Tibetan sentiment polarity score, as shown in formulas 8 and 9.

[0097] TibetanScore j =TotalScore j ×tfidf j (8)

[0098]

[0099] In Formula 9, considering the deviation of sentiment polarity calculation, the final Tibetan sentiment polarity score (TibetanScore) greater than 0.1 is set as positive sentiment; the result less than -0.1 is set as negative sentiment; the result in the interval [-0.1.0.1] indicates that the difference between the positive and negative aspects of the text is small, so the text sentiment is judged to be neutral. Through the above steps and techniques, the final sentiment polarity of each Tibetan word in the Tibetan vocabulary can be obtained and used in the subsequent Tibetan pre-training language model training task.

[0100] exist Figure 3 In Step B, the language model is combined with prompt learning for sentiment classification. Prompt learning adds templates to the input data to guide the model to learn specific information, which is used to guide the model to focus on specific contextual information or task goals. It can better utilize the knowledge of the Tibetan pre-trained language model, better understand and learn the associated features between texts, and thus improve the performance of classification tasks. In addition, by using appropriate prompts, the difference between pre-training and fine-tuning is reduced, so that the model can achieve good results by fine-tuning on a small number of samples.

[0101] The process of prompt learning includes several key steps, which aims to optimize the understanding and prediction ability of the Tibetan pre-trained language model for specific tasks through a specific structured approach. The following is a detailed description of the prompt learning process:

[0102] (1) Designing input templates: First, in order to guide the prediction of the Tibetan pre-trained language model, an input template needs to be designed. The template usually contains one or more placeholders (such as [MASK]) to indicate the focus of the model. These templates are pre-defined sentence structures that are used to guide the Tibetan pre-trained language model to understand the context of the input data and task requirements. For example, in a sentiment analysis task, the template may be: "The sentiment expressed by this sentence is [MASK]".

[0103] (2) Constructing answer space mapping: Secondly, in order to map the prediction results of the Tibetan pre-trained language model to specific labels or categories, it is necessary to define an answer space. This space contains all possible prediction results, each of which corresponds to a specific label or category. For example, in the sentiment analysis task, the answer space may include three categories: "positive", "neutral" and "negative";

[0104] (3) Combining input text with template: embed the actual input text into the designed template to form a complete input sequence. This process involves placing the key information in the input text into the placeholders of the Tibetan pre-trained language model to generate structured input for model prediction, such as: [MASK] Translation: I wish our collectors a happy New Year and all their wishes come true. This sentence conveys a [MASK] emotion.

[0110] (4) Prediction using the Tibetan pre-trained language model: The input text combined with the template is fed into the pre-trained language model, which predicts the content of the placeholder. This step is the core of prompt learning. The Tibetan pre-trained language model will predict the most likely value of the placeholder based on its learned knowledge and the context of the input sequence. The architecture is as follows: Figure 4 shown.

[0111] Example 2

[0112] See also Figure 5 The present application embodiment provides a training system for a Tibetan pre-trained language model, which includes:

[0113] The first training module 101 is configured to: pre-train the Tibetan pre-trained language model using Tibetan training data, wherein the Tibetan training data contains Tibetan-specific grammatical structures, vocabulary features and cultural background. The second training module 102 is configured to: perform knowledge enhancement processing on the Tibetan pre-trained language model at the data level, including using a dynamic multi-head attention mechanism to dynamically adjust the number of attention heads according to the length of the sentence to enhance the Tibetan pre-trained language model's ability to capture the core semantics and internal connections of the sentence. The third training module 103 is configured to: perform knowledge enhancement processing on the Tibetan pre-trained language model at the Tibetan language knowledge level, including enhancing the ability to understand the complex context and subtle features of Tibetan sentences by introducing part of speech and phrase information into the Tibetan pre-trained language model. The fourth training module 104 is configured to: perform knowledge enhancement processing on the Tibetan pre-trained language model through social media domain data and prompt learning strategies to optimize the performance of specific downstream tasks.

[0114] Furthermore, the present application example also provides a method for using the system based on the training system of the Tibetan pre-trained language model, wherein the system includes a memory for storing Tibetan training data, and a processor for executing the processing of the first training module 101, the second training module 102, the third training module 103 and the fourth training module 104. The method for using the system includes: loading Tibetan training data into the memory; executing the pre-training process through the processor, and performing knowledge enhancement processing at the data level and the Tibetan language knowledge level; applying the enhanced Tibetan pre-trained language model to the natural language processing task.

[0115] For the specific implementation process of the above system, please refer to the training method of a Tibetan pre-trained language model provided in Example 1, which will not be repeated here.

[0116] Example 3

[0117] See also Figure 6 , the embodiment of the present application provides an electronic device, which includes at least one processor 201 and at least one memory 202; wherein the processor 201 and the memory 202 are directly connected to each other, or communicate with each other through a communication interface 203, or are electrically connected through one or more communication buses or signal lines to achieve data transmission or interaction; the memory 202 stores program instructions that can be executed by the processor 201, and the processor 201 calls the program instructions to execute a training method for a Tibetan pre-trained language model. For example, the following is implemented:

[0118] Pre-train the Tibetan pre-trained language model using Tibetan training data, which contains Tibetan-specific grammatical structures, vocabulary features, and cultural background. Perform knowledge enhancement on the Tibetan pre-trained language model at the data level, including using a dynamic multi-head attention mechanism to dynamically adjust the number of attention heads according to the length of the sentence to improve the Tibetan pre-trained language model's ability to capture the core semantics and internal connections of the sentence. Perform knowledge enhancement on the Tibetan pre-trained language model at the Tibetan language knowledge level, including introducing part-of-speech and phrase information into the Tibetan pre-trained language model to enhance the ability to understand the complex context and subtle features of Tibetan sentences. Perform knowledge enhancement on the Tibetan pre-trained language model through social media domain data and prompt learning strategies to optimize the performance of specific downstream tasks.

[0119] Among them, the memory 202 can be, but is not limited to, a random access memory (RAM), a read only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable read-only memory (EEPROM), etc.

[0120] The processor 201 may be an integrated circuit chip with signal processing capability. The processor 201 may be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it may also be a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components.

[0121] Understandably, Figure 6 The structure shown is for illustration only. The electronic device may also include Figure 6 More or fewer components as shown, or with Figure 6 Different configurations are shown. Figure 6 Each component shown in the figure can be implemented by hardware, software or a combination thereof.

[0122] Example 4

[0123] The present application provides a computer-readable storage medium having a computer program stored thereon. When the computer program is executed by the processor 201, a training method for a Tibetan pre-trained language model is implemented.

[0124] For example, to implement:

[0125] Pre-train the Tibetan pre-trained language model using Tibetan training data, which contains Tibetan-specific grammatical structures, vocabulary features, and cultural background. Perform knowledge enhancement on the Tibetan pre-trained language model at the data level, including using a dynamic multi-head attention mechanism to dynamically adjust the number of attention heads according to the length of the sentence to improve the Tibetan pre-trained language model's ability to capture the core semantics and internal connections of the sentence. Perform knowledge enhancement on the Tibetan pre-trained language model at the Tibetan language knowledge level, including introducing part-of-speech and phrase information into the Tibetan pre-trained language model to enhance the ability to understand the complex context and subtle features of Tibetan sentences. Perform knowledge enhancement on the Tibetan pre-trained language model through social media domain data and prompt learning strategies to optimize the performance of specific downstream tasks.

[0126] If the above functions are implemented in the form of software function modules and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application can be essentially or partly embodied in the form of a software product that contributes to the prior art. The computer software product is stored in a storage medium and includes several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0127] It will be apparent to those skilled in the art that the present application is not limited to the details of the exemplary embodiments described above, and that the present application can be implemented in other specific forms without departing from the spirit or essential features of the present application. Therefore, the embodiments should be considered exemplary and non-limiting in all respects, and the scope of the present application is defined by the appended claims rather than the above description, and it is intended that all changes falling within the meaning and scope of the equivalent elements of the claims be included in the present application. Any reference numeral in a claim should not be considered as limiting the claim to which it relates.

Claims

1. A method for training a Tibetan pre-trained language model, characterized in that: The following steps are involved: Pre-training a Tibetan pre-trained language model using Tibetan training data, wherein the Tibetan training data includes grammatical structures, vocabulary characteristics, and cultural background unique to Tibetan; Perform knowledge enhancement processing on the Tibetan pre-trained language model at the data level, including using a dynamic multi-head attention mechanism to dynamically adjust the number of attention heads according to the length of the sentence, so as to improve the Tibetan pre-trained language model's ability to capture the core semantics and internal connections of the sentence; Enhance the Tibetan pre-trained language model at the Tibetan language knowledge level, including introducing part-of-speech and phrase information into the Tibetan pre-trained language model to enhance the ability to understand the complex context and subtle features of Tibetan sentences; The Tibetan pre-trained language model is knowledge-enhanced by social media domain data and prompt learning strategies to optimize the performance of specific downstream tasks.

2. The method for training a Tibetan pre-trained language model according to claim 1, characterized in that: The specific downstream task is a Tibetan microblog sentiment classification task; The Tibetan pre-trained language model is enhanced with knowledge through social media domain data and prompt learning strategies to optimize the performance of specific downstream tasks, including: Use Tibetan microblog data related to social media to train Tibetan pre-trained language model; Use the sentiment dictionary combined with the TF-IDF weight calculation algorithm to obtain the sentiment polarity of each word in the Tibetan Weibo data, and incorporate the sentiment polarity into the training process of the Tibetan pre-trained language model to improve the sentiment classification performance; The hint learning strategy is applied to transform the Tibetan microblog sentiment classification task into a fill-in-the-blank task to match the mechanism of Tibetan pre-trained language model in processing natural language.

3. The method for training a Tibetan pre-trained language model according to claim 2, characterized in that: The calculation process of the sentiment polarity includes: Translate the Tibetan words in the vocabulary of the Tibetan pre-trained language model into corresponding English words; Use the English sentiment dictionary tool to calculate the positive and negative sentiment scores of the translated English words, and add up the sentiment scores according to the number of translated English words to get the final sentiment polarity score of the Tibetan words; The TF-IDF weight calculation algorithm is used to calculate the weight of each word in the Tibetan vocabulary, so as to adjust the comprehensive score of Tibetan sentiment polarity according to the weight and optimize the accuracy of sentiment classification.

4. The method for training a Tibetan pre-trained language model according to claim 3, wherein: The TF-IDF weight calculation algorithm includes: Calculate the frequency TF of Tibetan words in the text, and calculate the importance IDF of the words in the entire database; Multiply the frequency TF and the importance IDF to get the TF-IDF weight value of each Tibetan word.

5. The method for training a Tibetan pre-trained language model according to claim 1, wherein: The dynamic multi-head attention mechanism includes: Dynamically determine the number of attention heads based on the length of the input sentence; Independent attention is computed for query Q, key K, and value V, and each head independently learns and reveals unique semantic connections in the sentence; The results of different numbers of heads are combined and transformed through the weight matrix to form a comprehensive output representation.

6. The method for training a Tibetan pre-trained language model according to claim 1, wherein: The suggested learning strategies include: Pre-designing an input template, the input template including a placeholder for indicating a focus of the Tibetan pre-trained language model; Pre-construct answer space mapping to map prediction results to specific labels or categories; Embed the actual input text into the input template to form a complete input sequence; The input text combined with the input template is fed into the Tibetan language pre-trained model to predict the content of the placeholder position.

7. A training system for a Tibetan pre-trained language model, characterized in that: include: A first training module is configured to: pre-train a Tibetan pre-trained language model using Tibetan training data, wherein the Tibetan training data includes Tibetan-specific grammatical structures, vocabulary features, and cultural background; The second training module is configured to: perform knowledge enhancement processing on the Tibetan pre-trained language model at the data level, including using a dynamic multi-head attention mechanism to dynamically adjust the number of attention heads according to the length of the sentence, so as to improve the Tibetan pre-trained language model's ability to capture the core semantics and internal connections of the sentence; The third training module is configured to: perform knowledge enhancement processing on the Tibetan pre-trained language model at the Tibetan language knowledge level, including enhancing the ability to understand the complex context and subtle features of Tibetan sentences by introducing part of speech and phrase information into the Tibetan pre-trained language model; The fourth training module is configured to perform knowledge enhancement processing on the Tibetan pre-trained language model through social media domain data and prompt learning strategies to optimize the performance of specific downstream tasks.

8. A method of using the system of claim 7, characterized in that: The system includes a memory for storing Tibetan training data, and a processor for executing the processing of the first training module, the second training module, the third training module and the fourth training module; The method includes: Load Tibetan training data into memory; The processor performs the pre-training process and performs knowledge enhancement processing at the data level and Tibetan language knowledge level; The enhanced Tibetan pre-trained language model is applied to natural language processing tasks.

9. An electronic device, characterized in that: include: A memory for storing one or more programs; processor; When the one or more programs are executed by the processor, the method according to any one of claims 1 to 6 is implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 6 is implemented.

Citation Information

Cited By

  • Natural language recognition processing method and device based on deep learning

    CN120297288A

  • Letter text analysis method driven by multilayer dynamic aggregation large language model

    CN120387459A

  • Semantic alignment method and related device

    CN121189494A