A vietnamese dependency syntax analysis method, system and electronic device based on multiple annotators

Abstract

The application relates to a Vietnamese dependency syntax analysis method and system based on multiple annotators, and belongs to the field of natural language processing. The application fine-tunes the parameters of an XLM RoBERTa model using unlabeled data. Then, an initial Vietnamese dependency syntax analysis model is trained using a UD tree library, a sentence is input into the model for analysis, and pseudo data containing noise is generated. The pseudo data is input into a pre-designed prompt template, a large language model DeepSeek is used for secondary annotation, noise data is gradually corrected through prompt learning, and high-quality annotation results are output. Finally, the pseudo data after secondary annotation is used as additional training corpus, combined with original annotation data, and a new Vietnamese syntax analysis model is trained. The application introduces a pseudo data enhancement and multiple annotator collaborative optimization mechanism, and significantly improves the performance of the model in the Vietnamese dependency syntax analysis task.

Description

Technical Field

[0001] The present invention relates to a Vietnamese dependency syntax analysis method, system, and electronic device based on multiple annotators, and belongs to the technical field of natural language processing. Background Art

[0002] Cultural and economic interactions between my country and Southeast Asian countries, including Vietnam, are becoming increasingly frequent and close. However, language differences can be a significant obstacle to smooth communication. Language, as a key element in the flow of thought, is the most natural and convenient way for people to exchange ideas and convey emotions. Therefore, in-depth research on the languages ​​of Southeast Asian countries is of irreplaceable importance for promoting communication and shared development between my country and neighboring countries. Therefore, research on dependency parsing methods for low-resource languages ​​in Southeast Asia, particularly Vietnamese, Thai, and Lao, has become increasingly crucial. By deeply analyzing the underlying syntactic structures within texts, we can more accurately grasp the connections within languages. Therefore, research on dependency parsing methods for low-resource languages ​​in Southeast Asia not only holds a significant position in the field of artificial intelligence, but also holds crucial practical significance for enhancing friendly cooperation between my country and Southeast Asian countries and strengthening regional communication and interaction.

[0003] In recent years, the rapid development of deep learning technology has driven progress in syntactic parsing tasks. However, deep learning methods rely on large amounts of annotated data, limiting their effectiveness in low-resource languages ​​such as Vietnamese. To overcome these challenges, researchers have proposed various solutions, among which data augmentation is considered an effective approach. By expanding the training dataset, data augmentation can help models better learn syntactic knowledge and improve their performance on low-resource languages. Traditional data augmentation methods typically rely on rules or simple substitution strategies, making it difficult to generate high-quality and diverse data. With the rise of large language models, leveraging their powerful language understanding and generation capabilities for data augmentation has become a new research direction. This paper designs a multi-annotator-based dependency parsing method for Vietnamese to improve the performance of dependency parsing in Vietnamese. As one of the major languages ​​in Southeast Asia, Vietnamese possesses unique grammatical structures and lexical features. It has a complex tonal system, rich word forms, and unique grammatical structures such as postpositioned verbs and frequent use of quantifiers. Research on dependency parsing in low-resource languages ​​such as Vietnamese remains relatively underdeveloped, primarily due to the lack of large-scale annotated data. Therefore, combining the advantages of traditional parsing models with those of large language models is a key issue. Summary of the Invention

[0004] The application provides a Vietnamese dependency syntax analysis method, system and electronic equipment based on multiple annotators to solve the Vietnamese dependency syntax tree bank conversion problem, improves the Vietnamese dependency syntax analysis performance, and achieves good experimental results in Vietnamese dependency syntax tree bank conversion.

[0005] The technical scheme of the application is as follows: a Vietnamese dependency syntax analysis method based on multiple annotators, and the specific steps of the method are as follows:

[0006] Step 1: download related Vietnamese dependency syntax UD tree bank as training corpus, VnDT tree bank as original corpus for constructing pseudo data, and collect Vietnamese unannotated data;

[0007] Step 2: fine-tune the XLM-RoBERTa model through the collected Vietnamese unannotated data, thereby enhancing the representation ability of the multilingual pre-training model;

[0008] Step 3: the UD tree bank is subjected to the fine-tuned XLM-RoBERTa model to obtain related embedding representation, character embedding representation and word-level embedding representation for splicing as the final input vector;

[0009] Step 4: encode the input vector through a three-layer bidirectional long short-term memory network to obtain its context representation, and use the context representation as the input of KAN;

[0010] Step 5: obtain the score of the dependency arc through the double-affine scoring layer, and finally decode the dependency syntax tree through the maximum spanning tree algorithm;

[0011] Step 6: use the dependency syntax analysis model constructed by the UD tree bank to parse 2000 sentences randomly extracted from the VnDT tree bank to obtain pseudo data containing noise;

[0012] Step 7: input the pseudo data into the large model DeepSeek according to the pre-designed prompt template for secondary annotation;

[0013] Step 8: use the pseudo data annotated by the secondary annotation as additional training corpus, combine the pseudo data with the UD tree bank, and retrain a new Vietnamese dependency syntax analysis model.

[0014] As a further scheme of the application, the specific steps of Step 1 are as follows:

[0015] Step 1.1: use the downloaded UD (Universal Dependencies) tree bank as training corpus, and use the VnDT tree bank as original corpus for constructing data.

[0016] Step1.2, the crawled Vietnamese web pages are subjected to rule extraction, deduplication, machine annotation and manual correction to form a Vietnamese text corpus, which is used as unlabeled data for constructing a model;

[0017] As a further scheme of the present application, the specific steps of Step2 are as follows:

[0018] Step2.1, download and select the original XLM-RoBERTa model as the base model through the huggingface website;

[0019] Step2.2, use the parameters in the original XLM-RoBERTa model as the starting point, and fine-tune the XLM-RoBERTa model parameters with Vietnamese unlabeled data.

[0020] As a further scheme of the present application, Step3 includes:

[0021] Step3.1, each word w i corresponding to the final input vector x i is represented by three parts: the first part is the character embedding representation which can capture the character-level information of the word; the second part is the representation generated by the XLM-RoBERTa model which provides higher-level context semantic information; the third part is the word-level embedding representation which provides basic semantic information of each word.

[0022] As a further scheme of the present application, the specific steps of Step4 are as follows:

[0023] Step4.1, input the vector x i through a three-layer bidirectional long short-term memory network for encoding, thereby obtaining the final context word representation h' i ;

[0024] Step4.2, input the context word representation h i After KAN processing, the low-dimensional core word representation and the modifier word representation

[0025] As a further scheme of the present application, Step5 includes:

[0026] Step5.1, input each word as the vector representation of the core word and the vector representation of the modifier word through a double affine layer to obtain the score S of the dependency arc;

[0027] Step 5.2: Use the maximum spanning tree algorithm to decode and find the dependency syntax tree with the highest score as the final syntactic analysis result.

[0028] As a further solution of the present invention, Step 6 includes:

[0029] Step 6.1. Input 2000 sentences randomly selected from the VnDT tree bank into the Vietnamese dependency parsing model trained by the UD tree bank to obtain pseudo data containing noise.

[0030] As a further solution of the present invention, Step 7 includes:

[0031] Step 7.1. Design a prompt template for the large model DeepSeek as required. The prompt template should include specific instructions and constraints on the output.

[0032] Step 7.2: Input 2000 pieces of pseudo data into the large model according to the prompt template to obtain the pseudo data after secondary annotation;

[0033] The present invention also provides a Vietnamese dependency syntax analysis system based on multiple annotators, the system comprising: a module for executing the above-mentioned Vietnamese dependency syntax analysis method based on multiple annotators.

[0034] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, the method for Vietnamese dependency parsing based on multiple annotators is implemented.

[0035] The beneficial effects of the present invention are:

[0036] 1. This invention combines the advantages of traditional models and large language models, and uses large language models to perform secondary annotation on pseudo data, thereby obtaining more accurate syntactic parsing results, significantly improving the quality of pseudo data, and thus effectively improving the performance of the Vietnamese syntactic analysis model.

[0037] 2. This paper fine-tunes the pre-trained language model XLM-RoBERTa using unlabeled Vietnamese data, improving the performance of Vietnamese dependency parsing. Replacing the traditional multi-layer perceptron with a KAN helps the model better capture complex syntactic features and provide a more accurate representation of grammatical structure.

[0038] 3. The Vietnamese dependency parsing method based on multiple annotators proposed in this paper has significantly improved the accuracy on the benchmark dataset. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] Figure 1 is a flow chart of the present invention; DETAILED DESCRIPTION

[0040] Example 1: Figure 1 As shown in FIG, a Vietnamese dependency parsing method based on multiple annotators is described, and the specific steps of the method are as follows:

[0041] a1. We crawl Vietnamese web pages from the website and generate a Vietnamese text corpus through rule extraction, deduplication, machine annotation, and manual proofreading to build unlabeled data. We download the Vietnamese treebank (UD) from the universal dependencies dataset as training data. We also download the VnDT treebank as the original pseudo-corpus.

[0042] a2. Use the crawled Vietnamese unlabeled data to fine-tune the XLM-RoBERTa model parameters to enhance the representation ability of the pre-trained model.

[0043] 1) Download and select the original XLM-RoBERTa model from the huggingface website as the base model;

[0044] 2) Fine-tune the model parameters of XML-RoBERTa using Vietnamese unlabeled data, without changing the parameter settings.

[0045] a3. After fine-tuning the XLM-RoBERTa model of the UD treebank, the relevant embedding representations, character embedding representations, and word-level embedding representations are concatenated as the final input vector;

[0046] 1) XLM-RoBERTa representation after splicing and fine-tuning Word embedding representation and word embedding representation Get each word w i The corresponding input vector x i ;

[0047]

[0048] a4. Encode the input vector through a three-layer bidirectional long short-term memory network to obtain the context word representation, which is used as the input of KAN;

[0049] 1) Input vector x i After encoding through a three-layer bidirectional long short-term memory network, the Vietnamese-related context word representation h′ is obtained i ;

[0050] h0h1…h n =BiLSTM(x0x1…x n ,θ BiLSTM ) (2)

[0051] 2) The core of the long short-term memory network layer is the cell state, denoted as C t The long short-term memory network deletes or adds information to the cell state through a gating mechanism. BiLSTM implements three gate calculations, namely the forget gate F t , input gate I t and output gate O t , used to protect and control cell state.

[0052] The forget gate is responsible for deciding how much of the cell state at the previous moment to retain in the cell state at the current moment.

[0053] F t =σ(W f ·[h t-1 ,x t ]+b f ) (3)

[0054] The input gate is responsible for deciding how much of the current input to keep in the current cell state. The sigmoid layer decides what value to update; the tanh layer creates a new cell state value vector Will be added to the state. t-1 Updated to C t , the old state and F t Multiply, discard the information that needs to be discarded, and add This completes the update of the cell state.

[0055] I t =σ(W i ·[h t-1 ,x t ]+b i )

[0056]

[0057] The output gate determines how much output the cell state has at the current moment. A sigmoid layer determines which portion of the cell state will be output. The cell state is processed through a tanh function to obtain a value between -1 and 1. This value is then multiplied by the output of the sigmoid gate, ultimately outputting only the portion that was determined to be output.

[0058]

[0059] Where W f 、W i 、W c 、W o 、b f 、b i 、b c and b oare all weight matrices; σ is the sigmoid activation function; x t is the input at time t, which represents the word vector corresponding to the t-th position.

[0060] 5) KAN uses context words to represent h' i As input. It is the word w i As the representation vector of the core word, It is the word w i As the representation vector of the modifier. KAN not only reduces the context-dependent representation w i More importantly, it retains semantically relevant information.

[0061]

[0062] a5. Use the dual affine attention mechanism for scoring and the maximum spanning tree for decoding to find the dependency syntax tree with the highest score as the final syntactic analysis result.

[0063] 1) The biaffine scoring layer is used to calculate the dependency arc score between words in a sentence, which represents the dependency arc score from the core word w j to the modifier w i In this layer, the score of the dependency arc is determined by the weight matrix U 1 OK. The scoring formula for the biaffine layer is: Note that all dependency arc scores can be calculated simultaneously using a matrix form.

[0064]

[0065] 2) The loss function is used to optimize the dependency parsing model to ensure that the model can accurately predict the dependency relationship between words and their dependency labels. Assume that word w i The correct core word is w j , the dependency label is l, then the loss function of the model is defined as follows:

[0066]

[0067] a6. Use the dependency parsing model built by the UD treebank to parse sentences randomly extracted from the VnDT treebank to obtain pseudo data containing noise;

[0068] a7. Input the pseudo data into the large model DeepSeek for secondary annotation according to the pre-designed prompt template;

[0069] a8. The pseudo data after secondary annotation is used as additional training corpus and merged with the UD tree bank to retrain a new Vietnamese dependency parsing model.

[0070] The application adopts a Vietnamese corpus preprocessed by web page crawling as unlabeled data, and selects a Vietnamese dependency tree library in a universal dependency (UD) and a Vietnamese VnDT tree library as experimental data. The application mainly adopts unlabeled attachment score (UAS) and labeled attachment score (LAS) as evaluation indexes of dependency syntax analysis performance. The detailed calculation methods of the two evaluation indexes are as follows:

[0071]

[0072] Among them, H refers to the total number of words with correct core words, HL refers to the total number of words with correct core words and dependency relation types, and T refers to the total number of words in the corpus.

[0073] In order to verify the effect of the method proposed in the application, two classical dependency syntax analysis methods are selected as benchmark models. 1) Language embedding model: additional language embedding representation is set to improve the understanding and representation ability of the model for Vietnamese. 2) Multi-task learning model: the syntax analysis of tree libraries with inconsistent annotation specifications is used as different tasks, and the idea of multi-task learning is used to simultaneously learn the syntax information between heterogeneous tree libraries. The model corresponding to the method of the application is a data augmentation model. The data augmentation model is a new syntax analysis model trained by using the pseudo data generated by the traditional model and the secondary annotation by the large model as additional training corpus. The application increases the representation of XLM-RoBERTa in the input layer, enhances the performance of the dependency syntax analysis base model, and uses KAN instead of the original MLP, which helps the model to better capture complex context information, especially in processing dependency syntax relations, which can provide more accurate syntax structure representation. Finally, the pseudo data after secondary annotation is combined with the UD tree library to form a large training set, which is used to train the Vietnamese dependency syntax analysis model.

[0074] The final experimental results are shown in Table 2. After adding the pseudo data re-labeled by the large model after iterative optimization to the training data, the parsing performance of the base model is significantly improved. This result proves the effectiveness of the method proposed in the application, especially after the strong reasoning ability of the large model is used to re-label the Vietnamese pseudo data generated by the traditional model, the quality of the pseudo data is improved, thereby further enhancing the performance of the model. Combining the reasoning ability of the large language model with the traditional dependency syntax analysis model can provide effective help in the dependency syntax parsing task.

[0075] Table 2 Comparison of other methods and the method of the present application on different models

[0076]

[0077] The specific embodiments of the present application are described above with reference to the accompanying drawings, but the present application is not limited to the above-described embodiments, and various changes can be made within the knowledge of those skilled in the art without departing from the spirit of the present application.

Claims

1. A multi-annotator-based Vietnamese dependency parsing method, characterized by: The method comprises the following steps: Step 1: Download the relevant Vietnamese dependency syntax UD treebank as training corpus and the VnDT treebank as the original corpus for constructing pseudo data, and collect Vietnamese unlabeled data at the same time; Step 2: Fine-tune the XLM-RoBERTa model using the collected Vietnamese unlabeled data to enhance the representation capabilities of the multilingual pre-trained model. Step 3: After fine-tuning the XLM-RoBERTa model of the UD treebank, the relevant embedding representations, character embedding representations, and word-level embedding representations are obtained and then concatenated as the final input vector; Step 4: Encode the input vector through a three-layer bidirectional long short-term memory network to obtain its contextual representation, which is used as the input of the KAN. Step 5: Obtain the scores of dependency arcs through biaffine layering, and finally decode them through the maximum spanning tree algorithm to obtain the dependency syntax tree; Step 6: Use the dependency parsing model built by the UD treebank to parse sentences randomly extracted from the VnDT treebank to obtain pseudo data containing noise; Step 7: Input the pseudo data into the large model DeepSeek for secondary annotation according to the pre-designed prompt template; Step 8: Merge the pseudo-data after secondary annotation into the UD treebank as additional training data to retrain a new Vietnamese dependency parsing model. Step 3 includes: Step 3.

1. Each word in the input layer The corresponding final input vector It consists of three parts: the first part is the character embedding representation , which can capture the character-level information of words; the second part is the representation generated by the XLM-RoBERTa model , which provides higher-level contextual semantic information; the third part is the word-level embedding representation , which provides basic semantic information of each word.

2. The Vietnamese dependency parsing method based on multiple annotators according to claim 1, characterized in that: The specific steps of Step 1 are as follows: Step 1.

1. Use the downloaded UD treebank as the training corpus and the VnDT treebank as the original data corpus. Step 1.2: The crawled Vietnamese web pages are subjected to rule extraction, deduplication, machine annotation, and manual proofreading to form a Vietnamese text corpus to build unlabeled data.

3. The Vietnamese dependency parsing method based on multiple annotators according to claim 1, characterized in that: The specific steps of Step 2 are as follows: Step 2.

1. Download and select the original XLM-RoBERTa model from the huggingface website as the base model. Step 2.2: Use the parameters in the original XLM-RoBERTa model as a starting point and fine-tune the XLM-RoBERTa model parameters using Vietnamese unlabeled data.

4. The Vietnamese dependency parsing method based on multiple annotators according to claim 1, characterized in that: The specific steps of Step 4 are as follows: Step 4.1, input vector The final context word representation is obtained by encoding through a three-layer bidirectional long short-term memory network. ; Step 4.2: Input context word representation After KAN processing, low-dimensional core word representation is generated and modifiers .

5. The Vietnamese dependency parsing method based on multiple annotators according to claim 4, characterized in that: The Step 5 includes: Step 5.1: Take each word as the vector representation of the core word and vector representation of modifiers After the double affine layer, the score S of the dependency arc is obtained; Step 5.2: Use the maximum spanning tree algorithm to decode and find the dependency syntax tree with the highest score as the final syntactic analysis result.

6. The Vietnamese dependency parsing method based on multiple annotators according to claim 1, characterized in that: The Step 6 includes: Step 6.

1. Input several sentences randomly selected from the VnDT tree bank into the Vietnamese dependency parsing model trained by the UD tree bank to obtain pseudo data containing noise.

7. The Vietnamese dependency parsing method based on multiple annotators according to claim 1, characterized in that: The Step 7 includes: Step 7.

1. Design a prompt template for the large model DeepSeek as required. The prompt template should include specific instructions and constraints on the output. Step 7.2: Input several pieces of pseudo data into the large model according to the prompt template to obtain the pseudo data after secondary annotation.

8. A Vietnamese dependency parsing system based on multiple annotators, characterized by: The system includes: a module for executing the Vietnamese dependency parsing method based on multiple annotators according to any one of claims 1 to 7.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the program, the multi-annotator-based Vietnamese dependency parsing method according to any one of claims 1 to 7 is implemented.

Images