Chinese-cross language dependency syntactic analysis method based on information screening network
By using information screening network and double affine syntax analysis model in the cross-language dependence syntax analysis of Chinese and Vietnamese cross-language dependence syntax analysis, the problem of language feature differences in cross-language syntax analysis is solved, and efficient analysis of Vietnamese syntax structure and cross-language syntax analysis performance is improved.
Patent Information
- Application Number
- CN202510212138.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-02-25
- Publication Date
- 2025-05-27
AI Technical Summary
In cross-language syntactic analysis, the difference in characteristics of different languages leads to poor syntactic analysis, especially in the processing of low-resource languages such as Vietnamese. The lack of high-quality labeled data is a major challenge.
The Hanyue cross-language dependence syntax analysis method based on information screening network is adopted. By fine-tuning the XLM-RoBERTa pre-trained model, combining the dual affine syntax analysis model and the information screening network, the matching relationship between specific representations of different languages is dynamically learned, and the representation ability of the target language feature space is enhanced.
It effectively alleviates the problem of differences in different language characteristics, improves the accuracy of Vietnamese syntax structure analysis, and significantly improves the performance of cross-language syntax analysis, especially under low resource data conditions.
Smart Images
Figure CN120046600A_ABST
Abstract
Description
[0001] Technical Language
[0002] The invention relates to a Chinese-Vietnamese cross-language dependency syntactic analysis method based on an information screening network, and belongs to the technical field of natural language processing. Background Art
[0003] Enhancing the ability to analyze syntactic dependencies in Southeast Asian languages such as Vietnamese is extremely important for promoting multiple tasks in natural language processing (NLP), including semantic role labeling, part-of-speech tagging, and more advanced artificial intelligence applications such as text summarization, public opinion analysis, and Chinese-Vietnamese neural machine translation.
[0004] With the rapid development of NLP technology, the ability of large language models to handle low-resource languages, especially Vietnamese, has been significantly improved. However, there are still great challenges in cross-language or multilingual tasks, especially in the accurate parsing of syntactic structures and dependency relationships with large language differences. Building an efficient syntactic analysis model often relies on a large amount of accurately labeled data, which poses a huge challenge for those resource-poor languages. Faced with this problem, this study proposed an innovative Chinese-Vietnamese cross-language dependency syntactic analysis method, using information screening networks to enhance the performance of the model. In this study, we developed a dual-affine syntactic analysis model based on information screening networks, aiming to improve the parsing accuracy of Vietnamese syntactic structures. Through a detailed comparative analysis of the syntactic structures of Chinese and Vietnamese, we found structural similarities between the two languages. For example, they both use the subject-verb-object syntactic model ("I love you" in Chinese corresponds to "I love you" in Vietnamese). ), adjectives modify nouns (the Chinese "blue sky" corresponds to the Vietnamese ), and adverbs modifying verbs (the Chinese word "carefully walk" corresponds to the Vietnamese word ). These commonalities provide us with valuable information for designing cross-language dependency parsing models. In addition, this has broad application value in many fields such as language education, translation services, information retrieval, etc., and is of great significance for promoting cross-cultural communication and improving the development of language technology. Summary of the invention
[0005] The technical problem to be solved by the present invention is: the present invention provides a Chinese-Vietnamese cross-language dependency syntactic analysis method based on an information screening network, which is used to alleviate the problem of differences in different language features in cross-language syntactic analysis, and at the same time has achieved good experimental results.
[0006] The technical solution of the present invention is: a Chinese-Vietnamese cross-language dependency syntactic analysis method based on an information screening network, and the specific steps of the method are as follows:
[0007] Step1: Fine-tune the XLM-RoBERTa pre-trained model using Chinese and Vietnamese UD datasets, and take the mean of the outputs of the last four layers of the model as the word embedding representations of Chinese and Vietnamese data, which serve as the main input to the bi-affine syntactic analysis model;
[0008] Step2: Input the word embedding representations obtained from the pre-trained model into an independent private bidirectional long short-term memory network (BiLSTM) for encoding to effectively capture the deep context information and semantic relationships of different languages;
[0009] Step3: Input the feature information of BiLSTM(src) of the source language and BiLSTM(tgt) of the target language into an information screening network. This network dynamically learns the matching relationships between different language-specific representations in the multi-layer network, enabling the model to pay more attention to source domain knowledge beneficial to the target language while ignoring useless or even harmful information, enhancing the representation ability of the target language feature space, and achieving knowledge transfer learning;
[0010] Step4: Use multi-layer perceptrons (MLPs) to reduce the dimension of the features to obtain the syntactic relationship and part-of-speech tag representations of each word;
[0011] Step5: Finally, decode using a bi-affine decoder while searching for the optimal dependency grammar tree;
[0012] Furthermore, the specific steps of Step1 are as follows:
[0013] Step1.1: Input Chinese and Vietnamese sentences w 0 w 1 ...w n into the cross-lingual pre-trained model and fine-tune to optimize the language representation ability of the pre-trained model for Chinese-Vietnamese;
[0014] Step1.2: Take the average of the outputs of the last four layers of the pre-trained model as the final embedding representation x of each word in the sentence i 。
[0015] Furthermore, the specific steps of Step2 are as follows:
[0016] Step2.1: Input the Chinese word embedding representations into BiLSTM(src) to capture richer Chinese-specific context semantic information;
[0017] Step2.2: Input the Vietnamese word embedding representations into BiLSTM(tgt) to capture richer Vietnamese-specific context semantic information.
[0018] Furthermore, the specific steps of Step3 are as follows:
[0019] Step 3.1: Input the network weight information of the BiLSTM (src) in the source language and the BiLSTM (tgt) in the target language into the information screening network for feature screening;
[0020] Step 3.2: Set learnable layer matching weights W for the matching pairs (n, m) between the three layers of BiLSTM (src) to BiLSTM (tgt). n,m This weight determines the scale when migrating the output of the nth layer of the Chinese BiLSTM (src) to the output of the mth layer of the Vietnamese BiLSTM (tgt);
[0021] Step 3.3: After the model learns the optimal layer matching weight W, use it to optimize the learning of the element matching weight n,m This element matching weight can make the model pay more attention to the intermediate elements beneficial to Vietnamese according to the influence of Chinese features on Vietnamese features;
[0022] Step 3.4: Repeatedly use the information screening network to enhance the context representations h 0 h 1 ...h n i to achieve cross - language information screening and knowledge transfer.
[0023] Furthermore, the specific steps of Step 4 are as follows:
[0024] Step 4.1: Input h i into the MLPs. The MLPs scale the dimensions of these representations to enhance their adaptability to more complex classification tasks, and then obtain the central word representations i containing the corresponding syntactic relationship and part - of - speech tag information for each word w and the modifier representations Furthermore, Step 5 specifically includes the following:
[0025] Step 5.1: Input these representations obtained in Step 4 into the bi - affine decoder BiAffines for decoding, and simultaneously calculate the scores of the dependency relationships without labels and the scores of the dependency relationships with labels for each word.
[0026] The present invention also provides a Chinese - Vietnamese cross - language dependency parsing method system based on an information screening network. The system includes: a module for executing the Chinese - Vietnamese cross - language dependency parsing method based on an information screening network.
[0027] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored on the memory and executable on the processor. When the processor executes the program, the method for Chinese-Vietnamese cross-lingual dependency parsing based on an information screening network is implemented.
[0028] The present invention also provides a non-transitory computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the method for Chinese-Vietnamese cross-lingual dependency parsing based on an information screening network is implemented.
[0029] The present invention also provides a computer program product, including a computer program. When the computer program is executed by a processor, the method for Chinese-Vietnamese cross-lingual dependency parsing based on an information screening network is implemented.
[0030] The beneficial effects of the present invention are as follows:
[0031] 1. The present invention performs independent context encoding on different languages to capture language-specific semantic and structural information, so as to avoid affecting the effect of the syntactic analysis task due to differences between languages.
[0032] 2. The present invention innovatively designs an information screening network to enhance the encoding layer of the model. By dynamically learning the matching weights between different language-specific representations, the model pays more attention to the source domain knowledge beneficial to the target language and ignores useless or even harmful information, enhancing the representation ability of the target language feature space and providing a solid foundation for cross-lingual syntactic analysis.
[0033] 3. The method of the present invention can efficiently screen useful knowledge in resource-rich languages to enhance the model's ability to capture syntactic information in low-resource languages. This method greatly alleviates the problem of the lack of high-quality low-resource data and achieves good results in experiments. Description of the Drawings
[0034] Figure 1 is a flow chart in the present invention; Detailed Embodiments
[0035] Embodiment 1: As Figure 1 shown, a method for Chinese-Vietnamese cross-lingual dependency parsing based on an information screening network uses a BiAffine Parser for experiments, which is mainly divided into five parts: a cross-lingual word embedding layer, a private BiLSTM encoding layer, an information screening network layer, an MLPs layer, and a BiAffines decoding layer. The specific steps of the method are as follows:
[0036] Step1: Fine-tune the XLM-RoBERTa pre-trained model using Chinese and Vietnamese UD datasets. Take the mean of the outputs of the last four layers of the model as the word embedding representations of Chinese and Vietnamese data, which serve as the main input for the bi-affine syntactic analysis model.
[0037] Furthermore, the specific steps of Step1 are as follows:
[0038] Step1.1: Input Chinese and Vietnamese sentences w 0 w 1 ...w n into the cross-lingual pre-trained model and fine-tune it to optimize the language representation ability of the pre-trained model for Chinese-Vietnamese.
[0039] Step1.2: Take the average of the outputs of the last four layers of the pre-trained model as the final embedding representation x i .
[0040] Step2: Input the word embedding representations obtained from the pre-trained model into an independent private bidirectional long short-term memory network (BiLSTM) for encoding to effectively capture the deep context information and semantic relationships of different languages.
[0041] Furthermore, the specific steps of Step2 are as follows:
[0042] Step2.1: Input the word embedding representations of Chinese into BiLSTM(src) to capture richer Chinese-specific context semantic information;
[0043] Step2.2: Input the word embedding representations of Vietnamese into BiLSTM(tgt) to capture richer Vietnamese-specific context semantic information. The above formula is as follows:
[0044] h i = BiLSTM(x i )(1)
[0045] Step3: Input the network weight information of BiLSTM(src) of the source language and BiLSTM(tgt) of the target language into the information screening network. This network dynamically learns the matching relationships between different language-specific representations in the multi-layer network, enabling the model to pay more attention to the source domain knowledge beneficial to the target language while ignoring useless or even harmful information, enhancing the representation ability of the target language feature space, and achieving knowledge transfer learning.
[0046] Furthermore, the specific steps of Step3 are as follows:
[0047] Step 3.1: Input the feature information of the BiLSTM (src) in the source language and the BiLSTM (tgt) in the target language into the information screening network for feature screening, enabling it to dynamically learn the matching weights between different domain-specific representations. These weights can make the model pay more attention to the source domain knowledge useful for the target domain and ignore the useless or even harmful information. Considering that using sufficient labeled data in the source domain can well complete the training of source domain feature representations, emulating these trained source domain features helps enhance the representation ability of the target domain feature space. In this paper, by minimizing l 2 Achieve knowledge transfer learning from the source domain to the target domain.
[0048]
[0049] Among them, f θ Represents a linear transformation, a commonly used activation function. Represents the output of the m-th layer of BiLSTM (src), Represents the output of the n-th layer of BiLSTM (tgt).
[0050] Step 3.2: Set learnable layer matching weights W n,m For each matching pair (n, m) between the three layers of BiLSTM (src) and BiLSTM (tgt). This weight determines the scale when migrating the output of the n-th layer of Chinese BiLSTM (src) to the output of the m-th layer of Vietnamese BiLSTM (tgt).
[0051]
[0052] Among them Is a non-linear transformation function, and the activation function used is RELU (*), mainly to ensure that the finally generated layer matching weight is a non-negative number;
[0053] Step 3.3: After the model learns the optimal layer matching weight W n,m , use it to optimize the learning element matching weight This element matching weight can make the model pay more attention to the intermediate elements beneficial to Vietnamese according to the influence of Chinese features on Vietnamese features. The definition of the matching loss function for the matching pair (n, m) is as follows:
[0054]
[0055] Among them, d ∈ {1, 2,..., D}, D is the output dimension of BiLSTM, Is the non-negative weight of the d-th element on the matching pair, where Moreover, since each important element to be transmitted in the input is different, the element transmission weights are set as follows in this paper:
[0056]
[0057] where n, m ∈ {1, 2, 3} are the number of layers output by the bidirectional long short-term memory network, and k = 3 * 3 = 9 is the number of matching pairs;
[0058] Step3.4: Repeatedly use the information screening network to enhance and optimize the context representation h obtained by the BiLSTM encoding layer 0 h 1 ...h n to achieve cross-language information screening and knowledge transfer.
[0059] Step4: Use a multi-layer perceptron (MLPs) to reduce the dimension of the features to obtain the syntactic relationship and part-of-speech tag representation of each word.
[0060] Furthermore, Step4 specifically includes the following:
[0061] Step4.1: Input h i into the MLPs. The MLPs scale the dimensions of these representations to enhance their adaptability to more complex classification tasks, and then obtain the central word representation i containing the corresponding syntactic relationship and part-of-speech tag information for each word w and the modifier representation
[0062] Step5: Finally, use a bi-affine decoder to decode and simultaneously find the optimal dependency grammar tree.
[0063] Furthermore, Step5 specifically includes the following:
[0064] Step5.1: Input these representations obtained in Step4 into the bi-affine decoder BiAffines for decoding, and simultaneously calculate the scores of the dependency relationships without labels for each word, as well as the scores of the dependency relationships with labels. Specifically, the calculation formula for the score of the dependency relationship without labels is as follows:
[0065]
[0066] where the matrix U 1 represents the bi-affine parameter, and score(i←j) represents the score of the dependency arc without labels pointing from the core word w j to the modifier w i ; the calculation method for the score of the dependency relationship with labels is as follows:
[0067]
[0068] Among them, U 2 and U 3 are weight matrices, and b is the bias.
[0069] The syntactic analysis model defines a local cross-entropy loss function for each position i to optimize the model parameters. The loss function is defined as follows:
[0070]
[0071] Among them, w j represents the correct head word of the word w i , and l represents the type of dependency relationship between them.
[0072] Step 5.2. Finally, the total loss function is as shown in the following formula:
[0073]
[0074] The present invention also provides a Chinese-Vietnamese cross-lingual dependency syntactic analysis method system based on an information screening network. The system includes:
[0075] A word embedding representation module (cross-lingual word embedding layer) for fine-tuning the XLM-RoBERTa pre-trained model using Chinese and Vietnamese UD datasets, taking the mean of the outputs of the last four layers of the model as the word embedding representations of Chinese and Vietnamese data, and using this as the main input of the bi-affine syntactic analysis model;
[0076] An encoding module (private BiLSTM encoding layer) for inputting the word embedding representations obtained from the pre-trained model into an independent private bidirectional long short-term memory network BiLSTM for encoding to effectively capture the deep context information and semantic relationships of different languages;
[0077] A transfer module (information screening network layer) for enabling the model to pay more attention to the source domain knowledge beneficial to the target language while ignoring useless or even harmful information by dynamically learning the matching relationship between different language-specific representations in a multi-layer network for the feature information of the BiLSTM of the source language and the BiLSTM of the target language input, enhancing the representation ability of the target language feature space, and realizing knowledge transfer learning;
[0078] An acquisition module (MLPs layer) for using a multi-layer perceptron MLPs to reduce the dimension of the features to obtain the syntactic relationship and part-of-speech tag representation of each word;
[0079] A decoding module (BiAffines decoding layer) for decoding using a bi-affine decoder and simultaneously finding the optimal dependency grammar tree.
[0080] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored on the memory and executable on the processor. When the processor executes the program, the method for Chinese-Vietnamese cross-lingual dependency parsing based on an information screening network is implemented.
[0081] The present invention also provides a non-transitory computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the method for Chinese-Vietnamese cross-lingual dependency parsing based on an information screening network is implemented.
[0082] The present invention also provides a computer program product, including a computer program. When the computer program is executed by a processor, the method for Chinese-Vietnamese cross-lingual dependency parsing based on an information screening network is implemented.
[0083] In the present invention, Chinese and Vietnamese datasets are selected from the Universal Dependencies (UD) for experiments, and the untagged dependency score UAS and the tagged dependency score LAS are used as evaluation indicators for the performance of the bi-affine syntactic analysis model. The calculation formulas are as follows:
[0084]
[0085] To prove the effectiveness of the present invention, two mainstream Vietnamese dependency parsing methods are selected as benchmark models. 1) Pre-training syntactic analysis model: Only the Vietnamese UD dataset is used and input into the pre-training model to enhance the word representation ability of the pre-training model to improve the performance of the syntactic analysis model. 2) Fine-tuning syntactic analysis model: Two datasets of Chinese and Vietnamese are used. The pre-training model is trained with Chinese and then fine-tuned on the Vietnamese dataset to directly transfer Chinese knowledge to Vietnamese. In the present invention, first, the XLM-RoBERTa pre-training model is cross-trained with Chinese and Vietnamese, and then a cross-lingual information screening network layer is designed to enhance the encoding effect of the BiLSTM, select the beneficial information from Chinese for Vietnamese, and ignore the useless or even harmful information, so as to enhance the spatial representation ability of the encoder for the target language features.
[0086] The experimental results of the present invention are shown in Table 1 below. First of all, it is worth pointing out that although fine-tuning the pre-trained model by adding Chinese and Vietnamese data together enhances the performance of the syntactic analysis basic model, the method proposed in the present invention still shows the best results, further verifying the effectiveness of the method. Secondly, applying the information screening network strategy on the encoder is also significantly helpful for syntactic analysis. Finally, the comprehensive application of this series of strategies further improves the performance of the model in syntactic analysis, affirms the value of the combination of cross-language information screening network and BiLSTM for the syntactic analysis task, and at the same time emphasizes the necessity of the Chinese-Vietnamese cross-language information screening network. The two complement each other and are mutually reinforcing.
[0087] Table 1 Comparison of test set scores of different models
[0088]
[0089] Table 2 Comparison of test set scores of different hyperparameter settings
[0090]
[0091] Table 2 is based on the model we proposed. Diffloss beta is the important weight determining the importance of the information screening network screening task. It is obvious from the table that the change of the hyperparameter diffloss beta has a great impact on the performance of the model. Especially when the parameter is set to 0.5, the performance is the best, further proving the necessity of information screening.
[0092] Table 3 Comparison of test set scores of different hyperparameter settings
[0093]
[0094] As shown in Table 3, meta loss alpha defines the weight coefficient of each matching loss term. It can be seen that the performance is better when the coefficient is 1, which also illustrates the importance of matching and screening network information.
[0095] The above tables and drawings fully prove the effectiveness and rationality of the present invention. However, the present invention is not limited to the above embodiments. Within the scope of knowledge possessed by those of ordinary skill in the art, various changes can be made without departing from the gist of the present invention.
Claims
1. A Chinese-Vietnamese cross-language dependency syntactic analysis method based on information screening network, characterized by: The specific steps of the method are as follows: Step 1: Fine-tune the XLM-RoBERTa pre-trained model using the Chinese and Vietnamese UD datasets, take the mean of the output of the last four layers of the model as the word embedding representation of the Chinese and Vietnamese data, and use this as the main input of the bi-affine syntactic analysis model; Step 2: Input the word embedding representation obtained by the pre-trained model into an independent private bidirectional long short-term memory network BiLSTM for encoding, effectively capturing the deep contextual information and semantic relationships of different languages; Step 3: Input the feature information of the source language BiLSTM and the target language BiLSTM into the information screening network. The network dynamically learns the matching relationship between different language-specific representations in the multi-layer network, so that the model pays more attention to the source domain knowledge that is beneficial to the target language while ignoring useless or even harmful information, thereby enhancing the representation ability of the target language feature space and realizing knowledge transfer learning. Step 4: Use multi-layer perceptrons (MLPs) to reduce the dimension of features to obtain the syntactic relationship and part-of-speech tag representation of each word; Step 5: Finally, use the dual affine decoder to decode and find the optimal dependency syntax tree.
2. The Chinese-Vietnamese cross-language dependency syntactic analysis method based on information screening network according to claim 1 is characterized by: The specific steps of Step 1 are as follows: Step 1.1: Translate Chinese and Vietnamese sentences into w0w1...w n The data is input into a cross-language pre-trained model and fine-tuned to optimize the pre-trained model's language representation capabilities for Chinese and Vietnamese. Step 1.2: Take the average of the outputs of the last four layers of the pre-trained model as the final embedding representation x for each word in the sentence i .
3. The Chinese-Vietnamese cross-language dependency syntactic analysis method based on information screening network according to claim 1 is characterized by: The specific steps of Step 2 are as follows: Step 2.1: Input the Chinese word embedding representation into BiLSTM to capture richer Chinese-specific contextual semantic information; Step 2.2: Input the Vietnamese word embedding representation into BiLSTM to capture richer Vietnamese-specific contextual semantic information.
4. The Chinese-Vietnamese cross-language dependency syntactic analysis method based on information screening network according to claim 1 is characterized by: The specific steps of Step 3 are as follows: Step 3.1: Input the network weight information of the BiLSTM of the source language and the BiLSTM of the target language into the information screening network for feature screening; Step 3.2: Set the learnable layer matching weights W for the matching pairs (n, m) between the three layers of the source language BiLSTM and the target language BiLSTM n,m , which determines the scale of the migration from the output of the nth layer of the BiLSTM in the Chinese source language to the output of the mth layer of the BiLSTM in the Vietnamese target language; Step 3.3: The model learns the optimal layer matching weight W n,m Afterwards, use it to optimize the learning element matching weights The element matching weight makes the model pay more attention to the intermediate elements that are beneficial to Vietnamese according to the influence of Chinese features on Vietnamese features; Step 3.4: Repeatedly use the information screening network to represent the context h0h1...h obtained by the BiLSTM encoding layer n Enhancements are made to achieve cross-language information screening and knowledge transfer.
5. The Chinese-Vietnamese cross-language dependency syntactic analysis method based on information screening network according to claim 4 is characterized by: The Step 4 specifically includes the following: Step 4.1: h i are input into MLPs, which scale these representations to enhance their adaptability to more complex classification tasks, and then obtain each word w from them i The center word representation contains the corresponding syntactic relationship and part-of-speech tag information and modifiers 6. The Chinese-Vietnamese cross-language dependency syntactic analysis method based on information screening network according to claim 5 is characterized by: The Step 5 specifically includes the following: Step 5.1: Input the representations obtained in Step 4 into the bi-affine decoder BiAffines for decoding, and calculate the score of each word without label dependency and the score of the label dependency.
7. A Chinese-Vietnamese cross-language dependency syntactic analysis method system based on information screening network, characterized in that: The system comprises: a module for executing the Chinese-Vietnamese cross-language dependency syntax analysis method based on information screening network as claimed in any one of claims 1 to 6.
8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, the Chinese-Vietnamese cross-language dependency syntax analysis method based on an information screening network as described in any one of claims 1 to 6 is implemented.
9. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method for Chinese-Vietnamese cross-language dependency syntactic analysis based on an information screening network as described in any one of claims 1 to 6 is implemented.
10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the method for Chinese-Vietnamese cross-language dependency syntactic analysis based on an information screening network as described in any one of claims 1 to 6 is implemented.