Implicit word reordering method for cross-language dependency analysis

Abstract

The invention relates to an implicit word reordering method for cross-language dependency analysis, which comprises the following steps of: adjusting a word order relationship in a feature space through a knowledge distillation implicit word reordering algorithm in a word reordering process aiming at an input natural language, and integrating the word order relationship into training of a cross-language dependency analyzer. And then processing of the low-resource natural language is completed. The double-affine dependence analytical model is composed of four layers, including an embedding layer, a bi-LSTM layer, an MLP layer and a scoring layer; wherein the implicit word reordering algorithm comprises word reordering teacher training and dependency analysis student learning, and the word reordering teacher training comprises head word search, training instance extraction and teacher model training; according to the implicit word reordering method for cross-language dependency analysis, the limitation of a previous method in terms of word order representation and high calculation cost is solved.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to the field of computers, and more specifically, to an implicit word reordering method for cross-lingual dependency parsing. BACKGROUND

[0002] Dependency parsing is a fundamental task aiming to extract low-level syntactic relations between words in a sentence, such as subject-verb relations. Recently, cross-lingual dependency parsing has attracted extensive attention from both academia and industry, for which a parser is trained on a source language and directly applied to a target language of interest. Multilingual pre-trained language models (mPLMs) have shown superior performance in cross-lingual dependency parsing. However, these mPLMs inevitably encode the word order feature to model the contextual representation, which is referred to as order-sensitive. Since the word order is essentially different across languages, there is a risk of overfitting to the source language word order, which can harm the performance of the target language.

[0003] Therefore, many studies have focused on the word order difference problem in cross-lingual dependency parsing. These methods can be mainly divided into two categories: first, as shown in (a), order-agnostic based methods utilize order-agnostic encoders, such as self-attention, frozen position embeddings, bag-of-words, to maintain robust performance to the change of word order. Second, as shown in (b), word reordering based methods usually first generate a set of new sentences with similar grammar to the target language by rearranging the words in the source sentence, and then select the generated high-quality sentences to train the dependency parser of the target language. Since these word reordering methods essentially reorder the words in the source language, the present disclosure refers to this kind of methods as explicit word reordering (EWR). Figure 1 Figure 1 (b).

[0004] However, previous methods have many limitations. Due to the lack of word order information, order-agnostic methods can weaken their representation ability, leading to the decline of dependency parsing. Since the permutation space grows exponentially with the sentence length, the computational cost of explicit word reordering (EWR) methods can be high. In addition, explicit word reordering can introduce language adversity, i.e., source language training data with unnatural word order, which can be regarded as a form of noise that hinders model learning. SUMMARY

[0005] The purpose of the embodiments of the present disclosure is to provide an implicit word reordering method for cross-lingual dependency parsing, which solves the limitations of previous methods in word order representation and high computational cost.

[0006] ​In one general aspect, there is provided an implicit word reordering method for cross-lingual dependency parsing, comprising: employing a graph-based bi-affine dependency parsing model in a cross-lingual dependency parsing process, adjusting word order relations in a feature space in a process of word reordering for an input natural language by an implicit word reordering algorithm through knowledge distillation, and integrating into training of a cross-lingual dependency parser, thereby completing processing for a low-resource natural language; the bi-affine dependency parsing model is composed of four layers, including an embedding layer, a bi-LSTM layer, an MLP layer, and a scoring layer; wherein the implicit word reordering algorithm comprises word reordering teacher training and dependency parsing student learning, the word reordering teacher training comprises center word searching, training instance extraction, and teacher model training, the teacher model training obtains POS embedding of a given part-of-speech (POS) label sequence {p1, p2,..., p L} by giving a POS label sequence Only general POS labels are used to eliminate morphological differences, and POS representations after introducing context information through an unordered Transformer layer

[0007]

[0008] The head representation of each and dependency representation are obtained through a two-dimensional dimension reduction MLP

[0009]

[0010] The word order of a given pair of POS labels with head-related edges is calculated where U and b are linear transformation and bias term respectively, and binary cross-entropy loss is used for word order learning,

[0011]

[0012] Further, the dependency parsing student learning gives a source input sentence s, which contains L words {w1, w2,..., w L} and their POS (part-of-speech) labels {p1, p2,..., p L}, obtains its representation sequence {r1, r2,..., r L} through an embedding layer and a BiLSTM layer, then uses the formula

[0013]

[0014]

[0015] Let each cycle be represented r i Specialization to center words And dependent words The edge connection, edge type and word order prediction are performed using a two-dimensional dimension reduction MLP respectively, and the word order prediction is calculated as follows:

[0016]

[0017] The student model learns the dependency syntax analysis by the following two losses:

[0018]

[0019] The word order knowledge distillation learning loss is formulated as a mean square error loss:

[0020]

[0021] The whole student training loss is the sum of the three losses:

[0022] L = L edge + λ1L label + λ2L order .

[0023] Further, the center word search is to predict the center word w i for each word w j in the sentence using a pre-trained parser in the source language.

[0024] Further, the training instance extraction includes creating a training instance for each pair of POS tags <p i , p j > of dependent center words <w i , w j >, and the word order label is made according to their positions in the original sentence, where represents that the part-of-speech tag p i is on the left side of the head p j , and represents that p i is on the right side of p j in the original sentence.

[0025] The technical effects to be achieved by the embodiments of the present application are:

[0026] The present application proposes an implicit word reordering method (IWR-KD) for cross-lingual dependency analysis, which learns to implicitly adjust word order in the representation space rather than truly arranging the words of a given sentence. Specifically, the present application uses target language part-of-speech (POS) data to train a word order model, which is easier to annotate than parse trees and generally reflects the syntax of a language. This trained target language word order prediction model is then used as a teacher model to decide the new order between two POS tags. Finally, a student model is trained using labeled source language training data to mimic the teacher's order prediction and dependency parsing.

[0027] The present application has the technical effect that the student model is ready, which not only can dynamically generate representations corresponding to the target language word order relationship, but also can retain the basic language structure of the original input, thereby avoiding the limitations of previous works and having considerable application value. BRIEF DESCRIPTION OF DRAWINGS

[0028] The above and other objects and features of the present disclosure will become more apparent from the following description of the disclosure when taken in conjunction with the accompanying drawings.

[0029] Figure 1 is a comparative schematic diagram illustrating a method for overcoming word order differences in cross-lingual dependency parsing in the prior art according to the present disclosure;

[0030] Figure 2 is an example of word reordering according to an embodiment of the present disclosure;

[0031] Figure 3 is a framework diagram of an implicit word reordering method according to an embodiment of the present disclosure;

[0032] Figure 4 is a word order distance and performance according to an embodiment of the present disclosure;

[0033] Figure 5 is a word order distance predicted by different order teacher models according to an embodiment of the present disclosure. DETAILED DESCRIPTION

[0034] The following detailed description is provided to help the reader obtain a complete understanding of the methods, devices, and / or systems described herein. However, various changes, modifications, and equivalents of the methods, devices, and / or systems described herein will be clear to those skilled in the art after understanding the disclosure of the present application. For example, the order of the operations described herein is only an example and is not limited to those set forth herein, but can be changed as will be clear to those skilled in the art after understanding the disclosure of the present application, except for operations that must occur in a specific order. In addition, the description of features known in the art can be omitted for more clarity and conciseness.

[0035] The features described herein can be implemented in different forms and should not be construed as limited to the examples described herein. Rather, these examples have been provided so that this disclosure will be thorough and will fully convey the scope of the methods, devices, and / or systems implemented herein to be understood after a study of the disclosure.

[0036] As used herein, the term "and / or" includes any one of the associated listed items, as well as any combination of any two or more of the associated listed items.

[0037] Although terms such as "first", "second", and "third" can be used herein to describe various components, assemblies, regions, layers, or sections, these components, assemblies, regions, layers, or sections should not be limited by these terms. Rather, these terms are only used to distinguish one component, assembly, region, layer, or section from another component, assembly, region, layer, or section. Thus, a first component, a first assembly, a first region, a first layer, or a first section referred to in the examples described herein can also be referred to as a second component, a second assembly, a second region, a second layer, or a second section without departing from the teachings of the examples.

[0038] In the specification, when an element (such as a layer, a region, or a substrate) is described as "on", "connected to", or "coupled to" another element, the element can be directly on, directly connected to, or directly coupled to the other element, or one or more other elements can be interposed therebetween. In contrast, when an element is described as "directly on", "directly connected to", or "directly coupled to" another element, no other element is interposed therebetween.

[0039] The terms used herein are only used to describe various examples and not to limit the disclosure. The singular form also intends to include the plural form unless the context clearly indicates otherwise. The terms "comprise", "include" and "have" indicate the presence of the stated feature, number, operation, component, element, and / or a combination thereof, but do not exclude the presence or addition of one or more other features, numbers, operations, components, elements, and / or a combination thereof.

[0040] Unless otherwise defined, all terms used herein (including technical terms and scientific terms) have the same meaning as commonly understood by one of ordinary skill in the art to which this disclosure belongs after an understanding of the present disclosure. Unless explicitly defined otherwise herein, terms such as those defined in a general dictionary should be interpreted as having a meaning consistent with their meaning in the context of the relevant art and this disclosure, and should not be interpreted ideally or overly formally.

[0041] Moreover, in the description of the examples, detailed descriptions of related structures or functions that are well-known in the art will be omitted when such detailed description would call for excessive deviation from the point of the present disclosure.

[0042] Figure 1 is a comparative schematic diagram showing a method for overcoming word order differences in cross-lingual dependency parsing in the prior art according to the present disclosure.

[0043] To address the limitations of explicit word reordering in the prior art, the present invention explores a new attempt, namely implicit word reordering, and proposes an implicit word reordering method (IWR-KD) for cross-lingual dependency analysis. Inspired by the surprisingly good performance of deep learning in feature linearization, IWR-KD learns to implicitly adjust the word order relationship in the word representation space, rather than truly arranging the words of a given sentence. Figure 1 (c) shows the difference between the proposed IWR-KD method and typical order-agnostic and EWR-based methods.

[0044] The application scenario of the present invention is that, with the acceleration of globalization, more and more enterprises need to provide services for multilingual users in different countries. For example, for an international e-commerce platform, when different language users submit complex problems, the system needs to analyze and process in real time and provide corresponding services. Therefore, how to understand and analyze different language texts has become a great challenge. Cross-lingual dependency parsing technology, as an underlying technology of natural language processing, can be directly used to improve the ability of multilingual text understanding and guarantee the quality of enterprise application services. For example, for "Where can I find the information for my order?" and "Where can I find the information for my order?", by analyzing the dependency relationship in them, including: <I, find, name subject>, <I, find, name subject>; <information, find, direct object>, <information, find, name subject>, etc., the system can accurately identify the needs of users in different languages and provide corresponding services.

[0045] Specifically, the present invention uses target language part-of-speech (POS) data to train a word order model, which is easier to annotate than parse trees and generally reflects the syntax of the language. Then, this trained target language word order prediction model is used as a teacher model to determine the new order between two POS tags. Finally, the present invention uses the labeled source language training data to train a student model to mimic the teacher's order prediction and dependency parsing. Once the student model is ready, it not only dynamically generates representations corresponding to the target language word order relationship, but also preserves the basic language structure of the original input, thus avoiding the limitations of previous works.

[0046] Dependency parsing is the task of creating a dependency tree for an input sentence. The tree is a directed graph that defines the grammatical relationship between the dependency word (such as Mary) and its head (such as prepared), such as Figure 2 (a). The goal of cross-lingual dependency parsing is to train a parser on a source language and perform well on an unseen target language. In this work, we systematically study the transferability of the proposed implicit word reordering between source and target languages ​​at different word order distances.

[0047] The present invention adopts a graph-based bi-affine dependency parsing model as the backbone of the parser of the present invention. The model consists of four layers, namely, the embedding layer, the bi-LSTM layer, the MLP layer and the scoring layer. Figure 3 As shown on the right. All parsers take words and their part-of-speech (POS) tags as input. Formally, given a set of L words {w1, w2, ..., w L} and its part-of-speech (POS) tag sequence {p1, p2, ..., p L}, the embedding layer creates an input embedding sequence e 1：L , where each e i is its word embedding and part-of-speech embeddings POS embeddings are trained from scratch, while word embeddings are initialized with a pre-trained multilingual language model such as mBERT. POS embeddings:

[0048]

[0049] To further introduce contextual information, we then encode each input embedding through a multi-layer bidirectional LSTM:

[0050] r i =BiLSTM(e 1：L , i)

[0051] Then, a 2D reduction MLP is used to represent each cycle r i Specialized into head word and dependent word representations for edge and label prediction.

[0052]

[0053] Next, we use a bi-affine classifier to compute edge attention scores and label attention scores, which can be decoded into a graph by retaining only edges that receive positive scores.

[0054]

[0055] where U and b are linear transformation and bias term respectively. For each position pair i, j, binary cross-entropy loss is used for the existence of edge i→j, and cross-entropy loss is used for the label of golden edge.

[0056]

[0057] The parsed training object is the sum of the above two losses.

[0058] L = L edge + λ1L label

[0059] The previous explicit word reordering gives a sentence s = {w1, w2,..., w L} in the given source language, the prior explicit word reordering (EWR) aims to arrange the words in it to be more similar to the order of the target language in syntax. Then the reordered sentence is represented as s' = {w'1, w'2,..., w' L}, as shown in Figure 2 (b) for training the cross-lingual dependency parser.

[0060] To alleviate the shortcomings of existing EWR methods, the present application proposes an implicit word reordering algorithm (IWR-KD) through knowledge distillation. Unlike traditional EWR, IWR-KD adjusts the word order relationship in the feature space and integrates this process into the training of the cross-lingual dependency parser. The overall structure of the proposed IWR-KD is shown in Figure 3 , which consists of two important components: a word reordering teacher that aims to determine the new word order according to the source input, and a dependency parsing student that learns dependency parsing not only from the labeled source input but also mimics the target word order prediction of the teacher.

[0061] The overview of the proposed IWR-KD includes two models: (i) a word reordering teacher that determines the new direction between the dependent word and the center word. (ii) a dependency parsing student that is supervised by both the teacher's predicted word order and dependency parsing labels.

[0062] IWR-KD framework

[0063] Word reordering teacher training

[0064] The goal of the word reordering teacher model is to determine the word order of each dependency according to the POS tags of the dependent word and the center word. Given the target language sentence {w1, w2,..., w L} and its part-of-speech tags {p1, p2,..., p L}, the following steps are applied to train the word reordering teacher:

[0065] Center word search. For each word w in the sentence i , the present invention uses a pre-trained parser in the source language to predict its central word w j .

[0066] Training example extraction. The present invention is a subordinate core word <w i , w j > Each pair of POS tags <p i , p j >Create a training instance. Word order label are made according to their position in the original sentence, where Represents the part-of-speech tag p i In its prefix p j on the left, and Indicates p i In the original sentence, p j on the right side.

[0067] Teacher model training. Figure 3 As shown on the left, the teacher network consists of four layers. Given a POS tag sequence {p1, p2, ..., p L}, the present invention first obtains its POS embedding The present invention only uses common part-of-speech tags to eliminate word form differences. Then, the present invention introduces POS representation after context information through the disordered Transformer layer. Compared with the BiLSTM layer, it can improve cross-lingual generalization ability.

[0068]

[0069] The present invention obtains each Head representation and dependency representation

[0070]

[0071] The present invention calculates the word order of a given pair of part-of-speech tags with head-related edges Where U and b are the linear transformation and bias terms respectively, and the binary cross entropy loss is used for word order learning.

[0072]

[0073]

[0074] When the teacher has been well trained in the target language, the present invention trains students in dependency parsing with the teacher as a guide.

[0075] Dependency Parsing Student Learning

[0076] The goal of the student model is to parse sentences based on the transformed features corresponding to the reordered source sentences. Figure 3 As shown on the right, the present invention implements the student model by adding a backbone-based word order learning network.

[0077] Here, the student model consists of four layers. Specifically, given a source input sentence s, which contains L words {w1, w2, ..., w L} and its POS (part of speech) tags {p1, p2, ..., p L The present invention first obtains its representation sequence {r1, r2, ..., r L}, as shown in the formula in Section 2.2. Then, the present invention uses the formula in Section 2.2

[0078]

[0079] Denote each cycle as r i Specialization as a central word and subordinate words Indicates that two-dimensional dimensionality reduction MLP is used to perform edge (edge ​​connection), label (edge ​​type) and word order prediction. In addition to calculating edge prediction and label prediction, the present invention also calculates word order prediction as shown below:

[0080]

[0081] The student model learns dependency parsing through the following two losses:

[0082]

[0083] The word order knowledge distillation learning loss is formulated as the mean squared error loss:

[0084]

[0085] The overall student training loss is the sum of three losses:

[0086] L=L edge +λ1L label +λ2L order

[0087] experiment

[0088] In this section, we conduct extensive experiments on 31 languages ​​from a wide range of language families to verify the effectiveness and rationality of the cross-language dependency parsing (IWR-KD) method proposed in this paper.

[0089] set up

[0090] Data sets The present invention conducted experiments on Universal Dependency (UD) treebank (v2.14), where 31 different languages were selected for evaluation. In the main experiments of the present invention, the present invention used English as the source language and the other 30 languages as the target languages. The present invention only used the source language for training and hyperparameter tuning. The evaluation metrics are unlabeled attachment score (UAS) and labeled attachment score (LAS). Each experiment was run three times with different random seeds and the average scores were reported. The present invention used mBERT to derive cross-lingual word embeddings. Since the mBERT embeddings are subword-level, the present invention followed previous work and used the first subword as the word-level embedding. The maximum subword sequence length was set to 512. The present invention trained the POS embeddings from scratch and set the dimension size of the POS embeddings to 50. The batch size was set to 32. The present invention trained the model using Adam with β1 of 0.9, β2 of 0.9, and L2 weight decay of 1e-5. The model was trained for 50 epochs with a learning rate of 1e-5 for mBERT and 3e-5 for other network layers. The present invention selected the best hyperparameters according to the development set in the source language. The present invention set λ1 = 1 and λ2 = 0.001 based on experience. The present invention implemented the method of the present invention using PyTorch 1.8.0 based on the Hugging Face transformer library.

[0091] To quantify the word order distance between two languages, the present invention selected the 52 most frequent dependency triplets in the 31 languages according to Ahmad The relative frequency of the left direction (dependent word) before its head was then concatenated as a word order feature for all triplets in each language. The present invention used Manhattan distance as the word order distance.

[0092] Baseline parsers The present invention compared IWR-KD with several competitive baselines that involve word order learning, as follows. SelfAtt Direct uses a self-attention based order-agnostic encoder for cross-lingual parsing. mBERT Direct is fine-tuned by adding a graph-based bi-affine dependency parser on top of it. Frozen PE freezes the position embeddings in mBERT during fine-tuning. Subtree-EWR explicitly reorders words by subtree constraints for cross-lingual parsing. WOL is a variant of the present invention that learns the word order knowledge in the source language through an auxiliary loss. For a fair comparison, all methods were implemented using the same data sets and training configurations.

[0093]

[0094] Table 1. Cross-lingual dependency parsing results (UAS% / LAS%) by language

[0095] Main results

[0096] Table 1 shows the results of the test set. These languages are sorted by their distance to English. From the experimental results, the following observations can be made. (1) The IWR-KD method proposed in this invention achieves state-of-the-art performance on most languages and average performance on all languages, demonstrating the effectiveness and universality of IWR-KD. (2) The order-sensitive direct transfer method (mBERT) significantly outperforms the traditional order-agnostic self-attention method (SelfAttn) by effectively capturing richer contextual information, including word order. (3) The performance of the mBERT-based order-agnostic model (Frozen PE) decreases relative to direct transfer (mBERT) in both source languages and similar languages (such as English (en) and Norwegian (no)) due to underfitting caused by frozen position representations. (4) The explicit word reordering method (SubtreeEWR) significantly drops in some languages, such as Slovenian (sl) and Finnish (fi), due to the introduction of noise that hinders model learning by unnatural sentence reordering. Note that the EWR method indirectly accesses target language dependency annotation information, so it can achieve high performance in some languages. Although IWR-KD only relies on low-cost target language POS tags, its average performance is better than SubtreeWR. (5) WOL performs better in languages close to English, while mBERT performs better in languages "far" from English. This indicates that capturing word order is crucial in cross-lingual dependency parsing.

[0097]

[0098] Table 2 Ablation study (average results on all languages).

[0099] Ablation study

[0100] To verify the effect of each part of the method, the following variants of IWR-KD are used in the experiments. (1) w / Pseudo labels, convert soft word order probability distribution to hard labels to guide student model training. Hard labels cause performance to drop, highlighting that the word order distillation loss of this invention conveys richer word order knowledge than hard labels. (2) w / Silver UPOS, use Stanza 4 tool to annotate POS tags to simulate a more realistic application scenario. In this case, the method of this invention can improve the performance of direct transfer method (mBERT in Table 2). (3) w / o FT, eliminate the fine-tuning of mPLM to make the model more lightweight. However, if there is no fine-tuning, the performance will drop significantly, indicating that fine-tuning can make the encoder effectively adapt to specific tasks.

[0101] Case study

[0102]

[0103] Table 3 Case study of implicit word reordering. Word order frequency represents the relative frequency of the left direction (depending on its head before). Green (red) highlights represent center words (dependent words).

[0104] In this section, the present invention conducts a series of case studies to gain insight into why the proposed IWR-KD works. The IWR-KD method can help the parser to correctly identify the dependency labels by exploiting the learned order relationship between the dependent words and the center words. Specifically, if the model can align the word order relationship between the source language and the target language, it can correct the misidentified dependency labels in the target language. As shown in Table 3, in the first example , the relative frequency of PRON before VERB is quite different in the source language and the target language, which leads to the failure of transfer. The present invention's model predicts the relative frequency of PRON before VERB in the source language close to that in the target language. At the same time, the present invention's model learns that their labels are obj in the source language training set, then the model can correctly predict the dependency labels of PRON and PRON. VERB in the target language are obj. The other two examples present the same results using different languages.

[0105] Performance vs. word order distance

[0106] As shown in Figure 4 , the transfer ability decreases as the word order distance increases from left to right, which indicates that the word order is a key factor for transfer performance. The Pearson correlation coefficient between the word order distance and the transfer performance is 0.8803 (p-value = 1e-10), showing a strong negative correlation. As shown in the lower subgraph of Figure 4 , the reduction of word order distance has a positive impact on performance improvement. The Pearson correlation coefficient is 0.4308 (p-value = 0.0175), indicating a strong positive correlation.

[0107] Effect of different teacher models

[0108] Figure 5 The green bar represents the original word order distance between English and the target language. This section studies the effect of different word reordering teacher models. The present invention compares three learning methods: (1) Random: randomly learns the word order relationship between pairs. (2) Heuristic: learns the ordering relationship of pairs belonging to 52 selected dependency triples. (3) Transfer (ours): based on the pre-trained parser in the source language, the present invention predicts the head of each word and learns their ordering relationship.

[0109] As shown in Figure 5As shown, the present application calculates the word order distance of three different teacher models for six different target languages. The three languages no, sv, da are closer to the source language, while the three languages zh, ko, ja are further from the source language. It can be seen that the present application's teacher model based on migration can effectively reduce the word order distance between the source language and the target language. The present method fails to reduce the word order distance between the source language and zh (Chinese), but it improves the transmission performance, as shown in Table 2. By more thorough observation, the present method reduces the word order difference in some aspects. Dependency triples. For example, for the word order frequency of EN: 0.0056, ZH: 0.9980, EN+IWR-KD: 0.5413.

[0110] Although some embodiments of the present disclosure have been shown and described, those skilled in the art should understand that modifications can be made to these embodiments without departing from the principles and spirits of the present disclosure, the scope of which is defined by the claims and their equivalents.

Claims

1. An implicit word reordering method for cross-language dependency parsing, characterized in that: include: In the process of cross-language dependency parsing, a graph-based bi-affine dependency parsing model is adopted. In the process of word reordering for the input natural language, the word order relationship in the feature space is adjusted by the implicit word reordering algorithm of knowledge distillation and integrated into the training of the cross-language dependency parser, thereby completing the processing of low-resource natural language. The bi-affine dependency parsing model consists of four layers, including an embedding layer, a bi-LSTM layer, an MLP layer and a scoring layer. The implicit word reordering algorithm includes word reordering teacher training and dependency parsing student learning. The word reordering teacher training includes center word search, training instance extraction and teacher model training. The teacher model training is performed by giving a part-of-speech (POS) label sequence {p1, p2, ..., p L }, get its POS embedding Only common part-of-speech tags are used to eliminate word form differences, and POS representations after introducing contextual information are introduced through the disordered Transformer layer. Each is obtained by two-dimensional dimensionality reduction MLP Head representation and dependency representation Compute word order for given pairs of part-of-speech tags with head-related edges Where U and b are linear transformation and bias terms respectively, and binary cross entropy loss is used for word order learning.

2. The implicit word reordering method for cross-language dependency parsing according to claim 1, wherein: The dependency parsing student learns that given a source input sentence s, which contains L words {w1, w2, ..., w L } and its POS (part of speech) tags {p1, p2, ..., p L }, and obtain its representation sequence {r1, r2, …, r through the embedding layer and BiLSTM layer L }, then, use the formula Denote each cycle as r i Specialization as a central word and subordinate words Indicates that two-dimensional dimensionality reduction MLP is used for edge connection, edge type and word order prediction, and word order prediction is calculated as follows: The student model learns dependency parsing through the following two losses: The word order knowledge distillation learning loss is formulated as the mean squared error loss: The overall student training loss is the sum of three losses: L=L edge +λ1L label +λ2L order 。 3. The implicit word reordering method for cross-language dependency parsing according to claim 2, wherein: The center word search is to find each word w in the sentence i , the present invention uses a pre-trained parser in the source language to predict its central word w j .

4. The implicit word reordering method for cross-language dependency parsing according to claim 3, wherein: The training instance extraction includes subordinate central words <w i , w j > Each pair of POS tags <p i , p j >Create a training instance, word order label are made according to their position in the original sentence, where Represents the part-of-speech tag p i In its prefix p j on the left, and Indicates p i In the original sentence, p j on the right side.

Images