A method for endangered language translation model fusing syntax information

Abstract

The application discloses a kind of endangered language translation model methods of fusing syntax information, comprising: endangered language dependency structure tree library of standard format is constructed using semi-automatic mode of dependency syntax;Endangered language is analyzed based on dependency syntax using double affine classifier, and endangered language dependency syntax analysis model based on double affine classifier is constructed;Word order index, part-of-speech tagging, governing word index, dependency syntax relationship marking contained in endangered language dependency structure tree library are added to the coding end of machine translation model as syntax features, and endangered language-Chinese neural machine translation model is constructed.The endangered language can be more accurately translated by the syntax information, and the effectiveness of endangered language translation is greatly improved by overcoming the deficiencies such as time-consuming and laborious, the need for a large amount of professional knowledge, less data, poor effect using conventional neural machine translation method, etc.

Description

Technical Field

[0001] This invention relates to the field of machine translation technology in artificial intelligence, and in particular to a method for translating endangered languages ​​by incorporating syntactic information. Background Technology

[0002] Because most endangered languages ​​have scarce linguistic resources and lack written records, the current practice is to first transcribe the collected audio data using the International Phonetic Alphabet (IPA), and then have native speakers and linguists annotate it in a common language (such as Chinese) before it can be understood by the public. This method of phonetic transcription requires a lot of manpower and time and lacks the scalability of linguistic resources.

[0003] Existing methods for translating endangered languages ​​using neural machine learning face several technical challenges, including:

[0004] First, endangered language corpus resources are scarce, there are few native speakers, and corpus annotation is difficult and time-consuming. There is still no standard dataset.

[0005] Secondly, endangered languages ​​do not follow the same grammatical rules as Chinese, and because they have no written script, they can only be represented by the International Phonetic Alphabet, which makes them difficult to understand.

[0006] Third, manually annotating endangered language corpora is time-consuming and labor-intensive, requires a lot of professional knowledge, and results in a relatively small amount of data. Summary of the Invention

[0007] To overcome the shortcomings of the existing technology, this invention provides a method for translating endangered languages ​​by integrating syntactic information. By constructing a translation model for endangered languages ​​that integrates syntactic information, the translation of endangered languages ​​can be completed more accurately.

[0008] This invention constructs a translation model for endangered languages ​​that integrates syntactic information, comprising three aspects: semi-automatic construction of an endangered language dependency structure tree library; implementation of endangered language dependency syntactic analysis based on a dual affine classifier; and establishment of a neural machine translation model for endangered languages ​​and Chinese that integrates syntactic information. This invention enables more accurate translation of endangered languages ​​through syntactic information and overcomes the problems of time-consuming and laborious manual annotation of endangered language corpora, the need for extensive professional knowledge, limited data volume, and poor performance of conventional neural machine translation methods.

[0009] This invention includes:

[0010] A semi-automatic approach is adopted to construct a dependency structure tree library for endangered languages ​​in the CoNLL-U standard format, thereby promoting the development of language resources for endangered languages. A graph-based method and a dual affine classifier model are used to integrate multiple language embedding methods and encoding models to better understand the language structure of endangered languages. Syntactic information of endangered languages ​​is combined with the Transformer machine translation model to assist in the annotation of endangered languages.

[0011] Among them, the endangered language dependency structure tree library is used for dependency syntactic analysis of endangered languages ​​and the establishment of a neural machine translation model of endangered languages-Chinese.

[0012] The Endangered Language Dependency Syntax Analysis module uses the Endangered Language Dependency Structure Tree Library as experimental data to perform dependency syntax analysis on endangered languages, supporting the subsequent establishment of an endangered language-Chinese neural machine translation model, including an embedding layer, an encoding layer, a feature dimensionality reduction layer, a dual affine parsing layer, and a decoding layer.

[0013] A neural machine translation model for endangered languages ​​and Chinese was established. The word order index, part-of-speech tagging, dominant word index, and dependency syntactic relation tagging contained in the endangered language dependency structure tree library were added as syntactic features to the encoding end of the Transformer model to construct the neural machine translation model for endangered languages ​​and Chinese.

[0014] The construction of the model in this invention consists of the following three steps: semi-automatic construction of an endangered language dependency structure tree database; construction of an endangered language dependency syntactic analysis model based on a dual affine classifier; and construction of an endangered language-Chinese neural machine translation model that integrates syntactic information, specifically including:

[0015] 1. Semi-automatic construction of the endangered language dependency structure tree database: First, collect the corpus, then preprocess the corpus, annotate the processed corpus with part-of-speech and dependency syntax relations, and finally construct the endangered language dependency structure tree database and perform manual verification.

[0016] 2. Constructing a dependency parsing model for endangered languages ​​based on a dual affine classifier: A graph-based dependency parsing model for endangered languages, TuParser, is proposed. It performs dependency parsing for endangered languages ​​through an embedding layer, an encoding layer, a feature dimensionality reduction layer, a dual affine parsing layer, and a decoding layer.

[0017] 3. Constructing a Neural Machine Translation Model for Endangered Languages ​​(Languages) - Chinese: The Transformer model was selected as the baseline model, and a neural machine translation model, TuSynTRM, incorporating syntactic information, was designed based on it. While maintaining the Transformer model structure, the focus was on enhancing the source input features by combining word order indexes, part-of-speech tagging, dominant word indexes, and dependency relation annotations from the endangered language dependency structure tree database with the encoding end of the traditional Transformer model.

[0018] Specifically, it includes the following steps:

[0019] A semi-automatic construction of an endangered language dependency structure tree library, resulting in a CoNLL-U format endangered language dependency structure tree library;

[0020] B uses a dual affine classifier to perform dependency parsing on endangered languages, obtaining the dependency structure of sentences in endangered languages.

[0021] Specifically, the graph-based dependency parsing model for endangered languages, TuParser, is used.

[0022] The TuParser model consists of five parts: an embedding layer, an encoding layer, a feature dimensionality reduction layer, a biaffine parsing layer, and a decoding layer.

[0023] C establishes a neural machine translation model for endangered languages, specifically Chinese, that integrates syntactic information. The Transformer is selected as the baseline model, and a neural machine translation model, TuSynTRM, integrating syntactic information is designed based on it.

[0024] C1 uses the TuParser dependency parsing model from B to determine whether there are dependency relationships between words in sentences of endangered languages, and the corresponding types of dependency relationships;

[0025] C2 integrates syntactic information features (including parts of speech and dependency relations of endangered languages) embedding and positional encoding (word order index and dominant word index of endangered languages);

[0026] C2.1 incorporates the two types of syntactic information extracted from A—part-of-speech tagging and dependency relation tagging of endangered languages—as part of the input feature embedding.

[0027] C2.2 treats the word order index and dominant word index of endangered languages ​​as positional information and encodes the word order index and dominant word index of endangered languages ​​at their positions, so that the TuSynTRM model can learn additional syntactic information of endangered languages.

[0028] The C3 TuSynTRM model uses traditional attention mechanisms to extract information from endangered languages ​​and Chinese, and models the relationship between the two languages ​​to construct an endangered language translation model that integrates syntactic information.

[0029] C31. Map the input sequence of endangered language word vectors into three matrices Q, K, and V, and obtain the relevance matrix after matrix multiplication, scaling, and masking operations;

[0030] C32. Perform matrix normalization, compare the similarity of Q and K to obtain the weight coefficients, and sum them with V to obtain the self-attention value;

[0031] C33. Perform self-attention operation on each self-attention head to obtain a single-head output;

[0032] C34. Finally, the outputs of the T self-attention heads are concatenated and linearly transformed to obtain the output of the multi-head attention.

[0033] D. Using the constructed endangered language translation model, realize the translation of endangered languages ​​that incorporates syntactic information.

[0034] Through the above steps, a standard database was established, and a translation model for endangered languages ​​incorporating syntactic information was constructed. This model, through syntactic information, can more accurately translate endangered languages ​​and overcomes the shortcomings of manually annotating endangered language corpora, such as being time-consuming and laborious, requiring a large amount of professional knowledge, having limited data, and exhibiting poor results using conventional neural machine translation methods, thus greatly improving the effectiveness of endangered language translation. Attached Figure Description

[0035] Figure 1 The diagram shows the structure of the TuParser model, which is based on a dual affine classifier and is constructed according to this invention.

[0036] Figure 2 This is a structural block diagram of the TuSynTRM model constructed for this invention. Detailed Implementation

[0037] The invention will now be further described with reference to the accompanying drawings and examples.

[0038] The method of this invention includes: 1) semi-automatic construction of an endangered language dependency structure tree library; 2) endangered language dependency syntactic analysis based on a dual affine classifier; and 3) establishment of an endangered language-Chinese neural machine translation model that integrates syntactic information.

[0039] Specifically, it includes the following steps:

[0040] A semi-automatic construction of an endangered language dependency structure tree library yields a CoNLL-U format endangered language dependency structure tree library, in which each endangered language sentence consists of the International Phonetic Alphabet of one or more words, and each endangered language word is represented by multiple fields of information, including: word order index, part-of-speech tagging, dominant word index, and dependency relationship tagging information;

[0041] The A1 corpus is exported by first selecting corpora containing different categories such as folk tales, historical stories, and conversations. These sentences are then exported using ELAN speech annotation software in the form of three lines of text: "International Phonetic Alphabet, Chinese translation (translating individual words), and Chinese translation (translating the entire sentence)".

[0042] A2 preprocesses the text exported from A1, including:

[0043] A2.1 Remove duplicate data and replace all punctuation marks with spaces to obtain parallel sentence groups for endangered language-Chinese translation.

[0044] A2.2 automatically segments the parallel sentence groups of endangered language-Chinese translations obtained in A2.1 into individual words using spaces as delimiters. Parallel sentence pairs that are perfectly aligned (one-to-one correspondence between Chinese annotations and IPA symbols for endangered languages) and sentences with errors (no one-to-one correspondence between Chinese annotations and IPA symbols for endangered languages) are output separately. The sentences with errors are re-segmented and manually aligned. The manually adjusted sentences are then merged with the sentences obtained from the automatic segmentation.

[0045] A2.3 When translating function words into Chinese, use English abbreviations to mark them, and then replace them with the corresponding Chinese words according to the context.

[0046] A3 performs part-of-speech tagging on the words obtained from the segmentation in A2, resulting in part-of-speech tagging for endangered languages.

[0047] A4 performs dependency parsing on the preprocessed sentences in A2 to obtain dependency parsing;

[0048] The A5 CoNLL-U format endangered language dependency structure tree database is constructed. Each endangered language sentence obtained from A2 consists of the IPA symbols of one or more words. Each endangered language word is represented by 10 fields of information, namely:

[0049] (1) ID: International Phonetic Index of Endangered Language Words, each new sentence is marked starting from the integer 1.

[0050] (2) FORM: International Phonetic Alphabet for words in endangered languages.

[0051] (3) LEMMA: a root morpheme of an endangered language, which is replaced by - here.

[0052] (4) UPOSTAG: Part-of-speech tag for endangered languages, see A3.

[0053] (5)XPOSTAG: a specific part-of-speech tag, which is replaced by - here.

[0054] (6)FEATS: Lexical or grammatical features of endangered languages, here replaced by -.

[0055] (7) HEAD: The dominant word number of the currently endangered language word, which is the value of ID or 0 (i.e., the root node).

[0056] (8) DEPREL: Dependency relationships for endangered languages ​​as defined, see A4.

[0057] (9) DEPS: Second-order dependency relationship, which is replaced by - here.

[0058] (10)MISC: Chinese annotations corresponding to the International Phonetic Alphabet of words in endangered languages.

[0059] A6 manual verification includes verification of Chinese annotations, part-of-speech tagging, and dependency relationship tagging.

[0060] B uses a dual affine classifier to perform dependency parsing on endangered languages, obtaining the dependency structure of sentences in endangered languages.

[0061] Zhang et al. designed a Chinese dependency parsing framework and proposed a graph-based dependency parsing model for endangered languages, TuParser. The TuParser model consists of five parts: an embedding layer, an encoding layer, a feature dimensionality reduction layer, a biaffine parsing layer, and a decoding layer.

[0062] The B1 embedding layer is designed to use input vectors for endangered languages, all of which use e i This represents the constructed input vector. For an endangered language sentence containing n words, W = {w1,...,w...} i ,...,w n}, where w i Let represent the i-th endangered language word. This method enriches the input vector e with part-of-speech features. i Information representation

[0063]

[0064] in Represents the word embedding vectors of endangered languages. Represents part-of-speech feature vectors, This indicates a vector concatenation operation.

[0065] The B2 encoding layer uses three consecutive BiLSTMs to perform context encoding on the input vector output by B1, for the i-th endangered language word w i The output vector e after the embedding layer i The data is fed into a BiLSTM, and the output of the last layer of the model is used as the context feature vector r. i .

[0066] r i =BiLSTM(e i ,θ bilstm )

[0067] Where θ bilstm These are the parameters for the BiLSTM model.

[0068] In the feature reduction layer, B3 uses an MLP network to process the output vector r of B2. i Nonlinear transformations are performed to remove redundant information irrelevant to the current decision, thereby improving the overall training speed and analysis accuracy of the TuParser model. The dimensionality-reduced syntactic feature vectors of the four endangered languages ​​are represented by the following formulas:

[0069]

[0070]

[0071]

[0072]

[0073] Among them, MLP (*) Refers to independent multilayer perceptron networks. and These represent the characteristics of dependency arc child nodes, dependency arc parent nodes, dependency relation types (including: subject-verb, object-verb, prepositional phrase, adverbial-head, verb-complement, attributive-head, locative, coordinate, double object structure, clause structure, pivotal structure, serial verb structure, function word structure, and core relation) and dependency relation type parent nodes of endangered languages, respectively.

[0074] B4 will reduce the syntactic feature vectors in the dimensionality-reduced B3. The input is fed into a dual affine parsing layer, where a dual affine attention mechanism is used in both the dependency arc classifier and the dependency relation classifier to obtain the dependency arc representation score S between nodes. arc The classification relationship representation score between nodes S rel The scoring function formula is as follows:

[0075]

[0076]

[0077] Among them U (*) It is the model weight matrix, H (*) I represents the syntactic feature matrix of endangered languages ​​in dependency arc prediction and relation type prediction, where I is the identity matrix.

[0078] B5 obtains the inter-node dependency arc representation score S from B4. arc The classification relationship representation score between nodes S rel Then, the first-order Eisner algorithm designed for dependency parsing tasks is used in the decoding layer. The essence of the Eisner algorithm is to find the maximum spanning tree through dynamic programming, that is, by continuously merging the analysis results of adjacent substrings, the dependency structure of the entire endangered language sentence is finally obtained.

[0079] C establishes a neural machine translation model for endangered languages, specifically Chinese, that integrates syntactic information. The Transformer is selected as the baseline model, and a neural machine translation model, TuSynTRM, integrating syntactic information is designed based on it.

[0080] C1 uses the TuParser dependency parsing model in B to determine whether there are dependency relationships between words in sentences of endangered languages, and the corresponding types of dependency relationships.

[0081] C2 integrates syntactic information features (including parts of speech and dependency relations of endangered languages) embedding and positional encoding (word order index and dominant word index of endangered languages);

[0082] C2.1 incorporates two types of syntactic information—endangered language part-of-speech tagging from A3 and dependency relation tagging from A4—as part of the input feature embedding.

[0083] For an endangered language sentence W = {w1,...,w...} containing n words... i ,...,w n}, where w i The input feature embedding e, which is extracted from W and incorporates syntactic information, represents the i-th endangered language word. i This can be expressed as Equation 5-1:

[0084]

[0085] in This represents pre-trained word vectors for endangered languages ​​(representing words in endangered languages). This represents a randomly initialized part-of-speech feature vector. Represents the feature vector of dependency relations in endangered languages. This indicates a vector concatenation operation.

[0086] C2.2 treats the word order index and dominant word index of endangered languages ​​as a form of positional information, enabling the TuSynTRM model to learn additional syntactic information about endangered languages. PEO(·) and PEH(·) represent the positional encoding functions of the word order index and dominant word index of endangered languages, respectively, and are calculated as follows:

[0087]

[0088]

[0089]

[0090]

[0091] Where, pos order pos indicates the word order index of endangered language words in a sentence. head The index of governing words in currently endangered languages, d model The dimension of the position vector is represented by , where i represents a certain dimension of the position vector.

[0092] The C3 TuSynTRM model uses traditional attention mechanisms to extract information from endangered languages ​​and Chinese, and models the relationship between the two languages.

[0093] For the input sequence of endangered language word vectors H is mapped to three matrices, denoted as Q, K, and V. After matrix multiplication, scaling, and masking, the relevance matrix is ​​obtained. This matrix is ​​then normalized using the Softmax function. Finally, the similarities of Q and K are compared with the weighted sum of V to obtain the self-attention value. The relevant formulas are shown below:

[0094] Q = W q H

[0095] K = W k H

[0096] V = W v H

[0097]

[0098] Where V is the matrix representing the input features, and Q and K are the feature matrices for calculating the Attention self-attention weights; K T It is the transpose of K, d k D is the number of columns in the Q and K matrices, i.e., the vector dimension. h This represents the length of the input word vectors for endangered languages. It is the projection matrix; This indicates a comparison of Q and K similarity.

[0099] Multi-head attention consists of multiple self-attention mechanisms. The calculation process first maps Q, K, and V into T subsets through a linear transformation, as shown in the following formula:

[0100]

[0101]

[0102]

[0103] Where t represents the t-th header. All are parameter matrices.

[0104] Then, a self-attention operation is performed on each head to obtain the single-head output head. t As shown in the formula below:

[0105] head t =Attention(Q) t ,K t V t )

[0106] Finally, after concatenating and linearly transforming the outputs of the T heads, the output of the multi-head attention is obtained, which can be represented as:

[0107] MultiHead(Q,K,V)=Concat(head1,…,head T W c

[0108] Among them W c It is the weight matrix. MultiHead(Q,K,V) represents the output of multi-head attention;

[0109] Concat(head1,…,head T This indicates that the outputs of T heads are concatenated and linearly transformed.

[0110] The present invention will be further described below in accordance with the steps and examples:

[0111] 1. Taking the northern dialect of Tujia language as the research object, a dependency structure tree library was established in a semi-automatic manner.

[0112] 1.1 A total of 6438 Tujia sentences were exported. These sentences were then exported as three lines of text in the form of "International Phonetic Alphabet, Chinese Translation, and Chinese Translation" using ELAN speech annotation software.

[0113] 1.2 Data Preprocessing

[0114] 1.2.1 After removing duplicate sentences, 6023 sentences were obtained. Subsequently, all punctuation marks appearing in the International Phonetic Alphabet layer and the Chinese translation layer were replaced with spaces.

[0115] 1.2.2 After removing duplicates and punctuation, the resulting Tujia-Chinese parallel sentence pairs were automatically segmented using spaces as delimiters, yielding 5102 perfectly aligned Tujia-Chinese parallel sentence pairs after automatic segmentation. Sentences that were not perfectly aligned after automatic segmentation were re-segmented and manually aligned. The manually adjusted sentences were then merged with the automatically segmented sentences, resulting in a total of 6023 Tujia-Chinese parallel sentence pairs.

[0116] 1.2.3 Based on the grammatical meaning of Tujia function words, English abbreviation symbols are used for marking, and during Chinese translation, the abbreviation symbols are replaced with corresponding Chinese words according to the context.

[0117] 1.3 To more accurately describe the part of speech of each word after segmentation of Tujia sentences, based on the part-of-speech tagging instructions of the jieba word segmenter and combined with the existing Tujia language corpus, 36 Tujia language part-of-speech tagging symbols were designed;

[0118] 1.4 A labeling relation table for the Tujia language dependency structure tree library was designed, with a total of 14 Tujia language dependency relation types. Corresponding relation descriptions and labeling examples are given. In the Tujia language labeling examples, the dependency relations between words are represented by bolding the text.

[0119] 1.5 Each Tujia sentence consists of the International Phonetic Alphabet (IPA) symbols of one or more words. Each Tujia word is represented by 10 fields of information, namely:

[0120] (1) ID: International Phonetic Index of Tujia words, each new sentence is marked starting from the integer 1.

[0121] (2) FORM: International Phonetic Alphabet for Tujia words.

[0122] (3) LEMMA: The root morpheme of Tujia language, which is replaced by - here.

[0123] (4) UPOSTAG: Part of speech marker in Tujia language.

[0124] (5)XPOSTAG: a specific part-of-speech tag, which is replaced by - here.

[0125] (6)FEATS: The morphological or grammatical features of the Tujia language, which are replaced by - here.

[0126] (7) HEAD: The dominant word number of the current Tujia word, which is the value of ID or 0 (i.e., the root node).

[0127] (8) DEPREL: Tujia language dependency relationship.

[0128] (9) DEPS: Second-order dependency relationship, which is replaced by - here.

[0129] (10)MISC: Chinese annotations corresponding to the International Phonetic Alphabet of Tujia words.

[0130] The content, after being automatically tagged with parts of speech and dependency syntax, was organized and a Tujia language dependency structure tree library was automatically generated by writing a Python script according to the CoNLL-U data format, which contains a total of 6023 entries.

[0131] 1.6 Manual verification, the specific verification approach is divided into three parts: Chinese annotation verification, part-of-speech tagging verification, and dependency relationship verification.

[0132] 2. Construct TuParser, a dependency parsing model for Tujia language based on a dual affine classifier;

[0133] 2.1 The embedding layer is the concatenation result of the language representation vector and the feature vector;

[0134] 2.2 The encoding layer uses three consecutive BiLSTMs to perform context encoding on the input vector output by the embedding layer.

[0135] 2.3 In the feature dimensionality reduction layer, the output vector of the coding layer is nonlinearly transformed by an MLP network.

[0136] 2.4 The dimensionality-reduced syntactic feature vectors are input into the dual affine parsing layer, and the dual affine attention mechanism is used in the dependency arc classifier and the dependency relation classifier respectively to obtain the dependency arc representation score between nodes and the classification relation representation score between nodes.

[0137] 2.5 The decoding layer uses the first-order Eisner algorithm designed for dependency parsing tasks to finally obtain the dependency structure of the entire Tujia sentence.

[0138] 3. Constructing a Tujia-Chinese neural machine translation model that integrates syntactic information

[0139] 3.1 Using the dependency parsing model TuParser, we can determine whether there are dependency relationships between words in sentences of endangered languages, and the corresponding relationship types.

[0140] 3.2 Feature embedding and position encoding that integrate syntactic information are used to extract two types of syntactic information, namely, part-of-speech tagging and dependency relation tagging of endangered languages, as part of the input feature embedding.

[0141] 3.3 By treating the word order index and dominant word index of endangered languages ​​as a kind of positional information, the TuSynTRM model can learn additional syntactic information of endangered languages.

[0142] 3.4 The TuSynTRM model uses a traditional attention mechanism to extract information from endangered languages ​​and Chinese, and models the relationship between the two languages.

[0143] Based on the above operations, a translation model for endangered languages ​​incorporating syntactic information was constructed. This model overcomes the problems of time-consuming and laborious manual annotation of endangered language corpora, the need for extensive professional knowledge, limited data volume, and poor performance of conventional neural machine translation methods.

[0144] Finally, it should be noted that the purpose of disclosing the embodiments is to help further understand the present invention. However, those skilled in the art will understand that various substitutions and modifications are possible without departing from the scope of the present invention and the appended claims. Therefore, the present invention should not be limited to the content disclosed in the embodiments, and the scope of protection of the present invention is defined by the claims.

Claims

1. A translation model method for endangered languages ​​that integrates syntactic information, characterized in that, To construct a translation model for endangered languages ​​that integrates syntactic information, the word order index, part-of-speech tagging, dominant word index, and dependency syntactic relation tagging contained in the endangered language dependency structure tree database are incorporated as syntactic features into the encoding end of the Transformer machine translation model. This process involves building a neural machine translation model for endangered languages ​​and Chinese, including the following steps: 1) Construct a dependency structure tree library for endangered languages ​​in a standard dependency syntax format using a semi-automatic approach; 2) Based on the dual affine classifier, dependency parsing of endangered languages ​​is performed. The TuParser dependency parsing model for endangered languages ​​based on the dual affine classifier is constructed to obtain the dependency structure of endangered language sentences. A graph-based approach and a dual affine classifier model are used to integrate multiple language embedding methods and encoding models to understand the structure of endangered languages. 3) Based on the Transformer machine translation model, establish a neural machine translation model TuSynTRM for endangered languages ​​(Chinese) that incorporates syntactic information; including: C1 uses a dependency parsing model to determine whether there are dependency relationships between words in sentences of endangered languages, and the corresponding types of dependency relationships; C2 fusion of syntactic information feature embedding and positional encoding; C3 employs an attention mechanism to extract information from endangered languages ​​and Chinese, and models the relationship between the two languages ​​to construct an endangered language translation model that integrates syntactic information; including: C31. Map the input endangered language word vector sequence to matrices Q, K, and V, and obtain the relevance matrix after matrix multiplication, scaling, and masking operations; C32. Perform matrix normalization, compare the similarity of Q and K to obtain the weight coefficients, and sum them with V to obtain the self-attention value; C33. Perform self-attention operation on each self-attention head to obtain a single-head output; C34. After concatenating and linearly transforming the outputs of multiple self-attention heads, the output of multi-head attention is obtained; D. Using the constructed endangered language translation model, realize the translation of endangered languages ​​that incorporates syntactic information.

2. The endangered language translation model method that integrates syntactic information as described in claim 1, characterized in that, In step 1), the standard format is specifically the CoNLL-U format.

3. The endangered language translation model method that integrates syntactic information as described in claim 1, characterized in that, In step C2, the features of syntactic information include: parts of speech and dependency relations of endangered languages; positional encoding includes word order index and dominant word index of endangered languages.

4. The endangered language translation model method that integrates syntactic information as described in claim 3, characterized in that, Step C2 includes the following specific processes: C2.1 incorporates the extracted syntactic information of endangered language part-of-speech tagging and dependency relation tagging as part of the input feature embedding; C2.2 uses the word order index and dominant word index of endangered languages ​​as positional information, and encodes the word order index and dominant word index of endangered languages ​​in a positional manner, so that the model can learn the syntactic information of endangered languages.

5. The endangered language translation model method that integrates syntactic information as described in claim 1, characterized in that, In step C3, the input sequence of endangered language word vectors H is mapped into three matrices. Three matrices, denoted as Q, K, and V, are subjected to matrix multiplication, scaling, and masking operations to obtain the relevance matrix. This matrix is ​​then normalized using the Softmax function. Finally, the weighted sum of the similarities of Q and K with V yields the self-attention value; represented as: Q=W q H K=W k H V=W v H Where V is the matrix representing the input features, and Q and K are the feature matrices used to calculate the Attention self-attention weights; d k D is the number of columns in the Q and K matrices, i.e., the vector dimension. h This represents the length of the input word vectors for endangered languages. It is a projection matrix.

6. The endangered language translation model method incorporating syntactic information as described in claim 5, characterized in that, The process of acquiring multi-head attention can be represented as follows: Q, K, and V are mapped to T subsets through a linear transformation, as shown below: Where t represents the t-th header, Both are parameter matrices; Perform self-attention operation on each head separately to obtain a single-head output head. t , represented as: head t =Attention(Q t ,K t ,V t ) After concatenating and linearly transforming the outputs of T heads, the output of the multi-head attention is obtained, which can be represented as: MultiHead(Q,K,V)=Concat(head1,…,head T )W c Among them, W c It is the weight matrix; MultiHead(Q,K,V) represents the output of multi-head attention; Concat(head1,…,head T This indicates that the outputs of T heads are concatenated and linearly transformed.

Images