Low-resource language machine translation method based on small sample machine learning and knowledge driving

Abstract

The invention provides a low-resource language machine translation method based on small sample machine learning and knowledge driving, and belongs to the field of machine learning.The method comprises the steps that firstly, an input low-resource language is preprocessed through a linguistics knowledge base, a cultural background knowledge base and a professional field knowledge base; and then establishing an end-to-end machine translation model. The model is trained on a high-resource language, then model fine tuning is carried out by using a small amount of low-resource language data, and training is carried out by adopting a small sample learning strategy. The model also uses a graph-based model and adopts a Transform neural model to decode and generate a target language sequence. Finally, an effective unsupervised evaluation mode is designed, and a language model is used for evaluation, so that the machine translation quality and performance of the low-resource language are effectively improved.

Description

Technical Field

[0001] The present invention belongs to the field of machine learning, and more specifically relates to a low-resource language machine translation method based on small sample machine learning and knowledge-driven. Background Art

[0002] In modern life, cross-language communication is becoming more and more frequent. For this reason, machine translation technology is also gradually becoming popular and developing. However, current machine translation is still mainly based on deep learning of large samples. For languages ​​with sufficient data sets, its translation effect is excellent and its accuracy is relatively high. However, with the deepening of globalization, more and more low-resource languages ​​have begun to attract attention. Due to insufficient corpus data and other reasons, the machine translation effect of these languages ​​is usually difficult to achieve as expected.

[0003] In addition, there are many challenges in applying traditional large-sample training methods to low-resource languages. For example, large-sample training methods require a large amount of manually annotated data, which is often difficult to obtain for low-resource languages. On the other hand, large-sample training methods usually require a large amount of computing resources, and the calculation process is complex and takes a lot of time, which is unacceptable for scenarios with a lack of computing resources and applications with high immediacy.

[0004] Therefore, there is an urgent need for an efficient and accurate low-resource language machine translation method. A low-resource language machine translation method based on small sample machine learning and knowledge-driven is invented to meet such needs. This method uses a small sample learning strategy for training, and enhances the input low-resource language through knowledge-driven processing to improve its translation effect. At the same time, this method also designs an effective evaluation method to evaluate the model performance to ensure the accuracy of the translation. Summary of the invention

[0005] The present invention mainly solves two technical problems. First, for low-resource languages, due to insufficient corpus data, unique linguistic characteristics and cultural background information, conventional machine translation methods are difficult to obtain ideal translation effects when processing. Therefore, the invention provides a method for preprocessing and feature extraction using a linguistic knowledge base, a cultural background knowledge base, and a professional field knowledge base; secondly, in response to the problem of small amounts of data in low-resource languages, the invention adopts a small sample learning strategy and combines it with the idea of ​​meta-learning to first train the model on a high-resource language and then fine-tune it with a small amount of low-resource language data; at the same time, since label data for low-resource languages ​​is difficult to obtain, an unsupervised evaluation method using a language model is designed to make machine translation of low-resource languages ​​more efficient and accurate.

[0006] In order to achieve the above object, the present invention is implemented by adopting the following technical scheme: the method comprises:

[0007] Use linguistic knowledge base, cultural background knowledge base, and professional domain knowledge base to pre-process the input low-resource language to enhance its expression and comprehension capabilities;

[0008] Using the preprocessed data as input, build an end-to-end machine translation model;

[0009] Use a small sample learning strategy for training, adopt the idea of ​​meta-learning, train the model on high-resource languages, and then fine-tune the model with a small amount of low-resource language data;

[0010] Design an effective evaluation method to evaluate model performance. Since labeled data for low-resource languages ​​is difficult to obtain, use language models for unsupervised evaluation.

[0011] In one embodiment, the low-resource language preprocessing includes:

[0012] Establish a knowledge base: collect linguistic knowledge, cultural background knowledge and expertise in specific fields related to low-resource languages, and automatically collect them through web crawlers;

[0013] The input text is preprocessed, including stem extraction, part-of-speech tagging and named entity recognition; the preprocessing adopts the hidden Markov model HMM, and the relevant formula is as follows:

[0014] P(W|O)=maxP(W)P(O|W), where W is the word sequence and O is the observation sequence;

[0015] According to the preprocessing results, extract features and use TF-IDF for feature extraction.

[0016] Assuming that a word appears TF times in a document, first calculate its word frequency:

[0017] TF(t) = (number of times word t appears in the document) / (number of tokens in the document)

[0018] Then calculate the value of the inverse document frequency IDF. If the document set contains N documents and word t appears in D of them, then:

[0019] IDF(t)=log e(总文件数N / 含词t的文件数D)

[0020] Finally, the TF-IDF value is calculated as the feature value of this word:

[0021] TF-IDF(t)=TF(t)*IDF(t);

[0022] Post-processing: Based on the knowledge base, the translated output is post-processed to replace common identifiers in the output text with corresponding proper nouns and terms, or to perform spelling and grammar correction on the output text.

[0023] In one solution, the translation model adopts a graph-based model, including:

[0024] Knowledge graph construction: First, construct a knowledge graph for the preprocessed data, converting each sentence or paragraph into a graph G = (V, E), where V is a node and E is an edge; each node contains a set of features, represented by H = {h1, h2, ..., hn}, where hi represents the feature vector of node i;

[0025] Next, we use a graph convolutional network to process the graph G, including node information propagation and update: first, we calculate the information aggregation of each node’s neighbor nodes, and then use this aggregate information to update the node’s features;

[0026]

[0027] Among them, A is the adjacency matrix of the graph, D is the degree matrix, and H (l) is the node feature matrix of the lth layer, W (l) is the learnable parameter of the lth layer, σ is the ReLU activation function;

[0028] After each layer of graph convolution, the hidden states of all nodes are aggregated using the readout function to form a graph-level hidden state;

[0029] The Transformer neural model is used to decode the graph-level hidden state read by GCN as input to generate the target language sequence. The decoding process is an autoregressive process, that is, only one target word is generated each time, and then the generated word sequence is input into the model as historical information to generate the next word.

[0030] In one embodiment, the small sample learning includes:

[0031] Pre-train the translation model on a high-resource language dataset to allow the model to learn common language representation knowledge and establish a semantic correspondence between the source language and the target language.

[0032] Meta-learning adjustment: Meta-learning MAML is then used to adjust the model; the goal of MAML is to find a model initialization method so that starting from this initial point, the model can adapt to the new task with a small number of steps, that is, a small number of samples.

[0033] In one embodiment, the unsupervised evaluation includes:

[0034] Train the target language model to learn and extract language information from existing large-scale unlabeled texts including web pages and public news resources; use a neural network language model. For a sentence of length N, the model's goal is to minimize the following negative log-likelihood:

[0035] Loss=-Σlog P(wi|w1,w2,...,wi-1;θ)

[0036] Where P(wi|w1,w2,...,wi-1;θ) represents the probability of generating word wi given the first i-1 words, and θ is the model parameter;

[0037] Use the trained machine translation model to translate. Apply the machine translation model trained by the above knowledge-driven and small sample learning strategies to translate the source language.

[0038] Evaluate the translation results, input the translation results into the trained language model, calculate the probability score of each generated sentence; evaluate the model based on the probability score;

[0039] Score=Σlog P(wi'|w1',w2',...,wi'-1';θ')

[0040] Among them, w1', w2', ..., wi' are the sentences generated by the translation model, P(wi'|w1', w2', ..., wi'-1'; θ') is the probability of generating word wi' given the previous i'-1 words, and θ' is the parameter of the trained language model. The higher the score, the better the translation quality.

[0041] Beneficial effects of the present invention:

[0042] 1. Improve translation results: Using linguistics, cultural background and professional domain knowledge base to preprocess and extract features of low-resource languages ​​can improve the translation model's ability to understand text content, thereby improving translation accuracy.

[0043] 2. Effectively utilize low-resource language data: Through small sample learning strategies and meta-learning ideas, the model can be trained on high-resource languages, and then a small amount of low-resource language data can be used to fine-tune the model, making better use of limited low-resource language data and saving computing resources.

[0044] 3. Improve translation efficiency: The translation model adopts a graph-based model and combines it with the Transformer neural model to improve the translation efficiency of the model. It has great application value for applications with high real-time requirements, such as online translation and video subtitle translation.

[0045] 4. Effectively evaluate translation quality: Since labeled data for low-resource languages ​​is difficult to obtain, it is difficult to evaluate translation quality using conventional supervised learning methods. The present invention provides an unsupervised evaluation method that can effectively evaluate the performance of the translation model. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] Figure 1 is a flow chart of the method of the present invention;

[0047] Figure 2 It is a flow chart of the graph-based model of the present invention;

[0048] Figure 3 This is the unsupervised evaluation flow chart of the present invention. DETAILED DESCRIPTION

[0049] In order to facilitate the understanding of the present invention, the present invention will be described more fully below with reference to the relevant drawings. Typical embodiments of the present invention are given in the drawings. However, the present invention can be implemented in many different forms and is not limited to the embodiments described in the present invention. On the contrary, the purpose of providing these embodiments is to make the disclosure of the present invention more thorough and comprehensive.

[0050] Unless otherwise defined, all technical and scientific terms used in the present invention have the same meaning as those understood by those skilled in the art of the present invention. The terms used in the specification of the present invention are only for the purpose of describing specific embodiments and are not intended to limit the present invention.

[0051] Embodiment 1

[0052] like Figure 1 As shown, a low-resource language machine translation method based on small sample machine learning and knowledge-driven includes:

[0053] S1. Use linguistic knowledge base, cultural background knowledge base, and professional field knowledge base to pre-process the input low-resource language to enhance its expression and comprehension capabilities.

[0054] Build a knowledge base: collect linguistic knowledge (such as grammar, vocabulary, pronunciation rules, etc.), cultural background knowledge (such as place names, personal names, historical events, etc.) about specific low-resource languages, and professional knowledge in specific fields (such as proper nouns and terminology in the fields of medicine, law, science and technology, etc.). The knowledge base can be automatically collected through web crawlers and other methods.

[0055] Preprocessing: Preprocess the input text according to the knowledge base. For example, perform word segmentation, part-of-speech tagging, named entity recognition, etc. on the input text, or replace proper nouns and terms in the input text with common identifiers.

[0056] Preprocess the input text, such as stem extraction, part-of-speech tagging, and named entity recognition. Language preprocessing uses the Hidden Markov Model (HMM), and the relevant formula is as follows:

[0057] P(W|O)=max_P(W)P(O|W), where W is the word sequence and O is the observation sequence.

[0058] Feature extraction: Extract features based on the results of preprocessing. For example, you can extract vocabulary features, grammatical features, context features, etc.

[0059] Take the TF-IDF (Term Frequency-Inverse Document Frequency) calculation formula as an example. This is a statistical method used to evaluate the importance of a word to a document set or one of the documents in a corpus.

[0060] Assume that a word appears TF (Term Frequency) times in a document, first calculate its term frequency:

[0061] TF(t) = (number of times word t appears in the document) / (number of tokens in the document)

[0062] Then calculate the value of the inverse document frequency IDF (Inverse Document Frequency). If the document set contains N documents, and word t appears in D of them, then:

[0063] IDF(t)=log e(总文件数N / 含词t的文件数D)

[0064] Finally, the TF-IDF value is calculated as the feature value of this word:

[0065] TF-IDF(t)=TF(t)*IDF(t).

[0066] Post-processing: Post-process the translated output according to the knowledge base, for example, replace common identifiers in the output text with corresponding proper nouns and terms, or perform spelling and grammar correction on the output text.

[0067] S2. Using the preprocessed data as input, establish an end-to-end machine translation model.

[0068] like Figure 2 As shown in Figure 1, graph-based models, such as Graph Convolutional Networks (GCN), can effectively transfer information in graph structured data, thereby capturing richer contextual information.

[0069] Here, each text obtained by knowledge-driven preprocessing can be regarded as a graph, where each node represents a word or phrase, and the edge represents the relationship between words, such as syntactic relationship, semantic relationship, etc. Then, through GCN, rich structural information can be extracted from local lexical features to perform embedding learning for each node.

[0070] S201. Knowledge graph construction—Node Representation: First, construct the knowledge graph for the preprocessed data, and convert each sentence or paragraph into a graph G=(V,E), where V is a node (which can be a word or phrase) and E is an edge (which can be a relationship between words, such as a syntactic relationship, a contextual relationship, etc.). Each node contains a set of features (such as a word vector based on TF-IDF), which can be represented as H={h1,h2,...,hn}, where hi represents the feature vector of node i.

[0071] S202, Graph Convolutional Network (GCN): Next, use the graph convolutional network to process these graphs G. The basic calculation process includes node information propagation and update: first calculate the information aggregation of each node's neighbor nodes, and then use this aggregate information to update the node's features. Expressed as:

[0072]

[0073] Among them, A is the adjacency matrix of the graph, D is the degree matrix, is the node feature matrix of the lth layer, is the learnable parameter of the lth layer, and σ is the ReLU activation function. (I+1) and H (I) Represent the node feature matrices of the lth layer and the (l+1th layer), W (I) represents the weight matrix of the lth layer.

[0074] S203, after each layer of graph convolution, use the readout function to aggregate the hidden states of all nodes to form a graph-level hidden state. The readout function includes summation, averaging, and maximization.

[0075] S204. Using a neural model such as Transformer, the graph-level hidden state read out by GCN is used as input for decoding to generate a target language sequence (the decoding process is an autoregressive process, that is, only one target word is generated each time, and then the generated word sequence is input into the model as historical information to generate the next word).

[0076] S3. Use a small sample learning strategy for training, adopt the idea of ​​meta-learning, train the model on high-resource languages, and then fine-tune the model with a small amount of low-resource language data.

[0077] After the above graph-based model, meta-learning (also known as learning how to learn) is introduced to fine-tune the model so that the model can better perform small sample learning:

[0078] S301, pre-training of high-resource languages: pre-train the above graph-based model on a dataset of high-resource languages. The goal here is to allow the model to learn common language representation knowledge so that a semantic correspondence is established between the source language and the target language.

[0079] S302, Meta-learning adjustment: Then use the meta-learning method Model-Agnostic Meta-Learning (MAML) to adjust the model. The goal of MAML is to find a model initialization method so that starting from this initial point, the model can adapt to the new task with a small number of steps (that is, a small number of samples). Here, the new task is the translation task of the low-resource language to be processed.

[0080] The training steps of MAML can be described by the following process:

[0081] (a) First, use the current model parameters θ to train a small batch of samples from a low-resource language and learn a new parameter θ'.

[0082] (b) θ' is then used to evaluate another batch of samples from the same task and the loss L is calculated.

[0083] (c) Finally, L is back-propagated to θ to update θ.

[0084]

[0085]

[0086] Here, represents the gradient, α and β represent the learning rate.

[0087] In this way, the model can quickly learn effective knowledge from a small number of low-resource language samples and achieve machine translation of low-resource languages.

[0088] S4. Design an effective evaluation method to evaluate model performance. Since labeled data for low-resource languages ​​is difficult to obtain, use language models for unsupervised evaluation.

[0089] The main purpose is to evaluate the performance of the implemented low-resource language machine translation method. Since the training data of low-resource languages ​​is scarce, directly using traditional evaluation indicators (such as BLEU, which requires a large number of corresponding manual translations as references) may not produce accurate performance evaluation results. Therefore, it is necessary to design an effective evaluation scheme to evaluate the performance of the translation model.

[0090] Unsupervised evaluation methods are mainly based on language models, and indirectly evaluate the performance of translation models by evaluating the language fluency and rationality of translation results. Expert evaluation directly invites experts with target language proficiency to evaluate the quality of translation results generated by the model.

[0091] like Figure 3 As shown, the unsupervised evaluation includes:

[0092] S401, training the target language model. Learn and extract language information from existing large-scale unlabeled text (such as web pages, news and other public resources). If a neural network language model is used, for a sentence of length N, the model's goal is to minimize the following negative log-likelihood:

[0093] Loss=-Σlog P(wi|w1,w2,...,wi-1;θ)

[0094] Where P(wi|w1,w2,...,wi-1;θ) represents the probability of generating word wi given the first i-1 words, and θ is the model parameter;

[0095] S402: Use the trained machine translation model for translation. Use the machine translation model trained by the aforementioned knowledge-driven and small sample learning strategy to translate the source language.

[0096] S403, evaluate the translation results. Input the translation results into the trained language model, calculate the probability score of each generated sentence, and evaluate the model according to the probability score.

[0097] Score=Σlog P(wi'|w1',w2',...,wi'-1';θ')

[0098] Among them, w1', w2', ..., wi' are the sentences generated by the translation model, P(wi'|w1', w2', ..., wi'-1'; θ') is the probability of generating word wi' given the previous i'-1 words, and θ' is the parameter of the trained language model. The higher the score, the better the translation quality.

[0099] Embodiment 2

[0100] Suppose you have collected a small amount of Maori (a low-resource language) corpus and want to use it to train a machine translation model. First, you collect linguistic knowledge, cultural background knowledge, and expertise in a specific field about Maori and build a knowledge base.

[0101] Preprocess the input Maori sentence. For example, there is a Maori sentence "Wahine tino toa". Use the information in the knowledge base to perform stem extraction, part-of-speech tagging and named entity recognition.

[0102] The features extracted then include the number of stems, the number of parts of speech, and the number of named entities.

[0103] Use the TF-IDF method to extract features from sentences. For "Wahine tino toa", assuming that the term frequency TF(t) = 1 / 3, the document set contains 100 documents, and the term t appears in 10 of them, then we can calculate TF-IDF(t) = (1 / 3)*log(100 / 10) = 0.1155. This is the feature value of this sentence.

[0104] For the preprocessed data, a knowledge graph was constructed, and each sentence or paragraph was converted into a graph G = {V, E}. Assuming there is only one node, the feature vector H = {0.1155}. Then the graph convolutional network was used to process the graph G, propagate and update the node information, and an updated feature vector H' = {0.2256} was obtained after calculation.

[0105] The Transformer neural model is used to decode the updated feature vector H'={0.2256} as input to generate an English sequence. After decoding, the word sequence obtained is "A brave woman". Using the knowledge base, the translation output is post-processed to replace the common identifiers with the corresponding proper nouns or terms.

[0106] Pre-training is performed on a dataset of a high-resource language, English, using a small sample learning strategy. Then, MAML is meta-learned on a small amount of Maori data to adjust the model.

[0107] Finally, an unsupervised evaluation is performed by training the model for the English language. Assume that for the sentence "A brave woman", the model's goal is to minimize the following negative log likelihood: -log(P("A"|θ'))-log(P("brave"|"A";θ'))-log(P("woman"|"A brave";θ')). Unsupervised evaluation is performed by calculating the probability score of each generated sentence and using it to evaluate the performance of the model translation.

[0108] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiments can be implemented by instructing the relevant hardware through a computer program, and the program can be stored in a computer-readable storage medium, and when the program is executed, it can include the processes of the embodiments of the above-mentioned methods. The storage medium can be a disk, an optical disk, a read-only memory (ROM) or a random access memory (RAM).

[0109] It should be understood that the detailed description of the technical solutions of the present invention by means of the preferred embodiments is illustrative rather than restrictive. A person skilled in the art may modify the technical solutions described in the embodiments, or replace some of the technical features by equivalents, based on reading the specification of the present invention; and these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A low-resource language machine translation method based on small sample machine learning and knowledge-driven, characterized by: The method includes: Use linguistic knowledge base, cultural background knowledge base, and professional domain knowledge base to pre-process the input low-resource language to enhance its expression and comprehension capabilities; Using the preprocessed data as input, build an end-to-end machine translation model; Use a small sample learning strategy for training, adopt the idea of ​​meta-learning, train the model on high-resource languages, and then fine-tune the model with a small amount of low-resource language data; Design an effective evaluation method to evaluate model performance. Since labeled data for low-resource languages ​​is difficult to obtain, use language models for unsupervised evaluation.

2. The low-resource language machine translation method based on small sample machine learning and knowledge-driven according to claim 1, characterized in that: The low-resource language preprocessing includes: Establish a knowledge base: collect linguistic knowledge, cultural background knowledge and expertise in specific fields related to low-resource languages, and automatically collect them through web crawlers; The input text is preprocessed, including stem extraction, part-of-speech tagging and named entity recognition; the preprocessing adopts the hidden Markov model HMM, and the relevant formula is as follows: P(W|O)=maxP(W)P(O|W), where W is the word sequence and O is the observation sequence; according to the preprocessing results, extract features and use TF-IDF for feature extraction. Assuming that a word appears TF times in a document, first calculate its word frequency: TF(t) = (number of times word t appears in the document) / (number of tokens in the document) Then calculate the value of the inverse document frequency IDF. If the document set contains N documents and word t appears in D of them, then: IDF(t)=log e(总文件数N / 含词t的文件数D) Finally, the TF-IDF value is calculated as the feature value of this word: TF-IDF(t)=TF(t)*IDF(t); Post-processing: Based on the knowledge base, the translated output is post-processed to replace common identifiers in the output text with corresponding proper nouns and terms, or to perform spelling and grammar correction on the output text.

3. The low-resource language machine translation method based on small sample machine learning and knowledge-driven according to claim 1, characterized in that: The translation model adopts a graph-based model, including: Knowledge graph construction: First, construct a knowledge graph for the preprocessed data, converting each sentence or paragraph into a graph G = (V, E), where V is a node and E is an edge; each node contains a set of features, represented by H = {h1, h2, ..., hn}, where hi represents the feature vector of node i; Next, we use a graph convolutional network to process the graph G, including node information propagation and update: first, we calculate the information aggregation of each node’s neighbor nodes, and then use this aggregate information to update the node’s features; Among them, A is the adjacency matrix of the graph, D is the degree matrix, and H (l) is the node feature matrix of the lth layer, W (l) is the learnable parameter of the lth layer, σ is the ReLU activation function; After each layer of graph convolution, the hidden states of all nodes are aggregated using the readout function to form a graph-level hidden state; The Transformer neural model is used to decode the graph-level hidden state read by GCN as input to generate the target language sequence. The decoding process is an autoregressive process, that is, only one target word is generated each time, and then the generated word sequence is input into the model as historical information to generate the next word.

4. The low-resource language machine translation method based on small sample machine learning and knowledge-driven according to claim 1, characterized in that: The small sample learning includes: Pre-train the translation model on a high-resource language dataset to allow the model to learn common language representation knowledge and establish a semantic correspondence between the source language and the target language. Meta-learning adjustment: Meta-learning MAML is then used to adjust the model; the goal of MAML is to find a model initialization method so that starting from this initial point, the model can adapt to the new task with a small number of steps, that is, a small number of samples.

5. The low-resource language machine translation method based on small sample machine learning and knowledge-driven according to claim 1, characterized in that: The unsupervised evaluation described includes: Train the target language model to learn and extract language information from existing large-scale unlabeled texts including web pages and public news resources; use a neural network language model. For a sentence of length N, the model's goal is to minimize the following negative log-likelihood: Loss=-Σlog P(wi|w1,w2,...,wi-1;θ) Where P(wi|w1,w2,...,wi-1;θ) represents the probability of generating word wi given the first i-1 words, and θ is the model parameter; Use the trained machine translation model to translate. Apply the machine translation model trained by the above knowledge-driven and small sample learning strategies to translate the source language. Evaluate the translation results, input the translation results into the trained language model, calculate the probability score of each generated sentence; evaluate the model based on the probability score; Score=Σlog P(wi'|w1',w2',...,wi'-1';θ') Among them, w1', w2', ..., wi' are the sentences generated by the translation model, P(wi'|w1', w2', ..., wi'-1'; θ') is the probability of generating word wi' given the previous i'-1 words, and θ' is the parameter of the trained language model. The higher the score, the better the translation quality.

Images