Method, medium and device for improving automatic evaluation of machine translation quality using retrieval

Abstract

The present invention discloses a method, medium, and device for improving automatic machine translation quality assessment using retrieval. The method comprises: a retrieval phase: for machine translation quality assessment sentence pairs, relevant parallel sentence pairs are retrieved from a database for the words to be assessed in the machine translation quality assessment sentence pairs; and a machine translation quality assessment phase: the retrieved parallel sentence pairs are encoded and incorporated into a machine translation quality assessment model. The present invention can directly and effectively utilize relevant parallel sentence pairs, while also alleviating the problem of sparse training data for machine translation quality assessment. It also better explains the reasons why the model makes relevant decisions, eliminates the need for model retraining, and avoids the drawback of end-to-end models forgetting training data during training, thereby improving the performance of machine translation quality assessment models.

Description

Technical Field

[0001] The present invention relates to the technical field of natural language processing, and in particular to a method, medium and device for automatically evaluating machine translation quality by utilizing retrieval. Background Art

[0002] With the accelerating globalization of today's society, machine translation technology has become crucial as a bridge connecting different languages. However, the quality of machine translation still lags behind that of human translation, necessitating quality assessment to help humans better utilize the results. Automatic machine translation quality assessment is the process of outsourcing this work to machines. In recent years, automatic machine translation quality assessment has garnered increasing attention and has become a widely researched and discussed topic in the field.

[0003] Currently, automatic evaluation of machine translation quality can be roughly divided into two directions. The first is machine translation quality evaluation with reference translations, that is, to provide a quantifiable indicator by comparing the output of the machine translation system with the reference translation. Classic works include BLEU [Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. BLEU: a method for automatic evaluation of machine translation. Proceedings of the 40th Annual Meeting on Association for Computational Linguistics: 311–318.] and METEOR [Alon Lavie and Abhaya Agarwal. 2007. Meteor: an automatic metric for MT evaluation with high levels of correlation with human judgments. Proceedings of the 45th Annual Meeting on Association for Computational Linguistics: 311–318.] Linguistics:228–231.] et al.

[0004] The second direction is more common, namely, machine translation quality assessment without a reference translation. Given only a bilingual sentence pair, consisting of a source sentence and a translation by a machine translation system, the quality assessment system automatically determines the quality of the translation. In this task, translation quality is assessed at both the word level and the sentence level. The word-level quality assessment involves judging the quality of each word in the translation, while the sentence-level quality assessment, like the previous direction, requires a score to indicate the overall quality of the translation. The current approach to this task is to manually annotate some examples, treating the problem as a supervised learning prediction task, and then learning through a deep network model.

[0005] Currently, effective annotated data for machine translation quality assessment is extremely scarce. Public datasets often contain only a few thousand records for a single language pair. To alleviate this data sparsity, researchers are currently exploring the use of external resources to aid in the training of machine translation quality assessment tasks. There are currently two commonly used methods. One is to use a large-scale cross-lingual pre-trained model, take the pre-trained model as the basis, and then fine-tune it using the supervised data of machine translation quality assessment [Reference Tharindu Ranasinghe, Constantin Orasan,and Ruslan Mitkov.2020.TransQuest:Translation Quality Estimation with Cross-lingual Transformers.(TransQuest: Using cross-lingual Transformers to perform translation quality assessment)Proceedings of the 28th International Conference on Computational Linguistics:5070–5081.Reference Tharindu Ranasinghe,Constantin Orasan,and Ruslan Mitkov.2021.An Exploratory Analysis ofMultilingual Word-Level Quality Estimation with Cross-Lingual Transformers.(Exploratory analysis of multilingual word-level quality assessment with cross-lingual Transformers))Proceedings of the59th Annual Meeting of the Association for Computational Linguistics and the11th International Joint Conference on Natural Language Processing:434–440.】;Secondly, with the help of large-scale parallel corpora, fake machine translation quality evaluation data is generated through parallel corpora to supplement the original data [Cui, Qu, Shujian Huang, Jiahuan Li, Xiang Geng, Zaixiang Zheng, Guoping Huang, and Jiajun Chen."Directqe:Direct pretraining for machinetranslation quality estimation.(DirectQE: Direct pretraining for machine translation quality estimation.)"In Proceedings of the AAAI Conference on Artificial Intelligence, vol.35,no.14,pp.12719-12727.2021.Zheng, Yuanhang, Zhixing Tan, Meng Zhang, Mieradilijiang Maimaiti, Huanbo Luan, Maosong Sun, Qun Liu, and Yang Liu."Self-Supervised Quality Estimation for Machine Translation.(Machine Translation Quality Evaluation Based on Self-Supervision Method)"In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing,pp.3322-3334.】.

[0006] Although both types of methods have effectively improved the performance of machine translation quality assessment, they are still completely based on black-box deep neural networks for end-to-end training and prediction, and therefore lack interpretability and flexibility, and their utilization of external parallel data is not complete.

[0007] First, explainability. When evaluating the quality of machine translation, when the model determines whether a translated word is OK or BAD, humans often want to know the reasons behind its judgment. This increases the credibility of the model's predictions and allows humans to better assess the model's judgments. However, the black-box nature of current deep neural network models significantly limits the interpretability of their outputs.

[0008] Secondly, flexibility is a key issue. Current machine translation quality assessment models are typically trained once and then applied to all subsequent machine translation quality assessments. This approach has limitations in some scenarios. Consider the following scenario: the model is trained on quality assessment data in the news field and then tested and evaluated on data in the medical field. Obviously, the existing model will suffer from performance degradation in this scenario. To solve this problem, the model needs to collect relevant samples again for continued training. However, on the one hand, training the model itself requires a very high overhead. On the other hand, due to the catastrophic forgetting nature of neural networks themselves, the model may forget the original knowledge during the training process, resulting in the inability to continuously reuse it.

[0009] Finally, there's the issue of complete utilization of external parallel data. While the parallel corpus used for forged quality assessment data may contain samples relevant to the machine translation being evaluated, information about these relevant samples may not be stored in the model during training. Therefore, when this information is needed to assist in judgment, existing models cannot guarantee the complete utilization of relevant parallel sentence pairs in the external data. Summary of the Invention

[0010] To overcome the problems of the existing machine translation quality assessment models in the above background technology, which have deficiencies in interpretability, flexibility, and completeness of utilizing external parallel corpora, the present invention provides a method, storage medium, and electronic device for improving the automatic assessment of machine translation quality by using retrieval.

[0011] To achieve the above objectives, a first aspect of the present invention provides a method for improving automatic evaluation of machine translation quality using retrieval, comprising:

[0012] Retrieval stage: for a machine translation quality assessment sentence pair, relevant parallel sentence pairs are retrieved from the database for the words to be assessed in the machine translation quality assessment sentence pair;

[0013] Machine translation quality assessment stage: The retrieved parallel sentence pairs are encoded and integrated into the machine translation quality assessment model.

[0014] In some possible implementations, the retrieval stage includes the following steps:

[0015] Step 1: Construct a database using parallel sentence pairs;

[0016] Step 2: For a machine translation quality assessment sentence pair, construct a query sequence for the words to be assessed in the machine translation quality assessment sentence pair in the database and perform retrieval;

[0017] Step 3: For the words to be evaluated in the machine translation quality evaluation sentence pairs, sort the retrieved parallel sentence pairs and retain the required parallel sentence pairs.

[0018] In some possible implementations, for all parallel sentence pairs, the database constructs an inverted index for each word in each parallel sentence pair, that is, when searching for any word in a parallel sentence pair, the parallel sentence pair can be retrieved.

[0019] In some possible implementations, the machine translation quality assessment stage includes the following steps:

[0020] Step 4: Encode the machine translation quality assessment sentence pair and the retrieved parallel sentence pair corresponding to the word to be assessed using a cross-language pre-trained model, respectively, to obtain the hidden layer representation of the word to be assessed in the machine translation quality assessment sentence pair and the hidden layer representation of the retrieved parallel sentence pair;

[0021] Step 5: concatenate the hidden layer representations of the retrieved parallel sentence pairs corresponding to the word to be evaluated;

[0022] Step 6: Use the machine translation quality evaluation sentence to extract information from the latent layer representation of the word to be evaluated using multi-head attention from the latent layer representation concatenated in step 5;

[0023] Step 7: The hidden layer representation of the word to be evaluated in the machine translation quality evaluation sentence pair and the hidden layer representation after information extraction in step 6 are fused through a gating mechanism to obtain the final representation, which is input into a multi-layer perceptron for classification to obtain the translation accuracy of the word to be evaluated.

[0024] In some possible implementations, the retrieval stage specifically includes the following steps:

[0025] Step 11: The database constructs a search engine using Lucene. Lucene uses the FST structure to construct an index, sharing an index path for words with the same prefix. That is, given a parallel sentence pair, Lucene constructs an inverted index for each word in the parallel sentence pair, including the source language and the target language.

[0026] Step 12: Assume that the machine translation quality assessment sentence pair is: X = (x1, ..., x i ,...,x m), translated as: Y=(y1,...,y j ,...,y n ), for the word y to be evaluated in the machine translation quality evaluation sentence pair j , the query sequence is:

[0027] MUST(y j )∧SHOULD(x1)∧...∧SHOULD(x m )∧SHOULD(y1)∧...∧SHOULD(y n );

[0028] Step 13: For the word to be evaluated, all its retrieved parallel sentence pairs are sorted using BM25, and the top-k results are retained.

[0029] In some possible implementations, the machine translation quality assessment stage specifically includes the following steps:

[0030] Step 44: Encode the machine translation quality assessment sentence pair through XLMR to obtain:

[0031]

[0032] For the word y to be evaluated j , extract its h MT The hidden layer representation of the corresponding position in In addition, for the word y to be evaluated j The corresponding retrieved parallel sentence pairs are:

[0033]

[0034] R1 to R k Encode using XLMR in the same way and obtain the hidden state of the last layer respectively to obtain:

[0035]

[0036] Step 55: to Stitched together to get

[0037] Step 66: Place the y j The corresponding hidden layer representation For h R Through MultiHead Attention (multi-head attention) to extract information, we get h R-Extract ,

[0038]

[0039] Step 77: and h R-Extract Through a gating mechanism, h final , h final Input into the multi-layer perceptron for classification, and output OK / BAD, that is, the word y to be evaluated j The translation is right or wrong.

[0040] In some possible implementations, the gating mechanism is:

[0041] described

[0042] In some possible implementations, if no parallel sentence pair containing the current evaluation word is found in the parallel corpus, then h R-Extract The value is 0, and the g value output by the gating mechanism model is also close to 0, that is, the model does not use the retrieval result; after obtaining the fused representation, h is converted to final Input into the multi-layer perceptron for classification.

[0043] A second aspect of the present invention provides an electronic device for improving automatic evaluation of machine translation quality using retrieval, the electronic device comprising a processor and a memory:

[0044] The memory is used to store program code and transmit the program code to the processor;

[0045] The processor is configured to execute the above-mentioned method for improving automatic evaluation of machine translation quality by using word-level fine-grained bilingual retrieval according to the instructions in the program code.

[0046] According to a third aspect of the present invention, a computer-readable storage medium is provided, wherein the computer-readable storage medium is used to store program code, wherein the program code is used to execute the above-mentioned method for improving the automatic evaluation of machine translation quality by using word-level fine-grained bilingual retrieval.

[0047] The beneficial effects of the present invention are:

[0048] 1. The present invention proposes, for the first time, a method for improving the automatic evaluation of machine translation quality through retrieval. This method uses fine-grained bilingual retrieval at the word level to assist in machine translation quality evaluation. This method can directly and effectively utilize relevant parallel sentence pairs, while also alleviating the problem of sparse training data for machine translation quality evaluation.

[0049] 2. The method proposed in this invention for automatically evaluating machine translation quality using retrieval has better interpretability than existing solutions. The model's judgment in this invention comes directly from the retrieved examples, which better explains why the model makes relevant decisions.

[0050] 3. Compared to existing methods, this method for improving automatic machine translation quality assessment through retrieval offers greater flexibility in the use of parallel sentence pairs. By simply replacing the database used for retrieval, the model can be adapted to new machine translation quality assessment scenarios, eliminating the need for retraining the model as required by existing methods.

[0051] 4. This method of improving automatic machine translation quality assessment through retrieval makes more comprehensive use of external parallel sentence pairs. In this method, all parallel sentence pairs are stored in a database and can be retrieved. This avoids the drawback of end-to-end models that can forget training data during training, thereby improving the performance of machine translation quality assessment models. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] Figure 1 This is a flowchart of the steps in the retrieval phase of the method for improving automatic evaluation of machine translation quality by using retrieval in an embodiment of the present application;

[0053] Figure 2 This is a flowchart of the steps in the machine translation quality assessment phase of the method for improving automatic machine translation quality assessment by using retrieval in an embodiment of the present application;

[0054] Figure 3 This is a model diagram of XLMR in Example 1 of the present application;

[0055] Figure 4 This is a schematic diagram of the model in Example 1 of the present application;

[0056] Figure 5 This is a schematic diagram of the model in Example 2 of the present application;

[0057] Figure 6 This is a structural diagram of an electronic device that uses retrieval to improve automatic evaluation of machine translation quality in an embodiment of the present application.

[0058] In the figure: 50, electronic device; 51, processor; 52, memory. DETAILED DESCRIPTION

[0059] The preferred embodiments of the present invention are described in detail below with reference to the accompanying drawings so that the advantages and features of the present invention can be more easily understood by those skilled in the art, thereby making a clearer and more precise definition of the protection scope of the present invention.

[0060] Existing machine translation quality assessment models have shortcomings in interpretability, flexibility, and the completeness of utilizing external parallel sentence pairs. To address these problems, the present invention proposes a method for automatically assessing machine translation quality by using retrieved parallel data.

[0061] This paper combines the characteristics of the machine translation quality assessment task itself and proposes a novel fine-grained bilingual retrieval scheme based on the word level; combined with the characteristics of the existing machine translation quality assessment model, it proposes a simple and effective mechanism to integrate retrieval results into word-level machine translation quality assessment.

[0062] The work of this invention mainly focuses on word-level machine translation quality assessment without reference translation. Specifically, given the source language machine translation quality assessment sentence pair X = (x1, ..., x i ,…,x m )(where x i represents the i-th word in the source language sentence, m represents the length of the sentence), and the translation Y = (y1,…,y j ,…,y n )(where y j represents the jth word in the source language sentence, and n represents the length of the sentence). A machine translation evaluation system is needed to give the word y to be evaluated in the translation Y. j The translation accuracy is OK if it is accurate, otherwise it is BAD.

[0063] A first aspect of this embodiment is a method for improving automatic evaluation of machine translation quality using retrieval, including:

[0064] Retrieval stage: For machine translation quality assessment sentence pairs, relevant parallel sentence pairs are retrieved from the database for the words to be assessed in the machine translation quality assessment sentence pairs. For all parallel sentence pairs, the database constructs an inverted index for each word in each parallel sentence pair, that is, when searching for any word in a parallel sentence pair, this parallel sentence pair can be retrieved.

[0065] The retrieval phase includes the following steps:

[0066] Step 1: Construct a database using parallel sentence pairs;

[0067] Step 2: For a machine translation quality assessment sentence pair, construct a query sequence for the words to be assessed in the machine translation quality assessment sentence pair in the database and perform retrieval;

[0068] Step 3: For the words to be evaluated in the machine translation quality evaluation sentence pairs, sort the retrieved parallel sentence pairs and retain the required parallel sentence pairs.

[0069] Machine translation quality assessment stage: The retrieved parallel sentence pairs are encoded and integrated into the machine translation quality assessment model.

[0070] The machine translation quality assessment stage includes the following steps:

[0071] Step 4: Encode the machine translation quality assessment sentence pair and the retrieved parallel sentence pair corresponding to the word to be assessed using a cross-language pre-trained model, and obtain the hidden layer representation of the word to be assessed in the machine translation quality assessment sentence pair and the hidden layer representation of the retrieved parallel sentence pair respectively;

[0072] Step 5: concatenate the hidden layer representations obtained from the retrieved parallel sentence pairs corresponding to the word to be evaluated;

[0073] Step 6: Using the hidden layer representation of the word to be evaluated in the machine translation quality evaluation sentence pair, extract information from the latent layer representation after splicing in step 5 using multi-head attention;

[0074] Step 7: The hidden layer representation of the word to be evaluated in the machine translation quality evaluation sentence pair and the hidden layer representation after information extraction in step 6 are fused through a gating mechanism to obtain the final representation, which is input into a multi-layer perceptron for classification to obtain the translation accuracy of the word to be evaluated.

[0075] The method for improving automatic evaluation of machine translation quality by using retrieval specifically includes the following steps:

[0076] The retrieval stage specifically includes the following steps:

[0077] Step 11: The database constructs a search engine using Lucene. The Lucene uses the FST structure to construct an index, and shares the index path for words with the same prefix. That is, given a bilingual parallel sentence pair, the Lucene constructs an inverted index for each word in the source language and the target language in the bilingual parallel sentence pair. Using Lucene (Reference Andrzej, Robert Muir, Grant Ingersoll, and Lucid Imagination. "Apache lucene 4." In SIGIR 2012 workshop on open source information retrieval, p. 17. 2012.) Constructing a search engine.

[0078] Searching for any word in this parallel sentence pair quickly finds it in a search engine. Lucene uses the FST (Finite State Transducer) structure to construct an index, sharing the index path for words with the same prefix. For example, when the word "source" is entered, Lucene searches the storage structure in the order "s->o->u->r->ce" to ultimately find the parallel sentence pair containing "source."

[0079] Step 12: Assume that the machine translation quality assessment sentence pair is: X = (x1, ..., x i ,...,x m ), translated as: Y=(y1,...,y j ,...,y n ), for the word y to be evaluated in the machine translation quality evaluation sentence pair j , the query sequence is:

[0080] MUST(y j )∧SHOULD(x1)∧...∧SHOULD(x m )∧SHOULD(y1)∧...∧SHOULD(y n );

[0081] That is, the search results must contain y j , and contains the rest of the words in the sample.

[0082] Step 13: Sort all the retrieved parallel sentence pairs using BM25, and retain the top-k results.

[0083] Using BM25 Jones, Karen, S. Walker, and Stephen E. Robertson. 2000. A probabilistic model of information retrieval: Development and comparative experiments. IP&M 36(6): 779-808, 809-840.] sort the results and retain the top-k results as needed. Specifically, BM25 calculates the similarity between two sentences (in this example, the bilingual sentence pairs are concatenated into one sentence). It is an optimization of the classic method TF-IDF and also takes into account term frequency and inverse document frequency. The calculation formula of BM25 is as follows:

[0084]

[0085] Where Q is the machine translation quality assessment sentence pair, q i represents each word in the machine translation quality assessment sentence pair, and d represents a bilingual parallel sentence pair retrieved from the database, R(q i ,d) used to calculate q i The frequency of occurrence in d is calculated as follows:

[0086]

[0087] W i To calculate q i The frequency of occurrence in the entire database is calculated as follows:

[0088]

[0089] In the above formula, k1, k2 and b are predefined coordination factors, and N represents q i The total number of occurrences in the database, f i Indicates q i The number of times qf appears in d i Indicates q i The number of times it appears in Q, dl represents the length of Q, and avg_dl represents the average sentence length in the database.

[0090] Such a search scheme can ensure that the retrieved parallel sentence pairs contain the words to be translated into y j The required information should be obtained, and the translation context should be as similar as possible to the current context to be evaluated, so that the model can better utilize the retrieved data.

[0091] The machine translation quality assessment stage specifically includes the following steps:

[0092] Step 44: Comprehensively consider the information contained in the pre-trained model and the information contained in the retrieved parallel sentence pairs. Suppose that for the word y to be evaluated in the machine translation quality evaluation sentence pair j , the retrieved parallel sentence pairs are: R1 to R k Encode using XLMR in the same way and obtain the hidden state of the last layer respectively to obtain:

[0093] In addition, the machine translation quality evaluation sentence pair is also encoded through XLMR to obtain:

[0094] In the most commonly used machine translation quality assessment model, the source sentence X and the translation Y are usually concatenated and input into the cross-language pre-trained model XLMR [Conneau, Alexis, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. "Unsupervised cross-lingual representation learning at scale." arXiv preprint arXiv:1911.02116 (2019).] to obtain the hidden state of the last layer, which is then input into a multi-layer perceptron for binary classification. See the attached Figure 3 shown.

[0095] Step 55: to Stitched together to get

[0096] Step 66: The word to be evaluated y j In h MT The corresponding position in For h R By [Document Vaswani, Ashish, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Kaiser,and Illia Polosukhin. "Attention is all you need."Advances in neural information processing systems 30(2017).】The proposed MultiHead Attention extracts effective information from the retrieval results and obtains h R-Extract ,

[0097] Step 77: Place h y and h R-Extract Through a gating mechanism, h final , h final The input is fed into a multilayer perceptron for classification and the output is OK / BAD.

[0098] The gating mechanism is:

[0099] described

[0100] If no sentence pair containing the current evaluation word can be found in the parallel corpus, then h R-Extract The value is 0, and the g value output by the gating mechanism model is also close to 0, that is, the model does not use the retrieval result; after obtaining the fused representation, h is converted to final Input into the multi-layer perceptron for classification, refer to the attached Figure 4 shown.

[0101] Through the above steps, the present invention can find relevant parallel sentence pairs for the translation word to be evaluated in a fine-grained manner, and effectively integrate the information in the parallel sentence pairs into the evaluation of the translation word.

[0102] Example 1: Assume that the model currently needs to evaluate the machine translation quality of the sentence pair "The otter feeds mainly on fish. The kettle mainly uses fish as food" to determine the translation accuracy of "kettle". First, according to Figure 1 The search process is shown. The following query is constructed: MUST(kettle) ∧ SHOULD(The) ∧… ∧ SHOULD(fish) ∧ SHOULD(mainly) ∧… ∧ SHOULD(food). This query is then searched against the database constructed using parallel sentence pairs. Two search results are retained, and the top-scoring parallel sentence pairs, based on the BM25 score, are: "The kettle was singing on the cooker." and "The kettle was steaming on the stove."

[0103] Next, follow Figure 2 The process shown in Figure 1 integrates the retrieved information into the quality assessment of machine translation. Figure 4 As shown in the figure, first, the machine translation quality assessment sentence pair and the parallel sentence pair corresponding to the word to be assessed are respectively encoded using the cross-language pre-trained model XLM-R to obtain the hidden layer representation of the last layer output of the model. The hidden layer representation of the machine translation quality assessment sentence pair to be assessed is:

[0104] h MT =[h The ,…,h fish ,h 水壶 ,…,h 食物 ];

[0105] The hidden representation of the retrieved parallel sentence pairs is encoded as:

[0106]

[0107] Concatenate the hidden representations of the retrieved parallel sentence pairs to obtain:

[0108]

[0109] Then use h MT The hidden layer representation h of the “kettle” 水壶 After splicing, it represents h R Using multi-head attention for information extraction, we get:

[0110] h R-Extract =Multihead-Attention(h 水壶 ,h R ,h R ).

[0111] Next, the hidden layer representation of “kettle” and the hidden layer representation extracted based on the retrieval information are fused through gating to obtain the final representation:

[0112] h final =g 水壶 ·h R-Extract +(1-g 水壶 )·h 水壶 , g 水壶 =MLP([h R-Extract ;h 水壶 ]).

[0113] Finally, we will express h final The input is fed into a multi-layer perceptron for classification, and the output is the translation accuracy of “kettle”.

[0114] In this actual example, the model can learn from the information contained in the retrieved parallel sentence pairs that "kettle" is the correct translation for "water kettle". Therefore, the current translation of "otter" as "water kettle" is incorrect, and the model will output OK.

[0115] Example 2: Assume that the model currently needs to evaluate the machine translation quality of the sentence pair "The last conquistador then rides on with his sword drawn. The last conqueror rode his sword and continued to move forward" to determine the translation accuracy of "conqueror". First, according to Figure 1 The process shown is used for retrieval. Construct the following retrieval query:

[0116] MUST (conqueror) ∧ SHOULD (The) ∧… ∧ SHOULD (drawn) ∧ SHOULD (finally) ∧… ∧ SHOULD (continue) and search the database constructed using parallel sentence pairs. Set to retain two search results. The top search sentence pairs based on the BM25 score are: "There is conquistador gold on the island." and "The entire population of the town was put to the sword by the conquistador."

[0117] Next, you need to refer to the attached Figure 2 The process shown incorporates the retrieved information into the quality assessment of machine translation. Figure 5 As shown in the figure, first, the machine translation quality evaluation sentence pairs and the parallel sentence pairs corresponding to the words to be evaluated are respectively encoded using the cross-language pre-training model XLM-R to obtain the hidden layer representation of the last layer output of the model.

[0118] The hidden layer representation of the machine translation quality assessment sentence pair to be evaluated is:

[0119] h MT =[h The ,…,h drawn ,h 最后 ,…,h 征服者 …,h 继续前进 ];

[0120] The hidden representation of the retrieved parallel sentence pairs is encoded as:

[0121]

[0122] Concatenate the hidden representations of the retrieved parallel sentence pairs to obtain:

[0123]

[0124] Then use h MT The hidden layer representation h of the "conqueror" 征服者 After splicing, it represents h R Using multi-head attention for information extraction, we get:

[0125] h R-Extract =Multihead-Attention(h 征服者 ,h R ,h R );

[0126] Next, the hidden layer representation of the “Conqueror” and the hidden layer representation extracted based on the retrieval information are fused through gating to obtain the final representation:

[0127] h final =g 征服者 ·h R-Extract +(1-g 征服者 )·h 征服者 , g 征服者 =MLP([h R-Extract ;h 征服者 ]).

[0128] Finally, we will express h final The input is fed into a multi-layer perceptron for classification, and the output is the translation accuracy of “Conqueror”.

[0129] In this example, the model can determine from the information contained in the retrieved parallel sentence pair that "conqueror" is the correct translation for "conquistador." Therefore, the current translation of "conquistador" as "conqueror" is correct, and the model outputs OK.

[0130] Refer to the attached Figure 6 As shown, the present invention provides an electronic device for automatically evaluating the quality of machine translation by using retrieval, such as Figure 6 As shown, the electronic device 50 includes a processor 51 and a memory 52 coupled to the processor 51 .

[0131] The memory 52 is used to store program codes and transmit the program codes to the processor 51;

[0132] The processor 51 is configured to execute the above-mentioned method for improving automatic evaluation of machine translation quality by using retrieval according to the instructions in the program code.

[0133] The memory 52 stores program instructions for implementing the method for improving automatic evaluation of machine translation quality by using retrieval in the above embodiment or the method for improving automatic evaluation of machine translation quality by using retrieval in the above embodiment.

[0134] The processor 51 is configured to execute program instructions stored in the memory 52 to perform automatic evaluation of machine translation quality through retrieval.

[0135] The processor 51 may also be referred to as a CPU (Central Processing Unit). The processor 51 may be an integrated circuit chip having signal processing capabilities. The processor 51 may be:

[0136] DSP (Digital Signal Processor, Digital Signal Processor, Digital Signal Processor is a processor composed of large-scale or ultra-large-scale integrated circuit chips used to complete certain signal processing tasks. It has gradually developed to meet the needs of high-speed real-time signal processing tasks. With the development of integrated circuit technology and digital signal processing algorithms, the implementation methods of digital signal processors are constantly changing, and the processing functions are constantly improving and expanding.)

[0137] ASIC (Application Specific Integrated Circuit) is an integrated circuit designed and manufactured to meet specific user requirements and the needs of specific electronic systems.

[0138] The FPGA (Field Programmable Gate Array) is a further development of programmable devices such as PAL (Programmable Array Logic) and GAL (Generic Array Logic). It emerged as a semi-custom circuit within the field of application-specific integrated circuits (ASICs), addressing both the shortcomings of custom circuits and the limited number of gates inherent in existing programmable devices.

[0139] A general-purpose processor may be a microprocessor or any conventional processor.

[0140] Other programmable logic devices, discrete gate or transistor logic devices and discrete hardware components, etc.

[0141] This embodiment further provides a computer-readable storage medium for storing program code, and the program code is used to execute the above-mentioned method for improving automatic evaluation of machine translation quality by using retrieval.

[0142] The storage medium stores program instructions capable of implementing all of the aforementioned methods. The program instructions may be stored in the storage medium in the form of a software product, including instructions for causing a computer device (such as a personal computer, server, or network device) or a processor to perform all or part of the steps of the methods described in various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as a USB flash drive, a mobile hard drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk, or terminal devices such as a computer, server, mobile phone, or tablet.

[0143] The above description is only an embodiment of the present invention. It should be pointed out that those skilled in the art can make improvements without departing from the creative concept of the present invention, but these improvements all fall within the scope of protection of the present invention.

Claims

1. A method for improving automatic evaluation of machine translation quality using retrieval, characterized in that: include: Retrieval stage: for a machine translation quality assessment sentence pair, relevant parallel sentence pairs are retrieved from the database for the words to be assessed in the machine translation quality assessment sentence pair; Machine translation quality assessment stage: After encoding the retrieved parallel sentence pairs, integrate them into the machine translation In the translation quality assessment model; The machine translation quality assessment stage includes the following steps: Step 4: Encode the machine translation quality assessment sentence pair and the retrieved parallel sentence pair corresponding to the word to be assessed using a cross-language pre-trained model, and obtain the hidden layer representation of the word to be assessed in the machine translation quality assessment sentence pair and the hidden layer representation of the retrieved parallel sentence pair respectively; Step 5: concatenate the hidden layer representations obtained from the retrieved parallel sentence pairs corresponding to the word to be evaluated; Step 6: Using the hidden layer representation of the word to be evaluated in the machine translation quality evaluation sentence pair, extract information from the latent layer representation after splicing in step 5 using multi-head attention; Step 7: The hidden layer representation of the word to be evaluated in the machine translation quality evaluation sentence pair and the hidden layer representation after information extraction in step 6 are fused through a gating mechanism to obtain the final representation, which is input into a multi-layer perceptron for classification to obtain the translation accuracy of the word to be evaluated.

2. The method for automatically evaluating machine translation quality using retrieval according to claim 1, wherein: The retrieval phase includes the following steps: Step 1: Construct a database using parallel sentence pairs; Step 2: For a machine translation quality assessment sentence pair, construct a query sequence for the words to be assessed in the machine translation quality assessment sentence pair in the database and perform retrieval; Step 3: For the words to be evaluated in the machine translation quality evaluation sentence pairs, sort the retrieved parallel sentence pairs and retain the required parallel sentence pairs.

3. The method for automatically evaluating the quality of machine translation by using retrieval according to claim 1 or 2, characterized in that: For all parallel sentence pairs, the database constructs an inverted index for each word in each parallel sentence pair, that is, when searching for any word in a parallel sentence pair, the parallel sentence pair can be retrieved.

4. The method for improving automatic evaluation of machine translation quality by using retrieval according to claim 1 or 2, characterized in that: The retrieval stage specifically includes the following steps: Step 11: The database constructs a search engine using Lucene. Lucene uses the FST structure to construct an index, sharing an index path for words with the same prefix. That is, given a bilingual parallel sentence pair, Lucene constructs an inverted index for each word in the source language and the target language in the bilingual parallel sentence pair. Step 12: Assume that the machine translation quality assessment sentence pair is: , translated as: , for the word y to be evaluated in the machine translation quality evaluation sentence pair j , the query sequence is: ； Step 13: For the word to be evaluated, all retrieved parallel sentence pairs are sorted using BM25, and the top-k results are retained. The calculation formula of the BM25 is as follows: ， Where Q is the machine translation quality assessment sentence pair to be evaluated, q i represents each word in the machine translation quality assessment sentence pair, and d represents a bilingual parallel sentence pair retrieved from the database, R(q i ,d) used to calculate q i The frequency of occurrence in d is calculated as follows: ， ； W i To calculate q i The frequency of occurrence in the entire database is calculated as follows: ； In the above formula, k1, k2 and b are predefined coordination factors, and N represents q i The total number of occurrences in the database, f i Indicates q i The number of times qf appears in d i Indicates q i The number of times it appears in Q, dl represents the length of Q, and avg_dl represents the average sentence length in the database.

5. The method for improving automatic evaluation of machine translation quality by using retrieval according to claim 4, characterized in that: The machine translation quality assessment stage specifically includes the following steps: Step 44: Encode the machine translation quality assessment sentence pair through XLMR to obtain: ； For the word y to be evaluated j , extract its The hidden layer representation of the corresponding position in ; In addition, for the word y to be evaluated j The corresponding retrieved bilingual parallel sentence pairs are: ； R1 to R k Encode using XLMR in the same way and obtain the hidden state of the last layer respectively to obtain: ; Step 55: to Stitched together to get ; Step 66: Place the y j The corresponding hidden layer representation For h R By extracting information through MultiHead Attention, we get , ； Step 77: and Fusion is achieved through a gating mechanism ,Will Input into the multi-layer perceptron for classification, and output OK / BAD, that is, the word y to be evaluated j The translation is right or wrong.

6. The method for improving automatic evaluation of machine translation quality by using retrieval according to claim 5, characterized in that: The gating mechanism is: ； described .

7. The method for improving automatic evaluation of machine translation quality by using retrieval according to claim 6, characterized in that: If no sentence pair containing the current evaluation word can be found in the parallel sentence pair, then The value is 0, the gating mechanism model outputs The value is also close to 0, that is, the model does not use the retrieved results; after obtaining the fused representation, the same way as the baseline model is used Input into the multi-layer perceptron for classification.

8. An electronic device for automatically evaluating the quality of machine translation using retrieval, characterized in that: The electronic device (50) includes a processor (51) and a memory (52): The memory (52) is used to store program code and transmit the program code to the processor (51); The processor (51) is configured to execute the method according to any one of claims 1 to 7 according to instructions in the program code.

9. A computer-readable storage medium, characterized in that The computer-readable storage medium is used to store program code, and the program code is used to execute the method for improving automatic evaluation of machine translation quality by using retrieval as described in any one of claims 1 to 7.

Images