A machine translation model performance detection method and related equipment

Abstract

The embodiment of the present application discloses a method for detecting the performance of a machine translation model and related equipment. The embodiment of the present application obtains a source text of a first language type and a target text of a second language type corresponding to the source text; replaces the second target words in the source text with multiple first target words of the first language type to obtain multiple candidate texts; translates the multiple candidate texts through a machine translation model to obtain a first translation text of the second language type corresponding to each candidate text; translates the source text through a machine translation model to obtain a second translation text of the second language type; detects the translation performance index of the machine translation model according to the first similarity between the first translation text and the target text and the second similarity between the second translation text and the target text. The accuracy of detecting the performance of the machine translation model is improved.

Description

Technical Field

[0001] The present application relates to the technical field of natural language processing, and in particular to a method for detecting the performance of a machine translation model, and related equipment. Background Art

[0002] Machine translation technology refers to the technology of using computer equipment to translate the original text in one natural language (generally called the source language or source end) into another natural language (generally called the target language or target end). Since machine translation can be completed autonomously by computer equipment, it can handle a large amount of translation work in a relatively short time compared to manual translation.

[0003] In the prior art, machine translation can be achieved through a machine translation model, for example, a statistical machine translation model (SMT). The machine translation model generally relies on a word or phrase alignment table to translate the source language into the target language, and when judging the performance of the machine translation model, a global comparison is performed between the target language translated by the machine translation model and the standard language corresponding to the source language. However, the architecture, number of parameters, and volume of training data of the machine translation model are becoming increasingly complex and large-scale, making it impossible to fully understand the source of the performance bottleneck through the internal structure of the machine translation model, and it is also impossible to effectively control the acceptance of the prediction of the machine translation model relative to the user after any input is given. Therefore, the global comparison method makes the accuracy of the judgment of the performance of the machine translation model low. Summary of the invention

[0004] The embodiments of the present application provide a method for detecting the performance of a machine translation model, and related equipment, wherein the related equipment may include a machine translation model performance detection device, a computer device, and a storage medium, etc., which can improve the accuracy of the machine translation model performance detection.

[0005] To solve the above technical problems, the present application provides the following technical solutions:

[0006] The present application embodiment provides a method for detecting performance of a machine translation model, including:

[0007] Acquire a source text in a first language type, and acquire a target text in a second language type corresponding to the source text;

[0008] Replacing the second target words in the source text with the plurality of first target words of the first language type respectively to obtain a plurality of candidate texts;

[0009] Translate the multiple candidate texts through a machine translation model to obtain a first translated text of the second language type corresponding to each candidate text;

[0010] Translating the source text by using the machine translation model to obtain a second translated text in the second language type;

[0011] The translation performance index of the machine translation model is detected according to a first similarity between the first translated text and the target text and a second similarity between the second translated text and the target text.

[0012] According to one aspect of the present application, a machine translation model performance detection device is also provided, comprising:

[0013] An acquisition unit, configured to acquire a source text in a first language type, and acquire a target text in a second language type corresponding to the source text;

[0014] A replacement unit, configured to replace second target words in the source text with a plurality of first target words of the first language type respectively, to obtain a plurality of candidate texts;

[0015] A first translation unit, configured to translate the plurality of candidate texts by using a machine translation model to obtain a first translation text of the second language type corresponding to each candidate text;

[0016] A second translation unit, configured to translate the source text by using the machine translation model to obtain a second translated text in the second language type;

[0017] A detection unit is used to detect a translation performance indicator of the machine translation model according to a first similarity between the first translated text and the target text and a second similarity between the second translated text and the target text.

[0018] According to one aspect of the present application, a computer device is also provided, including a processor and a memory, wherein a computer program is stored in the memory, and when the processor calls the computer program in the memory, any one of the machine translation model performance detection methods provided in the embodiments of the present application is executed.

[0019] According to one aspect of the present application, a storage medium is also provided, wherein the storage medium is used to store a computer program, and the computer program is loaded by a processor to execute any one of the machine translation model performance detection methods provided in the embodiments of the present application.

[0020] The embodiment of the present application can obtain a source text of a first language type, and obtain a target text of a second language type corresponding to the source text, and use multiple first target words of the first language type to replace the second target words in the source text respectively, to obtain multiple candidate texts; then the multiple candidate texts can be translated by a machine translation model to obtain a first translation text of the second language type corresponding to each candidate text, and the source text is translated by a machine translation model to obtain a second translation text of the second language type; at this time, the translation performance index of the machine translation model can be detected according to the first similarity between the first translation text and the target text, and the second similarity between the second translation text and the target text. This scheme translates the candidate text and the source text obtained after word replacement through a machine translation model, and performs statistical analysis based on the first similarity between the first translation text and the target text, and the second similarity between the second translation text and the target text, so as to accurately detect the translation performance index of the machine translation model, thereby improving the accuracy of the performance detection of the machine translation model. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For those skilled in the art, other drawings can be obtained based on these drawings without creative work.

[0022] Figure 1 It is a schematic diagram of a scenario in which the method for detecting the performance of a machine translation model provided in an embodiment of the present application is applied;

[0023] Figure 2 It is a flowchart of a method for detecting performance of a machine translation model provided in an embodiment of the present application;

[0024] Figure 3 It is a schematic diagram of the impact of different replacement strategies provided in the embodiments of the present application on the translation performance indicators of the machine translation model;

[0025] Figure 4 is another schematic diagram of the impact of different replacement strategies provided in the embodiments of the present application on the translation performance indicators of the machine translation model;

[0026] Figure 5 It is a flowchart of a method for detecting performance of a machine translation model provided in an embodiment of the present application;

[0027] Figure 6 It is a schematic diagram of the similarity distribution between the translation text obtained by replacing the second target word in the source text and the target text provided by the embodiment of the present application;

[0028] Figure 7is another schematic diagram of the distribution of similarities between a translation text obtained by replacing a second target word in a source text with a candidate text and a target text provided by an embodiment of the present application;

[0029] Figure 8 is another schematic diagram of the distribution of similarities between a translation text obtained by replacing a second target word in a source text with a candidate text and a target text provided by an embodiment of the present application;

[0030] Fig. 9 is another schematic diagram of the distribution of similarities between a translation text obtained by replacing a second target word in a source text with a candidate text and a target text provided by an embodiment of the present application;

[0031] Fig.10 is another schematic diagram of the distribution of similarities between a translation text obtained by replacing a second target word in a source text with a candidate text and a target text provided by an embodiment of the present application;

[0032] Fig.11 is another schematic diagram of the distribution of similarities between a translation text obtained by replacing a second target word in a source text with a candidate text and a target text provided by an embodiment of the present application;

[0033] Fig.12 is a schematic diagram of a machine translation model performance detection device provided in an embodiment of the present application;

[0034] Fig.13 It is a schematic diagram of the structure of a computer device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0035] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of this application.

[0036] The embodiments of the present application provide a method for detecting the performance of a machine translation model, and related equipment, wherein the related equipment may include a machine translation model performance detection device, a computer device, a storage medium, and the like.

[0037] See also Figure 1 , Figure 1Schematic diagram of the scenario of the application of the machine translation model performance detection method provided by the embodiment of the present application, the machine translation model performance detection method application may include a machine translation model performance detection device, and the machine translation model performance detection device may be specifically integrated in a computer device such as a server or a terminal, wherein the server may be an independent physical server, or a server cluster or a distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content distribution networks (Content Delivery Network, CDN), and big data and artificial intelligence platforms, but is not limited thereto. The terminal may be a mobile phone, a tablet computer, a laptop computer, a desktop computer, a vehicle-mounted device, or a wearable device, etc. The server and the terminal may be directly or indirectly connected by wired or wireless communication, and this application is not limited thereto.

[0038] Among them, the computer device can be used to obtain a source text of a first language type, and to obtain a target text of a second language type corresponding to the source text, and to use multiple first target words of the first language type to replace the second target words in the source text respectively to obtain multiple candidate texts; then the multiple candidate texts can be translated through a machine translation model to obtain a first translated text of the second language type corresponding to each candidate text, and the source text can be translated through the machine translation model to obtain a second translated text of the second language type; at this time, the translation performance index of the machine translation model can be detected based on the first similarity between the first translated text and the target text, and the second similarity between the second translated text and the target text, thereby improving the accuracy of the machine translation model performance detection.

[0039] It should be noted that Figure 1 The scenario diagram of the application of the machine translation model performance detection method shown is only an example. The application and scenario of the machine translation model performance detection method described in the embodiment of the present application are for more clearly illustrating the technical solution of the embodiment of the present application, and do not constitute a limitation on the technical solution provided by the embodiment of the present application. Ordinary technicians in this field can know that with the evolution of the application of the machine translation model performance detection method and the emergence of new business scenarios, the technical solution provided by the embodiment of the present application is also applicable to similar technical problems.

[0040] It should be noted that the description order of the following embodiments is not intended to limit the preferred order of the embodiments.

[0041] The machine translation model performance detection method provided in the embodiment of the present application may involve technologies such as machine learning technology in artificial intelligence. For example, text can be translated by a machine translation model that includes machine learning technology in artificial intelligence. The artificial intelligence technology and machine learning technology are first explained below.

[0042] Artificial intelligence is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results. In other words, artificial intelligence is a comprehensive technology in computer science that attempts to understand the essence of intelligence and produce a new intelligent machine that can respond in a similar way to human intelligence. Artificial intelligence is to study the design principles and implementation methods of various intelligent machines so that machines have the functions of perception, reasoning and decision-making.

[0043] Artificial intelligence technology is a comprehensive discipline that covers a wide range of fields, including both hardware-level and software-level technologies. The basic technologies of artificial intelligence generally include sensors, dedicated artificial intelligence chips, cloud computing, distributed storage, big data processing technology, operation / interaction systems, and mechatronics. Artificial intelligence software technology mainly includes computer vision technology, speech processing technology, natural language processing technology, and machine learning / deep learning.

[0044] Machine Learning (ML) is a multi-disciplinary subject that involves probability theory, statistics, approximation theory, convex analysis, and algorithmic complexity theory. It specializes in studying how computers simulate or implement human learning behavior to acquire new knowledge or skills and reorganize existing knowledge structures to continuously improve their performance. Machine learning is the core of artificial intelligence and the fundamental way to make computers intelligent. Its applications are spread across all areas of artificial intelligence. Machine learning and deep learning usually include artificial neural networks, belief networks, reinforcement learning, transfer learning, inductive learning, and self-learning.

[0045] In this embodiment, the description will be made from the perspective of a machine translation model performance detection device, which can be specifically integrated into a computer device such as a server.

[0046] See also Figure 2 , Figure 2 : is a flow chart of a method for detecting performance of a machine translation model provided in an embodiment of the present application. The method for detecting performance of a machine translation model may include:

[0047] S101: Acquire a source text in a first language type, and acquire a target text in a second language type corresponding to the source text.

[0048] Among them, the first language type and the second language type can be flexibly set according to actual needs. For example, the first language type can be Chinese, the source text can be Chinese text, the second language type can be English, and the target text can be English text; or, the first language type can be English, the source text can be English text, the second language type can be Chinese, and the target text can be Chinese text; or, the first language type can be Chinese, the source text can be Chinese text, the second language type can be Japanese, and the target text can be Japanese text; or, the first language type can be Chinese, the source text can be Chinese text, the second language type can be German, and the target text can be German text; or, the first language type can be Chinese, the source text can be Chinese text, the second language type can be French, and the target text can be French text; and so on.

[0049] The length, number and specific content of the source text and the target text can be flexibly set according to actual needs and are not specifically limited here. For example, the source text and the target text can be a sentence, a paragraph or an article, etc. The source text can include multiple ones, and each source text can correspond to one target text.

[0050] The source text of the first language type may be obtained from a local database of the computer device, or the source text of the first language type may be downloaded from a server, or the source text of the first language type input by a user may be received, or the voice information entered by the user may be collected and converted into the source text of the first language type, etc. Also, the target text of the second language type corresponding to the source text may be obtained through a trained translation model, or the target text of the second language type corresponding to the source text translated by the user may be received, or the target text of the second language type matching the source text may be obtained from a database pre-storing the text of the second language type, etc. Of course, the method of obtaining the source text and the target text may also be flexibly set according to actual needs, and is not specifically limited here.

[0051] S102: Use a plurality of first target words of the first language type to replace second target words in the source text respectively, to obtain a plurality of candidate texts.

[0052] Among them, the number and specific content of the first target word can be flexibly set according to actual needs. For example, the first target word can be a word, a word composed of two words, or a phrase corresponding to the first language type, and multiple words can be selected from the word list of the first language type as the first target word, and the word list stores multiple different words. After obtaining the source text, the source text can be divided into words to obtain n words constituting the source text X: X = [x1, x2, x3, ..., xn], and then the second target word (i.e., the replaced word) can be selected from the n words. At this time, multiple first target words can be replaced with the second target word in the source text respectively to obtain multiple candidate texts. For example, if the first target word includes "you", "I", "he", and "we", and the second target word is x3, then the second target word x3 in the source text can be replaced with "you" to obtain candidate text 1, the second target word x3 in the source text can be replaced with "I" to obtain candidate text 2, the second target word x3 in the source text can be replaced with "he" to obtain candidate text 3, and the second target word x3 in the source text can be replaced with "we" to obtain candidate text 4.

[0053] In one embodiment, using multiple first target words of the first language type to replace the second target words in the source text respectively to obtain multiple candidate texts may include: based on the source text and the target text, calculating the gradient value corresponding to the second target word in the source text through the loss function of the machine translation model; sampling multiple first target words from the vocabulary of the first language type according to the gradient value; and replacing the second target words in the source text with the multiple first target words respectively to obtain multiple candidate texts.

[0054] The type and structure of the machine translation model can be flexibly set according to actual needs. For example, the machine translation model can be a neural machine translation model (Neural Machine Translation, NMT) or a statistical machine translation model (Statistical Machine Translation, SMT).

[0055] In order to improve the reliability of replacement, the second target word in the source text can be replaced by using a gradient sampling replacement strategy. Specifically, the gradient value corresponding to the second target word in the source text can be calculated based on the source text and the target text by using the loss function L(X, Y; theta) of the machine translation model. For example, the gradient value of the loss function on the word vector of the second target word can be calculated. Where X can represent the source text, Y can represent the target text, and theta can represent the parameters of the machine translation model.

[0056] Then, a plurality of first target words may be sampled from a vocabulary of the first language type according to the gradient value. In one embodiment, sampling a plurality of first target words from a vocabulary of the first language type according to the gradient value may include: updating a word vector of a second target word in a source text according to a preset learning rate according to the gradient value to obtain an updated word vector; performing a dot product operation on the updated word vector and a word vector matrix corresponding to the vocabulary of the first language type to obtain a similarity vector between the word vector of the second target word in the source text and each word vector in the word vector matrix; and sampling a plurality of first target words having a similarity with the second target word in the source text greater than a preset similarity threshold from the vocabulary of the first language type according to the similarity vector.

[0057] For example, in order to improve the reliability of screening the first target word, the word vector of the second target word in the source text can be updated according to the gradient value at a preset learning rate to obtain an updated word vector. For example, the updated word vector embedding = word vector of the second target word - preset learning rate * gradient value, where the specific value of the preset learning rate can be flexibly set according to actual needs. For example, the preset learning rate can be 1.0.

[0058] The updated word vector can be subjected to a dot product operation with the word vector matrix corresponding to the vocabulary of the first language type to obtain a similarity vector between the word vector of the second target word in the source text and each word vector in the word vector matrix. The similarity vector can be a vector of the vocabulary size dimension, and the word vector matrix can be a vector matrix composed of the word vectors of each word in the vocabulary. At this time, multiple first target words whose similarity to the second target word in the source text is greater than a preset similarity threshold can be sampled (e.g., without replacement) from the vocabulary of the first language type according to the similarity vector. For example, a word with a larger similarity can be sampled from the vocabulary using a multinomial distribution as the first target word. After obtaining the first target word, multiple first target words can be respectively replaced with the second target word in the source text to obtain multiple candidate texts.

[0059] In one embodiment, using multiple first target words of the first language type to replace the second target words in the source text respectively to obtain multiple candidate texts may include: selecting multiple candidate words from the vocabulary of the first language type; replacing the second target words in the source text with the multiple candidate words respectively to obtain multiple replacement texts; calculating the loss value between the replacement text and the target text through the loss function of the machine translation model; selecting a preset number of candidate words corresponding to the minimum loss value to obtain multiple first target words; replacing the second target words in the source text with the multiple first target words respectively to obtain multiple candidate texts.

[0060] In order to improve the accuracy of replacement, the second target word in the source text can be replaced by a hierarchical sampling replacement strategy. Specifically, multiple candidate words can be randomly selected from the word list of the first language type or according to a preset screening strategy, and the multiple candidate words are respectively replaced with the second target word in the source text to obtain multiple replacement texts. For example, if the candidate words include word 1, word 2 and word 3, and the second target word is xi, then word 1 can be used to replace the second target word xi in the source text to obtain replacement text 1, word 2 can be used to replace the second target word xi in the source text to obtain replacement text 2, and word 3 can be used to replace the second target word xi in the source text to obtain replacement text 3. Then, the loss value between the replacement text and the target text can be calculated by the loss function L(X',Y;theta) of the machine translation model, where X' can represent the replacement text, Y can represent the target text, and theta can represent the parameters of the machine translation model. At this time, a preset number of replacement texts corresponding to the minimum loss value can be selected, and the candidate words corresponding to the preset number of replacement texts are used as multiple first target words, and the multiple first target words are respectively replaced with the second target words in the source text to obtain multiple candidate texts.

[0061] In one embodiment, using multiple first target words of the first language type to replace second target words in the source text respectively to obtain multiple candidate texts may include: determining a replacement position in the source text; randomly selecting multiple first target words from a word list of the first language type; and replacing the multiple first target words with second target words corresponding to the replacement positions in the source text respectively to obtain multiple candidate texts.

[0062] In order to improve the convenience of replacement, a random sampling replacement strategy can be used to replace the second target word in the source text. Specifically, the source text can be divided into positions to obtain multiple replacement positions, and the replacement positions to be replaced can be screened out from the multiple replacement positions. Multiple first target words are randomly screened out from the vocabulary of the first language type, and the multiple first target words are respectively replaced with the second target words corresponding to the replacement positions in the source text to obtain multiple candidate texts.

[0063] In one embodiment, using multiple first target words of the first language type to replace the second target words in the source text respectively to obtain multiple candidate texts may include: determining attribute information of the second target word in the source text; screening multiple first target words from a word list of the first language type according to the attribute information; and replacing the second target words in the source text with the multiple first target words respectively to obtain multiple candidate texts.

[0064] In order to improve the flexibility of replacement, the second target word in the source text can be replaced by an attribute sampling replacement strategy, wherein the attribute information may include the part of speech or semantic information of the second target word, etc. Specifically, the source text can be divided into words to obtain multiple words, and the second target word can be randomly or according to a preset screening strategy from the multiple words to determine the attribute information of the second target word. For example, the part of speech of the second target word can be analyzed or the second target word can be semantically analyzed to obtain the attribute information of the second target word. Then, according to the attribute information of the second target word, multiple first target words (such as first target words with the same part of speech or similar semantic information) matching the attribute information of the second target word can be screened from the vocabulary of the first language type, and the multiple first target words can be respectively replaced with the second target word in the source text to obtain multiple candidate texts.

[0065] S103: Translate multiple candidate texts using a machine translation model to obtain a first translated text in the second language type corresponding to each candidate text.

[0066] After obtaining multiple candidate texts, each candidate text can be translated separately through the machine translation model to obtain the first translation text of the second language type corresponding to each candidate text, and the first similarity between the first translation text and the target text can be calculated (for example, sentence BLEU: a sub-level machine translation evaluation standard, the higher the value, the better the effect). For example, the first translation text and the target text can be compared word by word to detect the same parts between the first translation text and the target text, and the percentage of the same parts is calculated as the first similarity. For another example, the first translation text and the target text can be compared word by word and semantically, and the first similarity between the first translation text and the target text is calculated based on the comparison results. The higher the first similarity, the more similar the first translation text is to the target text, and conversely, the lower the first similarity, the less similar the first translation text is to the target text.

[0067] S104: Translate the source text using a machine translation model to obtain a second translated text in a second language.

[0068] After obtaining the source text, the source text can be translated by a machine translation model to obtain a second translation text of the second language type corresponding to the source text, and the second similarity between the second translation text and the target text can be calculated. For example, the second translation text can be compared word by word with the target text to detect the same parts between the second translation text and the target text, and the percentage of the same parts can be calculated as the second similarity. For another example, the second translation text can be compared word by word and semantically with the target text, and the second similarity between the second translation text and the target text can be calculated based on the comparison results; and so on.

[0069] In one embodiment, obtaining a first similarity between a first translated text and a target text, and a second similarity between a second translated text and the target text may include: obtaining a first initial similarity between the first translated text and the target text, and a second initial similarity between the second translated text and the target text through a machine translation model; calculating a first translation score of the candidate text and a second translation score of the source text using a reranking algorithm; adjusting the first initial similarity according to the first translation score to obtain a first similarity between the first translated text and the target text, and adjusting the second initial similarity according to the second translation score to obtain a second similarity between the second translated text and the target text.

[0070] Among them, the reordering algorithm can be used to calculate the translation score of the text. For example, for source text A, in the case of unknown target text corresponding to source text A, the translation score of source text A that can be translated into target text can be calculated. The higher the translation score, the closer the translation result of source text A is to the target text. Conversely, the lower the translation score, the less close the translation result of source text A is to the target text. In order to improve the accuracy of similarity calculation, the similarity obtained by the machine translation model can be fine-tuned based on the translation score calculated by the reordering algorithm. For example, the first initial similarity between the first translated text and the target text and the second initial similarity between the second translated text and the target text can be obtained by the machine translation model, and the first translation score of the candidate text can be calculated by the reordering algorithm, and the second translation score of the source text can be calculated by the reordering algorithm. Then, the first initial similarity can be adjusted according to the first translation score to obtain the first similarity between the first translated text and the target text, and the second initial similarity can be adjusted according to the second translation score to obtain the second similarity between the second translated text and the target text. For example, the first translation score can be used as the first similarity between the first translated text and the target text, and the second translation score can be used as the second similarity between the second translated text and the target text. Alternatively, the first translation score and the first initial similarity can be taken as the first similarity between the first translated text and the target text, and the second translation score and the second initial similarity can be taken as the second similarity between the second translated text and the target text, and so on.

[0071] S105 . Detect a translation performance index of the machine translation model according to a first similarity between the first translated text and the target text and a second similarity between the second translated text and the target text.

[0072] Among them, the translation performance indicators of the machine translation model may include translation accuracy and stability, etc. After obtaining the first similarity between the first translated text and the target text, and the second similarity between the second translated text and the target text, the translation performance indicators of the machine translation model can be detected according to the first similarity and the second similarity.

[0073] For example, the source text and the candidate text can be used as test samples to obtain the generalization barrier words of each test sample in the machine translation model. The generalization barrier words can be words that affect the accurate translation of the machine translation model. The word's part of speech or semantics can be analyzed to illustrate that the word of this type of part of speech or semantics affects the accurate translation of the machine translation model, indicating whether the word hinders the successful generalization of the machine translation model on the test sample. The generalization barrier word detection algorithm can be used to detect a set of words in the sample-level input that causes the model to generalize poorly on the sample.

[0074] It should be noted that the detection of the translation performance index of the machine translation model in this embodiment can be called fine-grained analysis. The fine-grained performance can be, after a given machine translation model input, how to judge the specific performance of the machine translation model on the input. This instance-level analysis can be called fine-grained analysis. Fine-grained analysis can convey richer error information, which is conducive to more effective understanding of the behavior of the machine translation model, understanding the capabilities and incapabilities of the machine translation model, and thus more targeted and effective solutions.

[0075] It should be noted that in addition to detection in the presence of target text, it is also possible to use a scenario without target text as an alternative, for example, using a quality assessment model as a substitute for standard loss to detect the translation performance indicators of the machine translation model based on the standard loss.

[0076] In one embodiment, based on a first similarity between a first translated text and a target text, and a second similarity between a second translated text and a target text, detecting a translation performance indicator of a machine translation model may include: obtaining a first similarity between the first translated text and the target text, and a second similarity between the second translated text and the target text; screening out candidate texts having a first similarity greater than a second similarity from the first translated texts corresponding to a plurality of candidate texts to obtain a target candidate text; analyzing the part-of-speech distribution of a second target word in a source text based on the target candidate text; and determining a translation performance indicator of the machine translation model based on the part-of-speech distribution.

[0077] In order to improve the accuracy of the translation performance index detection of the machine translation model, the translation performance index of the machine translation model can be determined according to the part-of-speech distribution of the second target word in the source text. Specifically, the candidate text with a first similarity greater than the second similarity can be screened out from the first translation text corresponding to multiple candidate texts to obtain the target candidate text; for example, the first similarity between the second translation text Yo corresponding to the source text and the target text Y is b0, and the first similarities between the first translation text corresponding to the b candidate texts and the target text Y are b. At this time, a value greater than b0 is screened out from the b first similarities, and the candidate text corresponding to the first similarity greater than b0 is used as the target candidate text. Then, the source text corresponding to the target candidate text can be obtained, and the part-of-speech of the second target word in the source text corresponding to the target candidate text can be determined. The part-of-speech of the second target word in one or more source texts can form a part-of-speech distribution. At this time, the translation performance index of the machine translation model can be determined according to the part-of-speech distribution. For example, the part-of-speech that affects the accurate translation of the machine translation model can be analyzed to obtain the generalization barrier word of the machine translation model. Among them, the part-of-speech can include nouns, verbs, adjectives, pronouns, prepositions, adverbs, conjunctions, etc.

[0078] In one embodiment, analyzing the part-of-speech distribution of the second target word in the source text based on the target candidate text may include: dividing the target candidate text into multiple groups of candidate texts according to the replacement position of the second target word in the source text; calculating the mean of the similarities corresponding to each group of candidate texts; and determining the part-of-speech distribution of the second target word in the source text based on the previous preset group of candidate texts with the largest mean.

[0079] In the process of analyzing the part-of-speech distribution of the second target word in the source text, the target candidate text can be divided into multiple groups of candidate texts according to the replacement position of the second target word in the source text. For example, the candidate text group corresponding to the source text A may include: candidate text group 1 obtained by using the first target word 1, the first target word 2, the first target word 3, the first target word 4, and the first target word 5 to replace the second target word at replacement position a in the source text A, respectively; candidate text group 2 obtained by using the first target word 1, the first target word 6, the first target word 7, and the first target word 8 to replace the second target word at replacement position b in the source text A, respectively; candidate text group 3 obtained by using the first target word 9, the first target word 10, the first target word 11, the first target word 12, and the first target word 13 to replace the second target word at replacement position c in the source text A, and so on. Then, the mean of the similarity corresponding to each group of candidate texts can be calculated. For example, the mean (truncated mean, tm) of the first similarity corresponding to the five candidate texts in candidate text group 1 is calculated, and from multiple candidate text groups corresponding to the same source text, the candidate texts of the first preset group with the largest mean (also called tm value) are screened out (for example, the k% candidate texts with the largest tm value are screened out), and the part-of-speech distribution of the second target word in the source text corresponding to the candidate text of the first preset group with the largest mean is counted. Among them, the specific values ​​of the first preset group and k% can be flexibly set according to actual needs.

[0080] The following example will illustrate that based on the detection of generalized barrier words, the second target word in the source text can be modified to obtain better translation candidates for re-ranking, thereby selecting a better translation prediction. Part-of-speech distribution analysis of generalized barrier words: Based on the above-mentioned generalized barrier word detection algorithm, the tm value corresponding to each second target word in the source text X can be obtained, and the second target words with higher tm values ​​can be obtained by sorting. The second target words with top-k (k=10%, 20% or 30%) are taken as generalized barrier words on the source text X, and the characteristics of the second target words are captured by analyzing the part-of-speech distribution of these second target words. The following Tables 1 and 2 show the part-of-speech distribution of generalized barrier words in the Chinese-English direction and the English-Chinese direction (i.e., the part-of-speech distribution of the second target words). As a reference, the natural distribution of the part-of-speech of all words in the source text (i.e., the part-of-speech distribution of the words contained in the source text) is given. By comparing the changes (increases and decreases) in the proportion of barrier words compared to the proportion of natural distribution words, it is possible to determine whether a word of a certain part of speech is more likely to be a generalized barrier word or not.

[0081] The conclusions that can be drawn from observing Table 1 and Table 2 include: 1. Generalization barrier words are widely present in all parts of speech categories and are close to the natural distribution of parts of speech; 2. Prepositions (Prep.) and punctuation marks (Punc.) are more likely to become generalization barrier words; 3. Words that express content or meaning are less likely to become generalization barrier words.

[0082] The value of the above conclusions lies in: 1. It reflects the sensitivity of the machine translation model to the context, that is, the more context-related the words are, the more they affect the translation of other words; 2. It can be used as a feature to train the detection classifier of generalization barrier words; 3. It can guide system designers to pay more attention to the influence of certain category words on the semantics of the whole sentence of the source text (also called the source end), thereby providing a potential method to block contextual semantic interference.

[0083] Table 1: Part-of-speech distribution of generalization barrier words in Chinese in the Chinese-English translation direction

[0084] Part of speech k＝10％ k＝20％ k＝30％ Natural distribution Byte Pair Encoding BPE 9.80％- 10.74％- 11.26％- 12.00％ Noun 22.17％- 22.43％- 21.85％- 24.07％ Pronoun 1.94％- 2.18％+ 2.25％+ 2.15％ Verb. 11.57％+ 11.28％+ 11.00％- 11.26％ Adjective 6.74％- 7.19％- 7.26％- 8.19％ Adv. 3.24％+ 3.07％+ 2.83％- 2.93％ PrepositionPrep. 12.94％+ 13.05％+ 13.39％+ 11.88％ Punctuation Punc. 16.04％+ 13.98％+ 13.30％+ 10.41％ DeterminerDet. 8.11％- 8.84％- 9.42％+ 9.05％ Conjunction C&C 1.94％- 2.06％- 2.05％- 2.20％

[0085] Table 2: Part-of-speech distribution of generalization barrier words in English in the English-Chinese translation direction

[0086] Part of speech k＝10％ k＝20％ k＝30％ Natural distribution Byte Pair Encoding BPE 14.32％- 15.10％- 15.28％- 15.33％ Noun 16.52％- 16.23％- 15.83％- 17.63％ Proprietary Name Prop.N. 6.56％- 6.75％- 6.37％- 7.44％ Pronoun 1.75％- 1.91％- 2.32％- 2.35％ Verb. 18.37％+ 18.33％- 18.56％+ 18.36％ Adjective 2.50％- 2.56％- 2.60％- 3.19％ Adv. 4.30％+ 4.27％+ 4.14％+ 4.07％ PrepositionPrep. 4.70％+ 4.65％+ 4.58％+ 3.83％ Punctuation Punc. 16.65％+ 14.49％+ 14.40％+ 11.44％ Quantity Q&M 3.95％- 4.49％- 4.59％- 4.87％ Conjunction C&C 1.84％- 1.79％- 1.99％- 2.23％

[0087] Among them, in Table 1 and Table 2, k=10% can represent that from multiple candidate text groups corresponding to the source text, the top 10% candidate text groups with the largest mean (i.e., tm value) are screened out to count the part-of-speech distribution of the second target word in the source text corresponding to the 10% candidate text groups. Natural can represent the part-of-speech distribution of each word in the source text, "-" in "9.80%-" can represent that 9.80% is smaller than the natural distribution of 12.00%, and "+" in "11.57%+" can represent that 11.57%+ is larger than the natural distribution of 11.26%, and the percentage corresponding to each part of speech can represent the proportion of each part of speech in the words in the source text.

[0088] In one embodiment, using a plurality of first target words of the first language type to replace the second target words in the source text respectively to obtain a plurality of candidate texts may include: using a plurality of first target words of the first language type to replace words in the source text respectively according to different replacement strategies to obtain a plurality of candidate texts corresponding to each replacement strategy. According to the first similarity between the first translated text and the target text, and the second similarity between the second translated text and the target text, detecting the translation performance index of the machine translation model may include: selecting candidate texts with a first similarity greater than a second similarity from the first translated texts corresponding to the plurality of candidate texts obtained by replacement based on each replacement strategy, respectively, to obtain the target candidate texts corresponding to each replacement strategy; dividing the target candidate texts corresponding to each replacement strategy into a plurality of groups of candidate texts according to the replacement position of the second target word in the source text, and calculating the mean of the similarity corresponding to each group of candidate texts; sorting the candidate texts corresponding to each replacement strategy in order from high to low in terms of the mean, and analyzing the overlap of the source texts corresponding to each replacement strategy according to the sorting results; and determining the translation performance index of the machine translation model according to the overlap.

[0089] In order to improve the reliability of the detection of the translation performance index of the machine translation model, the translation performance index of the machine translation model can be determined according to the overlap corresponding to different replacement strategies, wherein the replacement strategy can include the above-mentioned gradient sampling replacement strategy, hierarchical sampling replacement strategy, and random sampling replacement strategy. Specifically, according to different replacement strategies such as gradient sampling replacement strategy, hierarchical sampling replacement strategy, and random sampling replacement strategy, multiple first target words of the first language type can be used to replace the words in the source text respectively, and multiple candidate texts corresponding to each replacement strategy can be obtained. In the process of detecting the translation performance index of the machine translation model, the candidate texts with the first similarity greater than the second similarity can be screened out from the first translation texts corresponding to the multiple candidate texts replaced based on each replacement strategy, and the target candidate texts corresponding to each replacement strategy can be obtained; and the target candidate texts corresponding to each replacement strategy are divided into multiple groups of candidate texts according to the replacement position of the second target word in the source text, and the mean of the similarity corresponding to each group of candidate texts is calculated; the candidate texts corresponding to each replacement strategy are sorted in order from high to low according to the mean, and the overlap of the source text corresponding to each replacement strategy is analyzed according to the sorting result, and the translation performance index of the machine translation model is determined according to the overlap.

[0090] The following will give an example, for example, Figure 3 As shown, it reflects the impact of different replacement strategies for generating candidate texts on the accuracy of the machine translation model. It can be seen that under different candidate texts b, as the number of candidate texts b increases, the accuracy increases, and the accuracy corresponding to each replacement strategy is almost the same. Figure 3In , the horizontal axis may represent the number of candidate texts b obtained by replacing the second target word in the source text, and the vertical axis may represent the overlap of the source texts corresponding to different replacement strategies (for example, whether k% of the words are the same).

[0091] Analysis of overlap with other source text word categories: The second target words in the top-k source texts obtained by the generalized barrier word identification algorithm can be compared with some other categories, and the overlap can be shown in Table 3. It can be seen that the overlap between low-frequency words (Frequency), words with high translation entropy (Entropy) and error words (Exception) and barrier words is comparable to the overlap between randomly selected words and barrier words, indicating that there is an essential difference between generalized barrier words and low-frequency words, high translation entropy words and error words. One reason may be that the words corresponding to these three categories are all global, while barrier words are the product of instance-level analysis, and there may be great differences; another reason may be that the generalized barrier words may be translated correctly, but they affect the translation of other words, while most of the low-frequency words, high translation entropy words and error words are words that are not easy to be translated correctly.

[0092] Table 3: The overlap between generalization barrier words and existing other categories of words in English-Chinese and Chinese-English translation

[0093]

[0094] In one embodiment, the source text includes multiple, and based on the first similarity between the first translated text and the target text, and the second similarity between the second translated text and the target text, detecting the translation performance index of the machine translation model may include: selecting candidate texts with a first similarity greater than a second similarity from the first translated texts corresponding to the multiple candidate texts corresponding to each source text, respectively, to obtain a target candidate text corresponding to each source text; dividing the target candidate text corresponding to each source text into multiple groups of candidate texts according to the replacement position of the second target word in the source text, and calculating the mean of the similarity corresponding to each group of candidate texts; calculating the variance value of the similarity corresponding to the candidate text in the candidate text group with the largest mean; and determining the translation performance index of the machine translation model based on the variance value.

[0095] In order to accurately detect the stability of the translation performance of the machine translation model, the translation performance index of the machine translation model can be determined according to the variance value of the similarity corresponding to the candidate texts, for example, Figure 4 As shown, in terms of stability, the variance of the hierarchical sampling replacement strategy (which can be simply referred to as hierarchical sampling) is the smallest, and the variances of gradient-based sampling and random sampling are similar. Therefore, hierarchical sampling can be preferably used to calculate the tm value. Figure 4In the figure, the horizontal axis can represent the number of candidate texts b obtained by replacing the second target word in the source text, and the vertical axis can represent the sorting correlation, that is, whether the rearrangement between the two k% candidate texts is related, which is reflected by the variance. The smaller the variance, the higher the stability.

[0096] In one embodiment, after detecting the translation performance index of the machine translation model based on the first similarity between the first translated text and the target text and the second similarity between the second translated text and the target text, the machine translation model performance detection method may also include: adjusting the machine translation model according to the translation performance index of the machine translation model to obtain an adjusted machine translation model; and translating the text to be translated using the adjusted machine translation model.

[0097] After the translation performance index of the machine translation model is detected, the parameters of the machine translation model can be adjusted according to the translation performance index of the machine translation model to obtain an adjusted machine translation model. When translating texts subsequently, the adjusted machine translation model can be used to accurately translate the texts to be translated.

[0098] It should be noted that the importance of unsupervised detection of generalized barrier words and the feasibility of modifying the second target word in the source text to generate re-ranking candidates are reflected by analyzing the gain of re-ranking related indicators brought by modifying the second target word in the source text. In order to be close to the real re-ranking task, each second target word in the source text can be modified to generate a translation file corresponding to the candidate text (which can be called a translation candidate), rather than modifying the candidate text when the target text is known. The candidate text is measured by the similarity value (the larger the better), difference (the smaller the better) and recall rate of the target text (the larger the better) to compare it with the top-m candidates generated by the standard beam search. It can be found that the candidates obtained by modifying the barrier word are better in all indicators, that is, the possibility of obtaining better translation candidates by modifying the second target word in the source text (also called modifying the source end word) is verified.

[0099] Table 4: Comparison of translation candidates obtained by modifying the source end words and beam search in the English-Chinese and Chinese-English translation directions

[0100]

[0101] The embodiment of the present application can obtain a source text of a first language type, and obtain a target text of a second language type corresponding to the source text, and use multiple first target words of the first language type to replace the second target words in the source text respectively, to obtain multiple candidate texts; then the multiple candidate texts can be translated by a machine translation model to obtain a first translation text of the second language type corresponding to each candidate text, and the source text is translated by a machine translation model to obtain a second translation text of the second language type; at this time, the translation performance index of the machine translation model can be detected according to the first similarity between the first translation text and the target text, and the second similarity between the second translation text and the target text. This scheme translates the candidate text and the source text obtained after word replacement through a machine translation model, and performs statistical analysis based on the first similarity between the first translation text and the target text, and the second similarity between the second translation text and the target text, so as to accurately detect the translation performance index of the machine translation model, thereby improving the accuracy of the performance detection of the machine translation model.

[0102] The method described in the above embodiment is further described in detail below with examples.

[0103] This embodiment takes the machine translation model performance detection device integrated in a computer device as an example, and takes the source text in Chinese and the target text in English as an example. Figure 5 , Figure 5 A flow chart of a method for detecting the performance of a machine translation model provided in an embodiment of the present application. The method flow may include:

[0104] S201: Acquire a source text in Chinese, and acquire an English target text corresponding to the source text.

[0105] For example, the computer device can obtain the source text from a local database, or can receive the source text input by the user, or can collect the voice information input by the user and convert the voice information into the source text in Chinese, etc. Also, the computer device can obtain the target text corresponding to the source text through the trained translation model, or can receive the target text translated by the user, etc.

[0106] S202: Using a plurality of first target words in Chinese to replace the second target words in the source text respectively according to the replacement strategy to obtain a plurality of candidate texts.

[0107] Among them, the replacement strategies may include gradient sampling replacement strategies, hierarchical sampling replacement strategies, and random sampling replacement strategies, etc. The computer device may use multiple first target words in Chinese to replace the second target words in the source text according to one or more replacement strategies to obtain multiple candidate texts corresponding to each replacement strategy.

[0108] For example, replacing the second target word in the source text using a gradient sampling replacement strategy may include: based on the source text and the target text, calculating the gradient value corresponding to the second target word in the source text through the loss function of the machine translation model, and updating the word vector of the second target word in the source text according to the gradient value according to a preset learning rate to obtain an updated word vector; performing a dot product operation on the updated word vector and the word vector matrix corresponding to the Chinese vocabulary to obtain a similarity vector between the word vector of the second target word in the source text and each word vector in the word vector matrix; based on the similarity vector, sampling multiple first target words from the Chinese vocabulary whose similarity to the second target word in the source text is greater than a preset similarity threshold, and replacing the multiple first target words with the second target word in the source text to obtain multiple candidate texts.

[0109] For another example, using a hierarchical sampling replacement strategy to replace a second target word in a source text may include: selecting multiple candidate words from a Chinese vocabulary, replacing the second target word in the source text with the multiple candidate words respectively, to obtain multiple replacement texts; calculating the loss value between the replacement text and the target text through the loss function of the machine translation model, selecting a preset number of candidate words corresponding to the minimum loss value, obtaining multiple first target words, and replacing the second target word in the source text with the multiple first target words respectively, to obtain multiple candidate texts.

[0110] For another example, replacing the second target word in the source text using a random sampling replacement strategy may include: determining a replacement position in the source text, randomly selecting a plurality of first target words from a vocabulary of the first language type, and replacing the plurality of first target words with the second target words corresponding to the replacement position in the source text, respectively, to obtain a plurality of candidate texts.

[0111] S203: Translate multiple candidate texts using a machine translation model to obtain a first English translation text corresponding to each candidate text.

[0112] S204: Translate the source text using a machine translation model to obtain a second English translation text corresponding to the source text.

[0113] S205: Obtain a first similarity between the first translated text and the target text, and a second similarity between the second translated text and the target text.

[0114] After obtaining multiple candidate texts, each candidate text can be translated by a machine translation model to obtain a first English translation text corresponding to each candidate text, and a first similarity between the first translation text and the target text can be calculated. After obtaining the source text, the source text can be translated by a machine translation model to obtain a second English translation text corresponding to the source text, and a second similarity between the second translation text and the target text can be calculated.

[0115] S206: Filter out candidate texts whose first similarity is greater than the second similarity from the first translation texts corresponding to the plurality of candidate texts to obtain a target candidate text.

[0116] S207: Analyze the part-of-speech distribution of the second target word in the source text according to the target candidate text.

[0117] For example, the target candidate text can be divided into multiple groups of candidate texts according to the replacement position of the second target word in the source text, the mean similarity corresponding to each group of candidate texts is calculated, and the part-of-speech distribution of the second target word in the source text is determined based on the first preset group of candidate texts with the largest mean.

[0118] S208. Determine a translation performance index of the machine translation model according to the part-of-speech distribution.

[0119] For example, the generalization barrier words that affect the accurate translation performance of the machine translation model can be determined based on the part-of-speech distribution. The generalization barrier word detection can use similar counterfactual samples, that is, fine-tuning the input (modifying only one word) and observing the average performance improvement of the sample (i.e., the source text) brought by a large number of modified samples (i.e., candidate texts), so as to determine the impact of the modified word on the generalization performance of the machine translation model on the sample. The strong interaction of each word in the source text (i.e., contextual correlation) can be fully considered, and the unique barrier words existing in a specific input sample can be obtained for the sample, which is more in line with the inherent logic of instance-level analysis. That is, the sample-level generalization barrier words are detected when the target text is known, and the average quality of the better translation output generated by the counterfactual is used to measure the risk of the modified part of the original input source text becoming a generalization barrier. Since the counterfactual requires a word in the source text to be modified by a certain number of times, the proposed approximate sampling method greatly reduces the complexity, and makes full use of the deterministic reduction detection algorithm in hierarchical sampling to estimate the variance of the risk. This detection algorithm can be used to analyze the defects of machine translation models, and specifically consider which types of words to model. It can also count the differences in the detection of generalization barrier words in the same input by two different machine translation models.

[0120] S209: Adjust the machine translation model according to the translation performance index of the machine translation model to obtain an adjusted machine translation model, and translate the text to be translated by using the adjusted machine translation model.

[0121] After the translation performance index of the machine translation model is detected, the parameters of the machine translation model can be adjusted according to the translation performance index of the machine translation model to obtain an adjusted machine translation model. When translating texts subsequently, the adjusted machine translation model can be used to accurately translate the texts to be translated.

[0122] The following is an example. For example, the Chinese source text Source is: The goal of genetic scientists is to provide diagnostic tools to discover defective genes that cause disease and ultimately provide therapies that can prevent these genes from causing disorders.

[0123] The English target text corresponding to the source text is Reference: the goal of geneticists is to provide diagnostic tools to identify defective genes that cause diseases soas to arrive eventually at treatments that can prevent those genes from malfunctioning.

[0124] The translated text obtained by translating the source text through the machine translation model is: the goal of genetic scientists is to provide diagnostic tools to discover the defects of the disease and, in the end, to provide treatments that can prevent the production of these genes.

[0125] The machine translation model is used to translate the candidate text to obtain a better translation text (better translation result after replacement). Much better hypothesis is: the goal of gene scientists is to provide iagnostic tools to detect genes that cause diseases, and eventually provide treatments that can prevent these genes from producing obstacles.

[0126] Multiple first target words in Chinese can be used to replace the second target words in the source text to obtain multiple candidate texts. For example, multiple first target words can be used to replace "diagnosis", "tool", "can", "defect", "finally" or "but" in the source text.

[0127] like Figure 6 As shown, Figure 6To obtain a histogram of the similarity distribution by replacing "diagnosis" in the source text with multiple first target words, where the abscissa can represent the tm value and the ordinate can represent the number of candidate texts. Figure 7 To obtain a histogram of the similarity distribution by replacing "tool" in the source text with multiple first target words. Figure 8 To obtain a histogram of the similarity distribution by replacing "can" in the source text with multiple first target words. Fig. 9 To obtain a histogram of the similarity distribution by replacing "defect" in the source text with multiple first target words. Fig.10 To obtain a histogram of the similarity distribution by replacing "end" in the source text with multiple first target words. Fig.11 To obtain a histogram of the similarity distribution by replacing "and" in the source text with multiple first target words. It can be seen that for the histogram of the sentence BLEU distribution of the similarity of the generated counterfactual examples, after changing the second target word "defect" in the source text to "according to", the machine translation model can produce better prediction results. In other words, the existence of the word "defect" leads to the translation of "pathogenic gene" and "therapy to prevent... from producing obstacles".

[0128] To measure each replacement strategy, the coincidence degree of the words in the top k of the tm value can be calculated from the sorted input source text X obtained by the exact tm values of multiple (e.g., 50) source texts to measure the accuracy of the replacement strategy; and the variance of different replacement strategies can be reflected by the rank correlation of the two sorted input source texts X obtained after two different samplings.

[0129] In the above embodiments, the descriptions of each embodiment have their own focuses. For the parts not detailed in a certain embodiment, reference can be made to the detailed description of the method for detecting the performance of the machine translation model above, and details will not be repeated here.

[0130] The embodiments of the present application can obtain the Chinese source text, and obtain the corresponding English target text. According to the replacement strategy, multiple first target words in Chinese are used to replace the second target word in the source text respectively to obtain multiple candidate texts; then the machine translation model can be used to translate the multiple candidate texts to obtain the first translation text in English corresponding to each candidate text, and the source text is translated by the machine translation model to obtain the second translation text in English; at this time, the translation performance index of the machine translation model can be detected according to the first similarity between the first translation text and the target text and the second similarity between the second translation text and the target text, so as to adjust the machine translation model according to the translation performance index, so that the adjusted machine translation model can accurately translate the text, improving the accuracy of detecting the performance of the machine translation model.

[0131] In order to better implement the machine translation model performance detection method provided in the embodiment of the present application, the embodiment of the present application also provides a device based on the above machine translation model performance detection method. The meanings of the terms are the same as those in the above machine translation model performance detection method, and the specific implementation details can refer to the description in the method embodiment.

[0132] See also Fig.12 , Fig.12 A structural schematic diagram of a machine translation model performance detection device provided in an embodiment of the present application, wherein the machine translation model performance detection device may include an acquisition unit 301, a replacement unit 302, a first translation unit 303, a second translation unit 304, and a detection unit 305, etc.

[0133] The acquisition unit 301 is used to acquire a source text in a first language type and acquire a target text in a second language type corresponding to the source text.

[0134] The replacing unit 302 is configured to use a plurality of first target words of the first language type to replace the second target words in the source text respectively, so as to obtain a plurality of candidate texts.

[0135] The first translation unit 303 is used to translate multiple candidate texts through a machine translation model to obtain a first translation text of the second language type corresponding to each candidate text.

[0136] The second translation unit 304 is used to translate the source text through a machine translation model to obtain a second translated text in a second language.

[0137] The detection unit 305 is used to detect the translation performance index of the machine translation model according to the first similarity between the first translated text and the target text and the second similarity between the second translated text and the target text.

[0138] In one embodiment, the replacement unit 302 can be specifically used to: calculate the gradient value corresponding to the second target word in the source text through the loss function of the machine translation model based on the source text and the target text; sample multiple first target words from the vocabulary of the first language type according to the gradient value; and replace the second target words in the source text with the multiple first target words respectively to obtain multiple candidate texts.

[0139] In one embodiment, the replacement unit 302 can be specifically used to: update the word vector of the second target word in the source text according to the gradient value according to a preset learning rate to obtain an updated word vector; perform a dot product operation on the updated word vector and the word vector matrix corresponding to the vocabulary of the first language type to obtain a similarity vector between the word vector of the second target word in the source text and each word vector in the word vector matrix; and sample multiple first target words whose similarity with the second target word in the source text is greater than a preset similarity threshold from the vocabulary of the first language type according to the similarity vector.

[0140] In one embodiment, the replacement unit 302 can be specifically used to: select multiple candidate words from a vocabulary of the first language type; replace the second target words in the source text with the multiple candidate words respectively to obtain multiple replacement texts; calculate the loss value between the replacement text and the target text through the loss function of the machine translation model; select a preset number of candidate words corresponding to the minimum loss value to obtain multiple first target words; replace the second target words in the source text with the multiple first target words respectively to obtain multiple candidate texts.

[0141] In one implementation, the replacement unit 302 can be specifically used to: determine a replacement position in a source text; randomly select a plurality of first target words from a vocabulary of a first language type; and replace the plurality of first target words with second target words corresponding to the replacement position in the source text, respectively, to obtain a plurality of candidate texts.

[0142] In one implementation, the replacement unit 302 can be specifically used to: determine the attribute information of the second target word in the source text; filter out multiple first target words from the vocabulary of the first language type according to the attribute information; and replace the second target words in the source text with the multiple first target words respectively to obtain multiple candidate texts.

[0143] In one embodiment, the detection unit 305 can be specifically used to: obtain a first similarity between a first translated text and a target text, and a second similarity between a second translated text and the target text; filter out candidate texts whose first similarity is greater than the second similarity from the first translated texts corresponding to multiple candidate texts to obtain a target candidate text; analyze the part-of-speech distribution of a second target word in a source text based on the target candidate text; and determine the translation performance index of a machine translation model based on the part-of-speech distribution.

[0144] In one embodiment, the detection unit 305 can be specifically used to: divide the target candidate text into multiple groups of candidate texts according to the replacement position of the second target word in the source text; calculate the mean of the similarity corresponding to each group of candidate texts; and determine the part-of-speech distribution of the second target word in the source text based on the previous preset group of candidate texts with the largest mean.

[0145] In one embodiment, the detection unit 305 can be specifically used to: obtain a first initial similarity between a first translated text and a target text, and a second initial similarity between a second translated text and the target text through a machine translation model; calculate a first translation score of the candidate text and a second translation score of the source text using a reranking algorithm; adjust the first initial similarity according to the first translation score to obtain a first similarity between the first translated text and the target text, and adjust the second initial similarity according to the second translation score to obtain a second similarity between the second translated text and the target text.

[0146] In one embodiment, the replacement unit 302 can be specifically used to: replace words in the source text with multiple first target words of the first language type according to different replacement strategies, and obtain multiple candidate texts corresponding to each replacement strategy; the detection unit 305 can be specifically used to: screen out candidate texts with a first similarity greater than a second similarity from the first translated texts corresponding to the multiple candidate texts obtained by replacement based on each replacement strategy, and obtain target candidate texts corresponding to each replacement strategy; divide the target candidate texts corresponding to each replacement strategy into multiple groups of candidate texts according to the replacement position of the second target word in the source text, and calculate the mean of the similarity corresponding to each group of candidate texts; sort the candidate texts corresponding to each replacement strategy in order from high to low according to the mean, and analyze the overlap of the source text corresponding to each replacement strategy according to the sorting results; determine the translation performance index of the machine translation model according to the overlap.

[0147] In one embodiment, the source text includes multiple, and the detection unit 305 can be specifically used to: screen out candidate texts with a first similarity greater than a second similarity from the first translation texts corresponding to the multiple candidate texts corresponding to each source text, to obtain a target candidate text corresponding to each source text; divide the target candidate text corresponding to each source text into multiple groups of candidate texts according to the replacement position of the second target word in the source text, and calculate the mean of the similarity corresponding to each group of candidate texts; calculate the variance value of the similarity corresponding to the candidate text in the candidate text group with the largest mean; and determine the translation performance index of the machine translation model according to the variance value.

[0148] In one embodiment, the machine translation model performance detection device may further include:

[0149] An adjustment unit, used to adjust the machine translation model according to the translation performance index of the machine translation model to obtain an adjusted machine translation model;

[0150] The translation unit is used to translate the text to be translated using the adjusted machine translation model.

[0151] In the embodiment of the present application, the acquisition unit 301 can acquire the source text of the first language type, and acquire the target text of the second language type corresponding to the source text, and the replacement unit 302 can use multiple first target words of the first language type to replace the second target words in the source text respectively, so as to obtain multiple candidate texts; then the first translation unit 303 can translate the multiple candidate texts through the machine translation model to obtain the first translation text of the second language type corresponding to each candidate text, and the second translation unit 304 can translate the source text through the machine translation model to obtain the second translation text of the second language type; at this time, the detection unit 305 can detect the translation performance index of the machine translation model according to the first similarity between the first translation text and the target text, and the second similarity between the second translation text and the target text. The scheme translates the candidate text and the source text obtained after the word replacement through the machine translation model, and performs statistical analysis based on the first similarity between the first translation text and the target text, and the second similarity between the second translation text and the target text, so as to accurately detect the translation performance index of the machine translation model, thereby improving the accuracy of the performance detection of the machine translation model.

[0152] The present application also provides a computer device, which may be a computer device such as Fig.13 As shown, it shows a schematic diagram of the structure of the computer device involved in the embodiment of the present application, specifically:

[0153] The computer device may include components such as a processor 401 with one or more processing cores, a memory 402 with one or more computer-readable storage media, a power supply 403, and an input unit 404. Those skilled in the art will appreciate that Fig.13 The computer device structure shown in the figure does not constitute a limitation on the computer device, and may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently. Among them:

[0154] The processor 401 is the control center of the computer device. It uses various interfaces and lines to connect various parts of the entire computer device. By running or executing software programs and / or modules stored in the memory 402 and calling data stored in the memory 402, it executes various functions of the computer device and processes data, thereby performing overall detection of the computer device. Optionally, the processor 401 may include one or more processing cores; preferably, the processor 401 may integrate an application processor and a modem processor, wherein the application processor mainly processes the operating system, user interface, and application programs, and the modem processor mainly processes wireless communications. It is understandable that the above-mentioned modem processor may not be integrated into the processor 401.

[0155] The memory 402 can be used to store software programs and modules. The processor 401 executes various functional applications and data processing by running the software programs and modules stored in the memory 402. The memory 402 may mainly include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function (such as a sound playback function, an image playback function, etc.), etc.; the data storage area may store data created according to the use of the computer device, etc. In addition, the memory 402 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other volatile solid-state storage devices. Accordingly, the memory 402 may also include a memory controller to provide the processor 401 with access to the memory 402.

[0156] The computer device also includes a power supply 403 for supplying power to various components. Preferably, the power supply 403 can be logically connected to the processor 401 through a power management system, so as to manage charging, discharging, and power consumption through the power management system. The power supply 403 can also include any components such as one or more DC or AC power supplies, recharging systems, power failure detection circuits, power converters or inverters, and power status indicators.

[0157] The computer device may further include an input unit 404, which may be used to receive input digital or character information and generate keyboard, mouse, joystick, optical or trackball signal input related to user settings and function control.

[0158] Although not shown, the computer device may further include a display unit, etc., which will not be described in detail herein. Specifically in this embodiment, the processor 401 in the computer device will load the executable files corresponding to the processes of one or more application programs into the memory 402 according to the following instructions, and the processor 401 will run the application programs stored in the memory 402, thereby realizing various functions, as follows:

[0159] A source text of a first language type is obtained, and a target text of a second language type corresponding to the source text is obtained; a plurality of first target words of the first language type are used to replace second target words in the source text respectively, to obtain a plurality of candidate texts; the plurality of candidate texts are translated by a machine translation model to obtain a first translated text of the second language type corresponding to each candidate text; the source text is translated by the machine translation model to obtain a second translated text of the second language type; and a translation performance index of the machine translation model is detected according to a first similarity between the first translated text and the target text, and a second similarity between the second translated text and the target text.

[0160] In the above embodiments, the description of each embodiment has its own focus. For the parts that are not described in detail in a certain embodiment, please refer to the detailed description of the machine translation model performance detection method above, and will not be repeated here.

[0161] According to one aspect of the present application, a computer program product or a computer program is provided, the computer program product or the computer program including computer instructions, the computer instructions being stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device executes the methods provided in various optional implementations in the above-mentioned embodiments.

[0162] Those skilled in the art will appreciate that all or part of the steps in the various methods of the above embodiments can be completed by computer instructions, or by controlling related hardware through computer instructions, and the computer instructions can be stored in a computer-readable storage medium and loaded and executed by a processor. To this end, an embodiment of the present application provides a storage medium (i.e., a computer-readable storage medium) in which a computer program is stored, and the computer program may include computer instructions, and the computer program can be loaded by a processor to execute any of the machine translation model performance detection methods provided in the embodiments of the present application.

[0163] The specific implementation of the above operations can be found in the previous embodiments, which will not be described in detail here.

[0164] The storage medium may include: a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, etc.

[0165] Since the instructions stored in the storage medium can execute the steps in any one of the machine translation model performance detection methods provided in the embodiments of the present application, the beneficial effects that can be achieved by any one of the machine translation model performance detection methods provided in the embodiments of the present application can be achieved. Please refer to the previous embodiments for details and will not be repeated here.

[0166] The above is a detailed introduction to a machine translation model performance detection method and related equipment provided in an embodiment of the present application. Specific examples are used in this article to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only used to help understand the method of the present application and its core idea; at the same time, for technical personnel in this field, according to the idea of ​​the present application, there will be changes in the specific implementation method and application scope. In summary, the content of this specification should not be understood as a limitation on the present application.

Claims

1. A method for detecting performance of a machine translation model, characterized in that: include: Acquire a source text in a first language type, and acquire a target text in a second language type corresponding to the source text; Replacing the second target words in the source text with the plurality of first target words of the first language type respectively to obtain a plurality of candidate texts; Translate the multiple candidate texts through a machine translation model to obtain a first translated text of the second language type corresponding to each candidate text; Translating the source text by using the machine translation model to obtain a second translated text in the second language type; Detecting a translation performance indicator of the machine translation model according to a first similarity between the first translated text and the target text and a second similarity between the second translated text and the target text; The detecting the translation performance index of the machine translation model according to the first similarity between the first translation text and the target text and the second similarity between the second translation text and the target text comprises: Acquire a first similarity between the first translated text and the target text, and a second similarity between the second translated text and the target text; Filtering out candidate texts whose first similarity is greater than the second similarity from first translation texts corresponding to multiple candidate texts to obtain a target candidate text; Analyzing the part-of-speech distribution of the second target word in the source text according to the target candidate text; A translation performance index of the machine translation model is determined according to the part-of-speech distribution.

2. The method according to claim 1, characterized in that: The step of replacing the second target words in the source text with the plurality of first target words of the first language type to obtain a plurality of candidate texts includes: Based on the source text and the target text, calculating a gradient value corresponding to a second target word in the source text by using a loss function of the machine translation model; Sampling a plurality of first target words from a vocabulary of the first language type according to the gradient value; The multiple first target words are used to replace the second target words in the source text respectively to obtain multiple candidate texts.

3. The method according to claim 2, characterized in that The step of sampling a plurality of first target words from a vocabulary of the first language type according to the gradient value comprises: Update the word vector of the second target word in the source text according to the gradient value at a preset learning rate to obtain an updated word vector; Performing a dot product operation on the updated word vector and the word vector matrix corresponding to the word table of the first language type to obtain a similarity vector between the word vector of the second target word in the source text and each word vector in the word vector matrix; A plurality of first target words whose similarity to the second target words in the source text is greater than a preset similarity threshold are sampled from the vocabulary of the first language type according to the similarity vector.

4. The method according to claim 1, characterized in that The step of replacing the second target words in the source text with the plurality of first target words of the first language type to obtain a plurality of candidate texts includes: Selecting a plurality of candidate words from the word list of the first language type; Replacing the second target words in the source text with the multiple candidate words respectively to obtain multiple replacement texts; Calculating the loss value between the replacement text and the target text by using the loss function of the machine translation model; Screening out a preset number of candidate words corresponding to the minimum loss value to obtain multiple first target words; The multiple first target words are used to replace the second target words in the source text respectively to obtain multiple candidate texts.

5. The method according to claim 1, characterized in that: The step of replacing the second target words in the source text with the plurality of first target words of the first language type to obtain a plurality of candidate texts includes: determining a replacement position in the source text; Randomly select a plurality of first target words from the word list of the first language type; The multiple first target words are used to replace the second target words corresponding to the replacement positions in the source text respectively, to obtain multiple candidate texts.

6. The method according to claim 1, characterized in that The step of replacing the second target words in the source text with the plurality of first target words of the first language type to obtain a plurality of candidate texts includes: Determining attribute information of a second target word in the source text; Filtering a plurality of first target words from the word list of the first language type according to the attribute information; The multiple first target words are used to replace the second target words in the source text respectively to obtain multiple candidate texts.

7. The method according to claim 1, characterized in that The step of analyzing the part-of-speech distribution of the second target word in the source text according to the target candidate text includes: Dividing the target candidate text into multiple groups of candidate texts according to the replacement position of the second target word in the source text; Calculate the mean of the similarity corresponding to each group of candidate texts; The part-of-speech distribution of the second target word in the source text is determined according to the first preset group of candidate texts with the largest mean.

8. The method according to claim 1, characterized in that The obtaining of a first similarity between the first translated text and the target text, and a second similarity between the second translated text and the target text, includes: Acquire, by the machine translation model, a first initial similarity between the first translated text and the target text, and a second initial similarity between the second translated text and the target text; Calculating a first translation score of the candidate text and a second translation score of the source text using a re-ranking algorithm; The first initial similarity is adjusted according to the first translation score to obtain a first similarity between the first translated text and the target text, and the second initial similarity is adjusted according to the second translation score to obtain a second similarity between the second translated text and the target text.

9. The method according to claim 1, characterized in that: The step of replacing the second target words in the source text with the plurality of first target words of the first language type to obtain a plurality of candidate texts includes: Using a plurality of first target words of the first language type to replace words in the source text respectively according to different replacement strategies, to obtain a plurality of candidate texts corresponding to each replacement strategy; The detecting the translation performance index of the machine translation model according to the first similarity between the first translation text and the target text and the second similarity between the second translation text and the target text comprises: Selecting candidate texts whose first similarity is greater than the second similarity from the first translation texts corresponding to the multiple candidate texts obtained by replacing based on each replacement strategy, respectively, to obtain target candidate texts corresponding to each replacement strategy; Divide the target candidate text corresponding to each replacement strategy into multiple groups of candidate texts according to the replacement position of the second target word in the source text, and calculate the average value of the similarity corresponding to each group of candidate texts; The candidate texts corresponding to each replacement strategy are sorted in descending order of the mean value, and the overlap degree of the source text corresponding to each replacement strategy is analyzed according to the sorting result; A translation performance index of the machine translation model is determined according to the overlap degree.

10. The method according to claim 1, characterized in that The source text includes a plurality of texts, and detecting the translation performance index of the machine translation model according to a first similarity between the first translation text and the target text and a second similarity between the second translation text and the target text includes: Selecting candidate texts whose first similarity is greater than the second similarity from the first translation texts corresponding to the multiple candidate texts corresponding to each source text, respectively, to obtain a target candidate text corresponding to each source text; Divide the target candidate text corresponding to each source text into multiple groups of candidate texts according to the replacement position of the second target word in the source text, and calculate the average value of the similarity corresponding to each group of candidate texts; Calculate the variance value of the similarity corresponding to the candidate text in the candidate text group with the largest mean; A translation performance index of the machine translation model is determined according to the variance value.

11. The method according to any one of claims 1 to 10, characterized in that: After detecting the translation performance index of the machine translation model according to the first similarity between the first translated text and the target text and the second similarity between the second translated text and the target text, the method further includes: Adjusting the machine translation model according to the translation performance index of the machine translation model to obtain an adjusted machine translation model; The text to be translated is translated using the adjusted machine translation model.

12. A machine translation model performance detection device, characterized in that: include: An acquisition unit, configured to acquire a source text in a first language type, and acquire a target text in a second language type corresponding to the source text; A replacement unit, configured to replace second target words in the source text with a plurality of first target words of the first language type respectively, to obtain a plurality of candidate texts; A first translation unit, configured to translate the plurality of candidate texts by using a machine translation model to obtain a first translation text of the second language type corresponding to each candidate text; A second translation unit, configured to translate the source text by using the machine translation model to obtain a second translated text in the second language type; a detection unit, configured to detect a translation performance indicator of the machine translation model according to a first similarity between the first translated text and the target text and a second similarity between the second translated text and the target text; The detection unit comprises: Acquire a first similarity between the first translated text and the target text, and a second similarity between the second translated text and the target text; Filtering out candidate texts whose first similarity is greater than the second similarity from first translation texts corresponding to multiple candidate texts to obtain a target candidate text; Analyzing the part-of-speech distribution of the second target word in the source text according to the target candidate text; A translation performance index of the machine translation model is determined according to the part-of-speech distribution.

13. The device according to claim 12, characterized in that The replacement unit is specifically configured to calculate, based on the source text and the target text, a gradient value corresponding to a second target word in the source text by using a loss function of the machine translation model; and sample a plurality of first target words from a vocabulary of the first language type according to the gradient value; The multiple first target words are used to replace the second target words in the source text respectively to obtain multiple candidate texts.

14. The device according to claim 13, characterized in that The replacement unit is specifically configured to update the word vector of the second target word in the source text according to the gradient value at a preset learning rate to obtain an updated word vector; perform a dot product operation on the updated word vector and a word vector matrix corresponding to the word table of the first language type to obtain a similarity vector between the word vector of the second target word in the source text and each word vector in the word vector matrix; and sample, from the word table of the first language type according to the similarity vector, a plurality of first target words whose similarity to the second target word in the source text is greater than a preset similarity threshold.

15. The device according to claim 12, characterized in that The replacement unit is specifically used to select multiple candidate words from the vocabulary of the first language type; replace the second target words in the source text with the multiple candidate words respectively to obtain multiple replacement texts; calculate the loss value between the replacement text and the target text through the loss function of the machine translation model; select a preset number of candidate words corresponding to the minimum loss value to obtain multiple first target words; The multiple first target words are used to replace the second target words in the source text respectively to obtain multiple candidate texts.

16. The device according to claim 12, characterized in that The replacement unit is specifically used to determine a replacement position in the source text; randomly select a plurality of first target words from the vocabulary of the first language type; and replace the second target words corresponding to the replacement position in the source text with the plurality of first target words respectively to obtain a plurality of candidate texts.

17. The device according to claim 12, characterized in that The replacement unit is specifically used to determine the attribute information of the second target word in the source text; select multiple first target words from the vocabulary of the first language type according to the attribute information; and replace the second target words in the source text with the multiple first target words respectively to obtain multiple candidate texts.

18. The device according to claim 12, characterized in that The detection unit is specifically used to divide the target candidate text into multiple groups of candidate texts according to the replacement position of the second target word in the source text; and calculate the average value of the similarity corresponding to each group of candidate texts; The part-of-speech distribution of the second target word in the source text is determined according to the first preset group of candidate texts with the largest mean.

19. The device according to claim 12, characterized in that The detection unit is specifically configured to obtain a first initial similarity between the first translated text and the target text, and a second initial similarity between the second translated text and the target text through the machine translation model; A first translation score of the candidate text and a second translation score of the source text are calculated using a reranking algorithm; the first initial similarity is adjusted according to the first translation score to obtain a first similarity between the first translated text and the target text; and the second initial similarity is adjusted according to the second translation score to obtain a second similarity between the second translated text and the target text.

20. The device according to claim 12, characterized in that The replacement unit is specifically configured to replace words in the source text with a plurality of first target words of the first language type according to different replacement strategies, so as to obtain a plurality of candidate texts corresponding to each replacement strategy; the detection unit is specifically configured to select candidate texts having a first similarity greater than a second similarity from the first translation texts corresponding to the plurality of candidate texts obtained by replacement based on each replacement strategy, so as to obtain a target candidate text corresponding to each replacement strategy; Divide the target candidate text corresponding to each replacement strategy into multiple groups of candidate texts according to the replacement position of the second target word in the source text, and calculate the average value of the similarity corresponding to each group of candidate texts; The candidate texts corresponding to each replacement strategy are sorted in descending order of the mean value, and the overlap of the source texts corresponding to each replacement strategy is analyzed according to the sorting result; and the translation performance index of the machine translation model is determined according to the overlap.

21. The device according to claim 12, characterized in that The source text includes multiple, detection units, which are specifically used to select candidate texts with a first similarity greater than the second similarity from the first translation texts corresponding to the multiple candidate texts corresponding to each source text, respectively, to obtain a target candidate text corresponding to each source text; Divide the target candidate text corresponding to each source text into multiple groups of candidate texts according to the replacement position of the second target word in the source text, and calculate the average value of the similarity corresponding to each group of candidate texts; Calculate the variance value of the similarity corresponding to the candidate texts in the candidate text group with the largest mean; and determine the translation performance index of the machine translation model according to the variance value.

22. The device according to any one of claims 12 to 21, characterized in that The device also includes: An adjustment unit, configured to adjust the machine translation model according to the translation performance index of the machine translation model to obtain an adjusted machine translation model; A translation unit is used to translate the text to be translated by using the adjusted machine translation model.

23. A computer device, characterized in that: The method comprises a processor and a memory, wherein a computer program is stored in the memory, and when the processor calls the computer program in the memory, the method for detecting the performance of a machine translation model according to any one of claims 1 to 11 is executed.

24. A storage medium, characterized in that The storage medium is used to store a computer program, and the computer program is loaded by a processor to execute the machine translation model performance detection method according to any one of claims 1 to 11.

Images