Multilingual translation evaluation system and execution method thereof

Abstract

A multilingual translation evaluation system, according to the present invention, comprises: a machine translation engine providing unit for generating machine-translated documents for a reference document by using different machine translation engines; an analysis option setting unit for receiving settings for evaluation weights and evaluation target documents generated on the basis of the reference document; a quality evaluation unit for evaluating quality differences among the evaluation target documents by comparing the evaluation target documents with each other; and a report generation unit for generating a report using analysis results evaluated by the quality evaluation unit.

Description

Multilingual Translation Evaluation System and Implementation Method

[0001] The present invention relates to a multilingual translation evaluation system and a method for implementing the same. More specifically, the invention relates to a multilingual translation evaluation system and a method for implementing the same in which a function that visually expresses translation quality intuitively shows the status of translation results to the user, thereby enabling the user to easily recognize differences between translations and help to quickly identify problems.

[0002] Machine translation has established itself as a crucial technology that breaks down language barriers and facilitates diverse global communication. This technology can translate large volumes of documents quickly and efficiently without human intervention, offering significant advantages in terms of both cost and time savings. For this reason, machine translation is widely used across various fields, including businesses, government agencies, and academia.

[0003] In particular, the rapid development of artificial intelligence (AI) technology over the past few years has significantly improved the performance of machine translation. Moving beyond simple rule-based translation systems of the past, deep learning-based Neural Machine Translation (NMT) systems have emerged, enabling translation that takes into account complex sentence structures and context to some extent.

[0004] Thanks to these technologies, machine translation systems have acquired the ability to understand the meaning and context of sentences rather than simply comparing words, and the accuracy and flexibility of translation have been greatly improved.

[0005] However, even with the advancement of artificial intelligence, limitations in machine translation still exist. Machine translation often fails to grasp the deep contextual understanding that humans possess, and it faces limitations particularly in translating nuances that account for cultural and situational differences. For instance, unnatural expressions in context, or the omission or distortion of important meanings, may occur.

[0006] Such issues can lead to fatal consequences, particularly in specialized fields. Accurate and consistent translation is essential in legal, medical, and technical documents, and even minor errors can result in significant problems. Therefore, a thorough verification process is necessary to ensure the reliability of machine translation results.

[0007] To address this, various quality evaluation technologies capable of comparing machine-translated texts with human-translated texts are being developed. In this process, a solution capable of evaluating translation accuracy and naturalness by comparing machine and human translations is essential. Such a solution must be able to visualize the differences between translations and quantitatively evaluate those differences.

[0008] For example, the qualitative difference in translation can be quantified using edit distance (i.e., Levenshtein distance) by calculating the difference between a human-translated text and a machine-translated version. Edit distance refers to the minimum number of operations, such as insertions, deletions, and replacements, required to change one text into another. This allows for an intuitive analysis of the differences between the two translations.

[0009] Furthermore, such a solution must be able to evaluate not only the differences between human and machine translations, but also the differences between the proofread version and the translation. By analyzing how much the translation was modified during the post-translation proofreading process, the quality and efficiency of the proofreading work can also be assessed. This approach plays a crucial role in simultaneously improving the quality of both the machine translation and the proofread version.

[0010] In conclusion, evaluating translation quality requires a sophisticated solution capable of comparing and analyzing machine translations with human translations or proofread versions. This enables the provision of natural and error-free translations, and allows for the delivery of higher-quality translations by combining the efficiency advantages of machine translation with the precision of human translation.

[0011] Accordingly, there is a need for a multilingual translation evaluation system and a method for implementing it, which includes a function to visually represent translation quality that intuitively shows users the status of translation results, thereby enabling them to easily recognize differences between translations and help them quickly identify problems.

[0012] The present invention aims to provide a multilingual translation evaluation system and a method for implementing the same, which allows for objective evaluation of translation quality by clearly expressing differences between translated sentences as numerical values ​​such as edit distances, thereby enabling a more intuitive understanding of differences between translations and providing reliability to the quality evaluation of machine translations.

[0013] Furthermore, the present invention aims to provide a multilingual translation evaluation system and a method for implementing the same, which can contribute to improving the completeness of a translation by specifically analyzing the differences between a machine translation and a human translation or proofreading version to easily detect translation errors, thereby enabling the prior discovery and correction of minor errors or contextual unnaturalness occurring during the translation process.

[0014] In addition, the present invention aims to provide a multilingual translation evaluation system and a method for implementing the same, which can detect whether a machine translation engine is used by comparing a machine translation with a human translation, thereby identifying whether machine translation is used and helping to verify the source of the translation for commercial or academic purposes.

[0015] In addition, the present invention aims to provide a multilingual translation evaluation system and a method for implementing the same, wherein the function of visually expressing translation quality intuitively shows the status of the translation results to the user, thereby enabling the user to easily recognize differences between translations and help to quickly identify problems.

[0016] In addition, the present invention aims to provide a multilingual translation evaluation system and a method for implementing the same, which can significantly improve the efficiency of translation work by providing a function to customize translation quality analysis results according to options set by the user, thereby enabling analysis tailored to user needs, such as processing large files and ignoring specific error types.

[0017] The objects of the present invention are not limited to those mentioned above, and other unmentioned objects and advantages of the present invention may be understood from the following description and will be more clearly understood by the embodiments of the present invention. Furthermore, it will be readily apparent that the objects and advantages of the present invention can be realized by the means and combinations thereof set forth in the claims.

[0018] A multilingual translation evaluation system for achieving this purpose includes a machine translation engine providing unit that generates a machine translation of a reference document through different machine translation engines, an analysis option setting unit that receives evaluation target documents generated based on the reference document and evaluation weights, a quality evaluation unit that evaluates quality differences between evaluation target documents by comparing each of the evaluation target documents, and a report generating unit that generates a report using the analysis results evaluated by the quality evaluation unit.

[0019] In one embodiment, the quality evaluation unit includes a first quality evaluation unit that calculates and evaluates the edit distance for each segment of the document to be evaluated, a second quality evaluation unit that generates each translation by translating each segment through the machine translation engine and determines whether it is a machine translation by comparing each translation with each segment, and a third quality evaluation unit that evaluates whether a grammatical error has occurred in each segment.

[0020] In one embodiment, the first quality evaluation unit calculates an edit distance by comparing each of the segments and using the number of editing operations to convert the first segment into a second segment, and can evaluate the similarity between the documents to be evaluated according to the edit distance.

[0021] In one embodiment, the second quality evaluation unit can compare each of the segments with a translation generated by the machine translation engine, evaluate the similarity between the segment and the machine translation, and determine whether to use the machine translation engine.

[0022] In addition, a multilingual translation evaluation method for achieving this purpose may include the steps of receiving evaluation target documents and evaluation weights generated based on a reference document, evaluating quality differences between evaluation target documents by comparing each of the evaluation target documents, and generating a report using the analysis results based on the evaluation.

[0023] According to the present invention as described above, by clearly expressing the difference between translated sentences as a numerical value such as the editing distance, the translation quality can be objectively evaluated, allowing for a more intuitive understanding of the differences between translations and providing reliability to the quality evaluation of machine translation.

[0024] In addition, according to the present invention, there is an advantage in that translation errors can be easily detected by specifically analyzing the differences between a machine translation and a human translation or proofreading version, thereby enabling the discovery and correction of minor errors or contextual unnaturalness occurring during the translation process in advance, which can contribute to improving the completeness of the translation.

[0025] In addition, according to the present invention, there is an advantage in that it is possible to detect whether a machine translation engine is used by comparing a machine translation with a human translation, thereby identifying whether machine translation is used and helping to verify the source of the translation for commercial or academic purposes.

[0026] In addition, according to the present invention, the function of visually expressing translation quality has the advantage of intuitively showing the status of the translation results to the user, thereby enabling the user to easily recognize differences between translations and helping to quickly identify problems.

[0027] In addition, according to the present invention, by providing a function that allows the translation quality analysis results to be customized according to options set by the user, it is possible to enable analysis tailored to user requirements, such as processing large files and ignoring specific error types, thereby having the advantage of significantly improving the efficiency of translation work.

[0028] FIG. 1 is a drawing for explaining a multilingual translation evaluation system according to one embodiment of the present invention.

[0029] FIGS. 2 to 7 are illustrative diagrams for explaining another embodiment of a multilingual translation evaluation system according to the present invention.

[0030] FIG. 8 is a drawing for illustrating a multilingual translation evaluation system according to another embodiment of the invention.

[0031] FIG. 9 is a flowchart illustrating an embodiment of a multilingual translation evaluation method according to the present invention.

[0032] A machine translation engine provider that generates machine translations of a reference document through different machine translation engines; and

[0033] An analysis option setting unit that receives evaluation target documents generated based on the above reference documents and evaluation weights; and

[0034] A quality evaluation unit that evaluates quality differences between the documents subject to evaluation by comparing each of the above-mentioned documents subject to evaluation; and

[0035] It includes a report generation unit that generates a report using the analysis results evaluated by the quality evaluation unit above, and

[0036] The above analysis option setting section is,

[0037] It further includes enabling users to set comparison targets and evaluation weights so that they can compare or analyze only specific parts of a translation, comparing results translated by a machine translation engine with results translated directly by humans and setting comparison targets, analyzing the quality of revision and proofreading based on the compared data, and enabling sampling settings that allow for the comparison of large files.

[0038] The above quality evaluation department,

[0039] It includes a first quality evaluation unit that calculates and evaluates the edit distance for each segment of the document subject to evaluation, a second quality evaluation unit that generates each translation by translating each segment through the machine translation engine and determines whether it is a machine translation by comparing each translation and each segment, and a third quality evaluation unit that evaluates whether a grammatical error has occurred in each segment.

[0040] The above-mentioned first quality evaluation department is,

[0041] The edit distance is calculated by comparing each of the above segments and using the number of editing operations to convert the first segment into the second segment, and the similarity between the documents to be evaluated is evaluated according to the edit distance.

[0042] The above second quality evaluation department is,

[0043] A multilingual translation evaluation system characterized by comparing each of the above segments with a translation generated by the machine translation engine to evaluate the similarity between the segment and the machine translation and determine whether to use the machine translation engine, and using BLEU scores to evaluate the similarity between the machine translation and the original text in machine translation detection and measuring the similarity through Equation 1.

[0044] [Mathematical Formula 1]

[0045]

[0046] (Here, N-grams refer to a sequence of N words, count(n-gramC) refers to the number of times a specific n-gram appears in the translated text, count(n-gramReference) refers to the number of times the corresponding n-gram appears in the reference text (i.e., the text translated by humans), and the min function means using the minimum number of n-grams found in the translated text and the reference text.)

[0047] The above-mentioned second quality evaluation department.

[0048] A multilingual translation evaluation system characterized by calculating the precision of multiple n-grams of BLEU scores calculated in the above mathematical formula 1, and then calculating the geometrically weighted average of them through mathematical formula 2.

[0049] [Mathematical Formula 2]

[0050]

[0051] (Here, N: maximum length of n-grams used, Pn: n-gram precision)

[0052] A multilingual translation evaluation system characterized by calculating a length penalty through mathematical formula 3 to prevent the translated sentence of the above BLEU score from becoming shorter.

[0053] [Mathematical Formula 3]

[0054]

[0055] (Here, c: number of words in the translated text, r: number of words in the reference text (i.e., human-translated text))

[0056] A multilingual translation evaluation system characterized by the above-mentioned second quality evaluation unit calculating a final BLEU score by multiplying the calculated geometrically weighted average by a length penalty, and calculating the final BLEU score through mathematical formula 4.

[0057] [Mathematical Formula 4]

[0058]

[0059] (Here, BP: length penalty, Pn: precision of each n-gram, N: maximum n-gram length)

[0060] In the method of execution using a multilingual translation evaluation system,

[0061] The above execution method is,

[0062] A step of receiving evaluation target documents and evaluation weights generated based on reference documents;

[0063] A step of evaluating quality differences between the documents to be evaluated by comparing each of the above-mentioned documents to be evaluated; and

[0064] It includes a step of generating a report using the analysis results based on the evaluation, and

[0065] The step of evaluating quality differences between the documents subject to evaluation by comparing each of the above-mentioned documents subject to evaluation is:

[0066] A step of calculating and evaluating the edit distance for each segment of the document to be evaluated;

[0067] A step of generating each translation by translating each segment through a machine translation engine, and determining whether each translation is machine translation by comparing each segment with each translation;

[0068] Characterized by including a step of evaluating whether a grammar error occurred in each segment

[0069] Method for executing the multilingual translation evaluation system.

[0070]

[0071] The aforementioned objectives, characteristics, and advantages are described in detail below with reference to the attached drawings, thereby enabling those skilled in the art to easily implement the technical concept of the present invention. In describing the present invention, detailed descriptions of known technologies related to the present invention are omitted if it is determined that such descriptions would unnecessarily obscure the essence of the invention. Hereinafter, preferred embodiments according to the present invention will be described in detail with reference to the attached drawings. In the drawings, the same reference numerals are used to indicate the same or similar components.

[0072] FIG. 1 is a drawing for illustrating a multilingual translation evaluation system according to one embodiment of the present invention. FIGS. 2 to 7 are illustrative drawings for illustrating another embodiment of a multilingual translation evaluation system according to the present invention.

[0073] Referring to FIG. 1, the multilingual translation evaluation system includes a machine translation engine selection unit (110), an analysis option setting unit (120), a quality evaluation unit (130), a result visualization unit (140), and a report generation unit (150). The quality evaluation unit (130) includes a first quality evaluation unit (131), a second quality evaluation unit (132), and a third quality evaluation unit (133).

[0074] The machine translation engine provider (110) generates a translation of the original text through various machine translation engines. To this end, the machine translation engine provider (110) receives a selection from the user of a machine translation engine to generate the translation, and then generates a translation of the original text through the machine translation engine selected by the user. In this way, by using various types of machine translation engines, translations generated by different machine translation engines can be compared and evaluated.

[0075] The analysis option setting unit (120) allows the user to set the comparison target and evaluation weight so that only specific parts of the translation can be compared or analyzed. That is, the analysis option setting unit (120) determines which translations to set as evaluation target documents when comparing evaluation target documents generated based on a reference document.

[0076] In one embodiment, the analysis option setting unit (120) sets up a comparison between a result translated by a machine translation engine (i.e., machine translation) and a result translated directly by a person (i.e., a human translation). Through this, the quality of the machine translation can be objectively evaluated and the differences between the machine translation and the human translation can be analyzed.

[0077] In another embodiment, the analysis option setting unit (120) compares a document translated by a person (i.e., a person's translation) with a final proofread version that has been revised after reviewing the translation. This allows for the analysis of which parts of the translation the reviewer modified, improvements to the translation, and the quality of the review.

[0078] In another embodiment, the analysis option setting unit (120) analyzes the potential for improvement of the machine translation by comparing the final translation reviewed by the reviewer with the machine translation. This method is useful for evaluating how close the machine translation is to the quality of the human-reviewed version.

[0079] As described above, the analysis option setting unit (120) allows the user to select and analyze the optimal pair of translations suitable for the purpose of analysis through comparison target settings. Each comparison setting plays an important role in evaluating errors and differences that occur at specific stages of the translation work.

[0080] Additionally, the analysis option setting unit (120) sets sampling to save time and hardware resources when processing large files. In particular, when comparing large documents, instead of analyzing the entire file, a method of analyzing only a part of the document using a sampling technique can be selected. Through this function, only a certain percentage of the file can be randomly extracted and compared.

[0081] The first quality evaluation unit (131) visually displays the differences of the documents to be evaluated set in the analysis option setting unit (120) and evaluates the translation quality by quantifying the differences using an edit distance algorithm.

[0082] To this end, the first quality evaluation unit (131) divides each sentence of the document to be evaluated, set in the analysis option setting unit (120), into segments (phrase units) and compares them individually. Here, a segment may be a sentence, a clause, or a smaller text unit.

[0083] In one embodiment, the first quality evaluation unit (131) maintains the same parts of each sentence of the documents to be evaluated set in the analysis option setting unit (120) and visually highlights and displays the parts that differ. For example, the first quality evaluation unit (131) displays the same parts of the sentences as they are, but displays the changed parts with a strikethrough and in red, and visualizes the added parts with an underline and in blue.

[0084] After that, the first quality evaluation unit (131) compares the sentences and calculates the edit distance to numerically evaluate the difference between the two sentences. Here, the first quality evaluation unit (131) calculates the Levenshtein distance using an edit (Levenshtein) distance algorithm. At this time, the edit distance is an algorithm that calculates the minimum number of edit operations (insertion, deletion, replacement) required to convert one string into another string by comparing two strings (sentences). For example, to change "cat" to "bat", only one operation of replacing 'c' with 'b' is required, so the edit distance is 1.

[0085]

[0086] [Table 1]

[0087]

[0088] The formula given in [Table 1] defines the Levenshtein distance for two strings a and b. Here, the length of string a is i and the length of string b is j. The condition refers to the case where one string is empty. In this case, the edit distance is equal to the length of the other string. For example, the Levenshtein distance between the string "cat" and the empty string "" is 3. This is because the edit distance is 3 since all characters of "cat" must be deleted.

[0089] Then, if both strings have a length of 1 or more, the edit distance becomes the minimum value among deletion (ld(i-1,j)+1), insertion (ld(i,j-1)+1), and replacement (ld(i-1,j-1)+k). Deletion (ld(i-1,j)+1) is the case where a character is deleted from the first string a. In this case, the edit distance is the previous state distance plus one. Insertion (ld(i,j-1)+1) is the case where a character is inserted into the second string b. In this case as well, it is the previous state distance plus one. For replacement (ld(i-1,j-1)+k), k=0 if the corresponding characters in the two strings are the same, and k=1 if they are different. If the two corresponding characters are the same, no replacement operation is needed, so there is no additional increase in the distance; if they are different, a replacement operation is needed, so the edit distance increases by 1.

[0090] For example, assuming we are comparing two strings "kitten" and "sitting", we compare the first characters. Since 'k' and 's' are different, a replacement operation is required, and the edit distance is + 1. Then, we compare the next characters; since 'i' and 'i' are the same, the edit distance does not increase. We compare the next characters in the same way and proceed with the editing operation. In this manner, we compare all characters and calculate the necessary insertion, deletion, and replacement operations.

[0091] Then, the first quality evaluation unit (131) quantifies the difference between each sentence based on the results of applying the edit distance algorithm. The first quality evaluation unit (131) determines that the smaller the edit distance, the higher the similarity between the translations and the better the quality, and determines that the larger the edit distance, the greater the difference between the translations and the more necessary the translation quality is to improve.

[0092] The second quality evaluation unit (132) determines whether the text was written by a human or generated by a machine translation engine (e.g., Google Translate, DeepL, etc.).

[0093] To this end, the second quality evaluation unit (132) selects some sentences or paragraphs based on the sampling information set in the analysis option setting unit (120) and determines whether machine translation is used. For example, if the sampling information set in the analysis option setting unit (120) is a certain percentage of the text (e.g., 10%, 20%), the second quality evaluation unit (132) takes only randomly selected sentences as subjects for analysis. These sampled texts are used in the process of detecting whether a machine translation engine is used. This process must be designed so that the selected samples can accurately represent the entire document.

[0094] After that, the second quality evaluation unit (132) compares the text with translations generated by three commercial machine translation engines (e.g., Google Translate, Microsoft Translator, DeepL) after the sampled text is extracted, and determines whether the text is a machine translation.

[0095] In one embodiment, the second quality evaluation unit (132) uses a BLEU score to evaluate the similarity between the machine translation and the original text in machine translation detection. The BLEU score is a representative indicator that quantitatively evaluates the similarity between the translated text and the reference text (i.e., the text translated by a human).

[0096] BLEU measures the similarity between an N-gram-based translated text and a reference text (i.e., a text translated by humans). Here, an N-gram refers to a sequence of N consecutive words. pn represents the n-gram precision and is given by [Equation 1].

[0097]

[0098] [Mathematical Formula 1]

[0099]

[0100]

[0101] Count(n-gramC) represents the number of times a specific n-gram appears in the translated text, and Count(n-gramReference) represents the number of times the corresponding n-gram appears in the reference text (i.e., the text translated by humans). The min function uses the minimum number of n-grams found in the translated text and the reference text.

[0102] After that, BLEU typically calculates the precision for several n-grams (e.g., 1-gram, 2-gram, 3-gram, 4-gram) and then takes a weighted average of them. To do this, the geometric mean of each n-gram precision is used. The geometric mean of the BLEU score is calculated as in [Equation 2].

[0103]

[0104] [Mathematical Formula 2]

[0105]

[0106] N: Maximum length of n-grams used,

[0107] Pn: n-gram precision,

[0108]

[0109] As mentioned above, the reason for using the geometric mean is to ensure that n-grams with very low precision do not have a significant impact on the overall BLEU score.

[0110] The BLEU score applies a length penalty to prevent the translated sentence from becoming excessively short. The length penalty is calculated based on the total length c of the translated text and the total length r of the reference text (i.e., the text translated by humans).

[0111]

[0112] [Mathematical Formula 3]

[0113]

[0114]

[0115] c: number of words in the translated text,

[0116] r: Number of words in the reference text (i.e., human-translated text),

[0117]

[0118] As described above, the second quality evaluation unit (132) calculates the final BLEU score by multiplying the calculated geometric mean by the length penalty. In one embodiment, the second quality evaluation unit (132) calculates the final BLEU score as in [Equation 4].

[0119]

[0120] [Mathematical Formula 4]

[0121]

[0122] BP: Length penalty,

[0123] Pn: precision for each n-gram,

[0124] N: maximum n-gram length,

[0125]

[0126] The third quality evaluation unit (133) detects various errors that may occur in the segment and provides them visually so that the user can easily identify the errors. To this end, the third quality evaluation unit (133) evaluates the quality of the translation by primarily checking the consistency of terms, identifying punctuation errors, and performing spelling checks in the segment.

[0127] First, the third quality evaluation unit (133) automatically checks whether a word or phrase in a segment matches a term in a predefined glossary. If the third quality evaluation unit (133) finds that an expression different from the term defined in the glossary is used in the segment, it detects that the term in the translation is inconsistent and marks it as an error.

[0128] Additionally, the third quality evaluation unit (133) detects incorrect punctuation usage in a sentence and visually displays it to the user. Punctuation errors are detected based on grammar rules. Each language has punctuation usage rules, and failure to follow these rules is considered an error. For example, punctuation errors occur when commas are overused or omitted, when a period is missing at the end of a sentence, and when quotation marks, parentheses, etc., are mismatched.

[0129] Additionally, the third quality evaluation unit (133) detects spelling errors by comparing each word included in the segment with the correct spelling stored in the database. For example, if the word "word" is misspelled as "danwo," the system detects this and provides a correction suggestion to the user.

[0130] The result visual section (140) is displayed in color according to the similarity of the segments so that the quality of the translation can be intuitively understood. For example, the result visual section (140) can be displayed in red if the similarity is 90% or higher.

[0131] The report generation unit (150) generates a report summarizing the analysis results. At this time, the report generation unit (150) displays segments with errors and the corresponding error types, and may ignore punctuation errors, etc., depending on the selected option. Translation quality is quantified based on the number of analyzed segments.

[0132]

[0133] FIG. 8 is a diagram illustrating a multilingual translation evaluation system according to another embodiment of the present invention.

[0134] Referring to FIG. 8, the multilingual translation evaluation system includes a multi-file input unit (210), a quality evaluation unit (220), an AI model (230), and an AI learning unit (240).

[0135] The multi-file input section (210) allows the user to upload multiple translation files to the system. These files may be the result of applying different translation methods (machine translation, proofreading, etc.) to the same reference document. Accordingly, the AI ​​model (230) segments each file input from the multi-file input section (210) based on the same criteria. At this time, the segments can be divided into sentence units or paragraph units, and each segment is evaluated independently.

[0136] The quality evaluation unit (220) compares each segment and assigns a score to each segment. At this time, the score calculated for each segment is subsequently reflected in AI learning.

[0137] In one embodiment, the quality evaluation unit (220) generates a first quality evaluation result by calculating an edit distance to measure the consistency between each segment using an edit distance (Levenshtein distance) algorithm. The quality evaluation unit (220) may assign a high score to each segment by determining that the segment is more accurate and similar the shorter the edit distance.

[0138] In another embodiment, the quality evaluation unit (220) compares each segment with a translation produced by a commercial machine translation engine. At this time, a second quality evaluation result is generated by using a BLEU score to detect whether the segment was generated based on the output of the machine translation engine. The quality evaluation unit (220) measures the similarity with the machine translation engine through the BLEU score and evaluates the quality of the segment according to the score. The AI ​​model uses this score to select the segment that is more natural and closer to human translation.

[0139] In another embodiment, the quality evaluation unit (220) generates a third quality evaluation result by verifying whether each segment has used the correct terms according to a predefined glossary. At this time, if the terms are not used consistently, they are recorded as errors, and the quality score of the segment is lowered. At this time, segments with punctuation and spelling errors are detected by an AI model, and the quality score of the segment is determined according to the degree of the error.

[0140] The AI ​​model (230) compares the scores assigned to each segment and selects the segment with the highest quality among multiple translations of the same original text. This is repeated throughout the file and ultimately produces a single result file in which all segments are of high quality.

[0141] In one embodiment, the AI ​​model (230) combines selected highest quality segments to generate a single result file. This file is composed of only the highest quality segments collected from various translations.

[0142] The AI ​​learning unit (240) evaluates the quality of the segments based on the evaluation results by the quality evaluation unit (220). The AI ​​learning unit (240) comprehensively evaluates the quality of each segment and generates an AI model that selects the segment with the highest quality.

[0143] To this end, the AI ​​learning unit (240) learns a pattern in which the AI ​​model connects each quality evaluation result with an actual good translation based on the learning data. This enables the evaluation of each segment when a new translation is given in the future. At this time, the input data is the characteristics of each segment, such as the edit distance, BLEU score, and error detection result, and each segment is labeled with an evaluation result evaluated by a user.

[0144] The AI ​​learning unit (240) receives the characteristics of a segment as input and learns to output a score or grade for translation quality. At this time, the AI ​​learning unit (240) uses a loss function to reduce the difference between the model's predicted value and the actual correct value. For example, when the AI ​​learning unit (240) predicts a translation quality score, a regression loss function (such as MSE) may be used.

[0145]

[0146] FIG. 9 is a flowchart illustrating an embodiment of a multilingual translation evaluation method according to the present invention.

[0147] Referring to FIG. 9, the multilingual translation evaluation system receives evaluation target documents and evaluation weights generated based on reference documents (step S310).

[0148] The multilingual translation evaluation system evaluates the quality difference between the documents to be evaluated by comparing each of the documents to be evaluated (step S320).

[0149] In one embodiment of step S320, the multilingual translation evaluation system evaluates each segment of the document to be evaluated by calculating the edit distance.

[0150] In the above embodiment, the multilingual translation evaluation system calculates an edit distance by comparing each segment and using the number of editing operations to convert a first segment into a second segment, and can evaluate the similarity between the documents to be evaluated according to the edit distance.

[0151] In another embodiment of step S320, the multilingual translation evaluation system may generate each translation by translating each segment through the machine translation engine, and determine whether each translation is machine translation by comparing each translation with each segment.

[0152] In another embodiment of step S320, the multilingual translation evaluation system may include a step of evaluating whether a grammatical error occurred in each segment.

[0153] The multilingual translation evaluation system generates a report using the analysis results based on the evaluation (step S330).

[0154] Meanwhile, since the description of the technology disclosed in this specification is merely an example for structural or functional explanation, the scope of the disclosed technology should not be interpreted as being limited by the examples described in the text. That is, since the examples are subject to various modifications and may take various forms, the scope of the disclosed technology should be understood to include equivalents capable of realizing the technical concept. Furthermore, since the purposes or effects presented in the disclosed technology do not imply that a specific example must include all of them or only such effects, the scope of the disclosed technology should not be understood as being limited by them.

[0155] Furthermore, when it is stated that one component is “connected” to another component, it should be understood that while it may be directly connected to that other component, there may also be other components in between. Conversely, when it is stated that one component is “directly connected” to another component, it should be understood that there are no other components in between. Meanwhile, other expressions describing the relationships between components, such as “between” and “between” or “adjacent to” and “directly adjacent to,” should be interpreted in the same way.

[0156] A singular expression should be understood to include a plural expression unless the context clearly indicates otherwise, and terms such as “include” or “have” are intended to specify the existence of the set-up features, numbers, steps, actions, components, parts, or combinations thereof, and should be understood not to preclude the existence or addition of one or more other features, numbers, steps, actions, components, parts, or combinations thereof.

[0157] The present invention is not limited to the specific preferred embodiments described above, and any person skilled in the art to which the invention pertains can make various modifications without departing from the essence of the invention claimed in the claims, and any modifications are within the scope of the claims as long as they relate to the technical concept forming such modifications.

Claims

1. A machine translation engine provider that generates machine translations of a reference document through different machine translation engines; and An analysis option setting unit that receives evaluation target documents generated based on the above reference documents and evaluation weights; and A quality evaluation unit that evaluates quality differences between the documents subject to evaluation by comparing each of the above-mentioned documents subject to evaluation; and It includes a report generation unit that generates a report using the analysis results evaluated by the quality evaluation unit above, and The above analysis option setting section is, It further includes enabling users to set comparison targets and evaluation weights so that they can compare or analyze only specific parts of a translation, comparing results translated by a machine translation engine with results translated directly by humans and setting comparison targets, analyzing the quality of revision and proofreading based on the compared data, and enabling sampling settings that allow for the comparison of large files. The above quality evaluation department, It includes a first quality evaluation unit that calculates and evaluates the edit distance for each segment of the document subject to evaluation, a second quality evaluation unit that generates each translation by translating each segment through the machine translation engine and determines whether it is a machine translation by comparing each translation and each segment, and a third quality evaluation unit that evaluates whether a grammatical error has occurred in each segment. The above-mentioned first quality evaluation department is, The edit distance is calculated by comparing each of the above segments and using the number of editing operations to convert the first segment into the second segment, and the similarity between the documents to be evaluated is evaluated according to the edit distance. The above second quality evaluation department is, A multilingual translation evaluation system characterized by comparing each of the above segments with a translation generated by the machine translation engine to evaluate the similarity between the segment and the machine translation and determine whether to use the machine translation engine, and using BLEU scores to evaluate the similarity between the machine translation and the original text in machine translation detection and measuring the similarity through Equation 1. [Mathematical Formula 1] (Here, N-grams refer to a sequence of N words, count(n-gramC) refers to the number of times a specific n-gram appears in the translated text, count(n-gramReference) refers to the number of times the corresponding n-gram appears in the reference text (i.e., the text translated by humans), and the min function means using the minimum number of n-grams found in the translated text and the reference text.) 2. In Paragraph 1, The above-mentioned second quality evaluation department. A multilingual translation evaluation system characterized by calculating the precision of multiple n-grams of BLEU scores calculated in the above mathematical formula 1, and then calculating the geometrically weighted average of them through mathematical formula 2. [Mathematical Formula 2] (Here, N: maximum length of n-grams used, Pn: n-gram precision) 3. In Paragraph 2, A multilingual translation evaluation system characterized by calculating a length penalty through mathematical formula 3 to prevent the translated sentence of the above BLEU score from becoming shorter. [Mathematical Formula 3] (Here, c: number of words in the translated text, r: number of words in the reference text (i.e., human-translated text)) 4. In Paragraph 3, A multilingual translation evaluation system characterized by the above-mentioned second quality evaluation unit calculating a final BLEU score by multiplying the calculated geometrically weighted average by a length penalty, and calculating the final BLEU score through mathematical formula 4. [Mathematical Formula 4] (Here, BP: length penalty, Pn: precision of each n-gram, N: maximum n-gram length) 5. A method of execution using a multilingual translation evaluation system according to any one of claims 1 to 4, The above execution method is, A step of receiving evaluation target documents and evaluation weights generated based on reference documents; A step of evaluating quality differences between the documents to be evaluated by comparing each of the above-mentioned documents to be evaluated; and It includes a step of generating a report using the analysis results based on the evaluation, and The step of evaluating quality differences between the documents subject to evaluation by comparing each of the above-mentioned documents subject to evaluation is: A step of calculating and evaluating the edit distance for each segment of the document to be evaluated; A step of generating each translation by translating each segment through a machine translation engine, and determining whether each translation is machine translation by comparing each segment with each translation; Characterized by including a step of evaluating whether a grammar error occurred in each segment Method for executing the multilingual translation evaluation system.

Images