Corpus matching method after excluding non-translated elements

Abstract

The invention provides a corpus matching method after excluding non-translation elements, which comprises corpus data processing and storage, a non-translation regular mode and a corpus matching process, and improves the translation efficiency. The method focuses on essential contents of the text instead of formats or digits and the like, can enable translation to pay more attention to accurate conveying of semantics, and improves the overall quality of translation.

Description

Technical Field

[0001] The present invention relates to the field of translation technology, and in particular to a corpus matching method after excluding non-translation elements. Background Art

[0002] Corpus matching is a very common process in the translation process. The current corpus matching is performed based on the equality of the original sentence and the original corpus, but some similar (equal after removing non-translated) contents cannot be matched. Non-translated elements usually refer to content that does not need to be translated during the translation process, such as date, time, number, proper noun, code snippet, label, special character, etc. The present invention is to improve the success rate of corpus matching and ensure the correctness of the translation results. Summary of the invention

[0003] The purpose of the present invention is to provide a corpus matching method after excluding non-translation elements in view of the above-mentioned deficiencies in the prior art.

[0004] To achieve the above object, the present invention adopts the following technical solutions:

[0005] The present invention provides a corpus matching method after excluding non-translation elements, comprising: S1, corpus data processing and storage, specifically:

[0006] S101, read the Chinese and English material file EXCEL, and obtain the original text and the translated text of the material;

[0007] S102, space removal processing: perform md5 on the original corpus and the translated corpus to obtain the original corpus MD5 and the translated corpus MD5; according to the language judgment, perform space removal processing on the original corpus and the translated corpus to obtain the original corpus with similar content and the similar translated corpus, and then perform md5 processing to obtain the similar original corpus MD5 and the similar translated corpus MD5;

[0008] S103, non-translation regular mode processing: the similar corpus original text and similar corpus translation are processed based on the non-translation regular mode. When the match is successful, the S-CSRC and S-CTAR are modified to obtain the non-translation corpus original text and the non-translation corpus translation, obtain the non-translation placeholder mapping data, and then perform md5 processing to obtain the non-translation corpus original text MD5 and the non-translation corpus translation MD5;

[0009] S104. Create the elasticsearch index es_ze_index of the current corpus to store all the above fields.

[0010] Furthermore, it also includes: S2 non-translation regular mode, specifically:

[0011] S201, define grouping regular rules, the levels are AZ and are defined by 26 English letters, which play the role of grouping and identification; the regular content is the specific regular to be matched from the original corpus and the translated corpus;

[0012] Whether it is a standard value: 0-does not exist, 1-non-standard, 2-standard;

[0013] S202, define the regular expression to be deleted to exclude abnormal intervention;

[0014] S203, non-translation content extraction:

[0015] Pass in the original text of similar corpus, cyclically group the non-translation regular expressions, and if there is a match, use the regular expression to determine whether the content is to be deleted. If it is to be deleted, skip the matched content; if it is not to be deleted, replace the matched content with a placeholder, and obtain the original text of the non-translation corpus and the placeholder metadata;

[0016] S204, matching process:

[0017] Corpus non-translation extraction rules: The number and value of non-translation content in the original and translated texts must be consistent.

[0018] Further, there are the similar corpus original text and the similar corpus translation, and the operation is:

[0019] Circularly group the non-translation regular expressions. If there is a match, use the regular expression to determine whether the content is to be deleted. If it is to be deleted, skip the matched content. If the matching content in the original and translated texts is consistent, modify the original and translated texts to obtain the non-translation corpus original text, non-translation corpus translation, and non-translation placeholder mapping content.

[0020] Furthermore, the S2 non-translation regular mode further includes:

[0021] S205, non-translation restoration process:

[0022] The input corpus non-translation placeholder translation N-CTAR, the corpus non-translation placeholder mapping original text non-translation placeholder metadata;

[0023] Loop N-SRC-META:

[0024] If N-CSRC-PC is empty, then directly replace the non-translation placeholder in N-CTAR with the real non-translation content according to N-SRC-META-PC in N-SRC-META to obtain the final translation data;

[0025] If N-CSRC-PC is not empty, loop N-CSRC-PC and replace the non-translation placeholders in N-CTAR with real non-translation content according to N-SRC-META-PC in N-SRC-META to obtain the final translation data.

[0026] Furthermore, it also includes the S3 corpus matching process:

[0027] S301, input the original text and language direction for corpus matching; perform mb5 to obtain the original text MD5; perform the space removal process in S102 to obtain similar original text, perform mb5 to obtain similar original text MD5; based on the similar original text, perform the non-translated content extraction in S203 to obtain the non-translated original text and non-translated placeholder metadata, perform md5 on N-SRC to obtain the non-translated original text MD5;

[0028] S302, perform elasticsearch query, query incoming SRC-D, S-SRC-D, N-SRC-D, store temporary variable original text matching MD set, match CSRC-D, S-CSRC-D and N-CSRC-D fields of es_ze_index index respectively, as long as one of the three fields can match, the match is successful, and all fields of the matching corpus data are obtained;

[0029] S303, matching corpus translation restoration processing

[0030] Get the matched original corpus MD5, similar original corpus MD5, and non-translated original corpus MD5. If ES-SM-D contains the first three values, it means that the original corpus matches, otherwise it means that the translated corpus matches.

[0031] Further, the S303 specifically includes:

[0032] S3031, if it is a corpus original text match

[0033] If the original text MD5 is equal to the matching CSRC-D, then the current original text matching corpus translation is CTAR of ES-F;

[0034] If the similar original text MD5 is equal to the matching S-CSRC-D, then the current original text matching corpus translation is S-CTAR of ES-F;

[0035] If the non-translation source text MD5 is equal to the matching N-CSRC-D, then the current source text matching corpus translation is the N-CTAR of ES-F;

[0036] S3032: If the original text does not match the corpus

[0037] If the original text MD5 is equal to the matching CTAR-D, then the current original text matching corpus translation is ES-F's CSRC;

[0038] If the similar original text MD5 is equal to the matching S-CTAR-D, then the current original text matching corpus translation is the S-CSRC of ES-F;

[0039] If the non-translation source text MD5 is equal to the matching N-CTAR-D, then the current source text matching corpus translation is the N-CSRC of ES-F;

[0040] S3033、Translation processing for corpus matching

[0041] If it is not a non-translated original text or a non-translated translation, the matching ends and the matching translation data is directly returned;

[0042] If it is a non-translated original text or a non-translated translation match, it is necessary to restore the non-translated content of the translation. According to the S205 non-translation restoration process, the non-translated original text N-CSRC of the corpus, the non-translated placeholder mapping data of the corpus, whether it is reversed, and the non-translated placeholder metadata of the original text are input to obtain the final translation.

[0043] The beneficial effects of the present invention are: improving translation efficiency: excluding non-translatable elements can significantly reduce the processing load of the translation engine, because the translation engine does not have to parse and process these elements, thereby speeding up the translation speed.

[0044] Maintain format and structure: Non-translatable elements such as date, time format, phone number, etc. often have specific formats. Keeping the original format of these elements helps to maintain the accuracy and readability of the text.

[0045] Reduce errors: Excluding non-translatable elements such as numbers and code snippets can prevent the translation engine from misunderstanding their meaning, thereby reducing translation errors.

[0046] Improve translation quality: Focusing on the substantive content of the text rather than format or numbers can allow the translator to pay more attention to the accurate communication of semantics and improve the overall quality of the translation.

[0047] Reduce costs: Reducing unnecessary translation workload, especially when processing large-scale texts, can effectively reduce translation costs. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] Figure 1 It is a diagram of the corpus storage process;

[0049] Figure 2 This is the corpus matching flow chart. DETAILED DESCRIPTION

[0050] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.

[0051] See also Figure 1 and Figure 2 , a corpus matching method after excluding non-translation elements, comprising: S1, corpus data processing and storage, specifically:

[0052] S101, read the Chinese and English (LANG-Z, LANG-E) corpus file EXCEL, and obtain the original corpus (CSRC) and the translated corpus (CTAR);

[0053] S102, space removal processing: perform md5 on the original corpus (CSRC) and the translated corpus (CTAR) to obtain the original corpus MD5 (CSRC-M) and the translated corpus MD5 (CTAR-M); according to the language judgment, perform space removal processing on the original corpus (CSRC) and the translated corpus (CTAR) (remove all spaces for Chinese, Japanese, Korean and Thai; remove redundant spaces for other languages, and only keep one space in the middle of each word), and obtain the content-similar original corpus (S-CSRC) and similar translated corpus (S-CTAR), and then perform md5 processing to obtain the similar original corpus MD5 (S-CSRC-M) and similar translated corpus MD5 (S-CTAR-M);

[0054] S103, non-translation regular mode processing: the similar corpus original text (S-CSRC) and the similar corpus translation (S-CTAR) are processed based on the non-translation regular mode. When the match is successful, S-CSRC and S-CTAR are modified to obtain the non-translation corpus original text (N-CSRC) and the non-translation corpus translation (N-CTAR), and obtain the non-translation placeholder mapping data (N-CSRC-PC, including N-CSRC and N-CTAR, and the corresponding relationship between the original text and the translation placeholder (PLAC)), and then md5 processing is performed to obtain the non-translation corpus original text MD5 (N-CSRC-M) and the non-translation corpus translation MD5 (N-CTAR-M);

[0055] S104. Create the index es_ze_index of the current corpus elasticsearch to store all the above fields (ES-F).

[0056] S2 non-translation regular mode, specifically:

[0057] S201, define grouping regular rules (level, regular content, whether it is a standard number), the level is AZ, which is defined by 26 English letters, and plays a role in grouping and identification; the regular content is the specific regular that is defined to be matched from the original corpus (CSRC) and the translated corpus (CTAR);

[0058] Whether the value is standard: 0-does not exist, 1-non-standard, 2-standard; (solves the problem that "." will be translated into ",". The thousand separator is "," and the decimal point is ".");

[0059] For example, there are grouping regular expressions AR-1, AR-2, and FR-1. Define the placeholder (PLAC) content after the regular expression is matched. Currently, it is fixed to level + matching sequence number + curly brackets, such as {{A1}}. Non-translation regular expressions are divided into multiple levels according to priority, and multiple levels can be configured;

[0060] S202, define the regular expression (DR) to be deleted to exclude abnormal intervention; for example, HTML tag attribute content;

[0061] S203, non-translation content extraction:

[0062] Input the original text of similar corpus (S-CSRC), cyclically group the non-translation regular expression (DR), and if there is a match, use the regular expression to determine whether the content is to be deleted. If it is to be deleted, skip the matched content; if it is not to be deleted, replace the matched content with a placeholder to obtain the original text of non-translation corpus (N-CSRC) and placeholder metadata (N-CSRC-META, including fields: placeholder, non-translation value, whether it is a standard number, etc., used to restore the translation);

[0063] S204, matching process:

[0064] Corpus non-translation extraction rules: The number and value of non-translation content in the original and translated texts must be consistent.

[0065] With the similar corpus original text (S-CSRC) and similar corpus translation (S-CTAR), the operation is:

[0066] Circularly group the non-translation regular expressions. If there is a match, the regular expression (DR) is used to determine whether the content is to be deleted. If it is to be deleted, the matched content is skipped. If the matched content in the original and translated texts is consistent (if it is non-standard, the thousand separator "," is converted to a decimal point "." before comparison), the original and translated texts are modified to obtain the non-translation corpus original text (N-CSRC), non-translation corpus translation (N-CTAR) and non-translation placeholder mapping content (N-CSRC-PC, the correspondence between the original placeholder and the translated placeholder).

[0067] The S2 non-translation regular mode also includes:

[0068] S205, non-translation restoration process:

[0069] The input corpus non-translation placeholder translation N-CTAR (or non-translation placeholder original text N-CSRC, if the translation is matched, the reverse matching field needs to be passed), the corpus non-translation placeholder mapping (N-CSRC-PC) and the original non-translation placeholder metadata (N-SRC-META, including the specific non-translation content corresponding to the non-translation placeholder (N-SRC-META-PC));

[0070] Loop N-SRC-META:

[0071] If N-CSRC-PC is empty, then directly replace the non-translation placeholder in N-CTAR with the real non-translation content according to N-SRC-META-PC in N-SRC-META (to determine whether it is non-standard, it is necessary to replace the real non-translation content with a thousand separator "," and convert it to a decimal point "." (If it is a reverse match, it is necessary to replace the decimal point "." with a thousand separator ",")) (If it is a reverse match, the non-translation placeholder in N-CSRC is replaced with the real non-translation content according to N-SRC-META-PC in N-SRC-META) to obtain the final translation data;

[0072] If N-CSRC-PC is not empty, loop N-CSRC-PC and replace the non-translation placeholders in N-CTAR with real non-translation content according to N-SRC-META-PC in N-SRC-META to obtain the final translation data.

[0073] Also included is the S3 corpus matching process:

[0074] S301, input the original text (SRC) and the language direction for corpus matching; perform mb5 to obtain the original text MD5 (SRC-D); perform the space removal process of S102 to obtain the similar original text (S-SRC), perform mb5 to obtain the similar original text MD5 (S-SRC-D); according to the similar original text (S-SRC), perform the non-translated content extraction in S203 to obtain the non-translated original text (N-SRC) and the non-translated placeholder metadata (N-SRC-META), perform md5 on N-SRC to obtain the non-translated original text MD5 (N-SRC-D);

[0075] S302, perform elasticsearch query, query incoming SRC-D, S-SRC-D, N-SRC-D, store temporary variable original text matching MD set (ES-SM-D), match CSRC-D (or CTAR-D), S-CSRC-D (or S-CTAR-D) and N-CSRC-D (or N-CTAR-D) fields of es_ze_index index respectively, as long as one of the three fields can match, the match is successful, and all fields of the matching corpus data (ES-F) are obtained;

[0076] S303, matching corpus translation restoration processing

[0077] Get the matching corpus original text MD5 (CSRC-M), corpus similar original text MD5 (S-CSRC-M), corpus non-translated original text MD5 (N-CSRC-M). If ES-SM-D contains the first three values, it means that the corpus original text matches (M-SRC), otherwise it is a corpus translation match (the final translation needs to be transformed).

[0078] The S303 specifically includes:

[0079] S3031, if it is corpus original text matching (M-SRC)

[0080] If the original text MD5 (SRC-D) is equal to the matched CSRC-D, then the current original text matching corpus translation is CTAR of ES-F;

[0081] If the similar source text MD5 (S-SRC-D) is equal to the matching S-CSRC-D, then the current source text matching corpus translation is S-CTAR of ES-F;

[0082] If the non-translation source text MD5 (N-SRC-D) is equal to the matching N-CSRC-D, then the current source text matching corpus translation is the N-CTAR of ES-F;

[0083] S3032, if it is not a corpus original text match (M-SRC)

[0084] If the original text MD5 (SRC-D) is equal to the matched CTAR-D, then the current original text matching corpus translation is ES-F's CSRC;

[0085] If the similar original text MD5 (S-SRC-D) is equal to the matching S-CTAR-D, then the corpus translation of the current original text is S-CSRC of ES-F;

[0086] If the non-translation source text MD5 (N-SRC-D) is equal to the matching N-CTAR-D, then the current source text matching corpus translation is the N-CSRC of ES-F;

[0087] S3033、Translation processing for corpus matching

[0088] If it is not a non-translated original text or a non-translated translation match, the matching ends and the matching translation data (CTAR or S-CTAR or CSRC or S-CSRC) is directly returned;

[0089] If it is a non-translated original text or a non-translated translation match, it is necessary to restore the non-translated content of the translation. According to the S205 non-translated restoration process, the non-translated original text N-CSRC (or non-translated translation N-CTAR) of the corpus, the non-translated placeholder mapping data of the corpus (N-CSRC-PC of ES-F), whether it is reverse (corpus translation match), and the non-translated placeholder metadata (N-SRC-META) of the original text are input to obtain the final translation.

[0090] For example: the existing grouping non-translation regular expression is as follows:

[0091] Level A, regular content: pure numbers;

[0092] Level F, regular content: numbers with thousands signs and decimal parts, non-standard (that is, thousands signs need to be converted to decimal points);

[0093] Deleted regular expression (DR): attribute content of HTML tags;

[0094] Example 1: Only one level A regular expression is matched;

[0095] The original text is: I have 1 apple and 2 pears.

[0096] Translation: I have 1 apple and 2 pearls.

[0097] The original translation is: I have {{A1}} apples and {{A2}} pears;

[0098] The non-translated version is: I have {{A1}}apple and {{A2}}pears.

[0099] Original sentence: I have 1 apple and 2 pears.

[0100] If the match is equal to the original text, the translation is returned directly if the match is successful.

[0101] I have 1 apple and 2pears.

[0102] Original sentence: I have 1 apple and 2 pears.

[0103] After matching and removing spaces, the translation is returned directly if the md5 match is successful based on similar corpus.

[0104] I have 1 apple and 2pears.

[0105] Original sentence: I have 3 apples and 2 pears.

[0106] After matching, the non-translation rule pattern is used to obtain the non-translation original text "I have {{A1}} apples {{A2}} pears". The non-translation original text is matched successfully, and the non-translation content is restored to obtain the translation.

[0107] I have 3apples and 2pears.

[0108] Original sentence: I have 1 apple and 2pears.

[0109] After matching, the non-translation rule pattern is obtained to obtain the non-translation original text "I have {{A1}} apple and {{A2}} pears". The non-translation translation is matched successfully (reverse matching), and the non-translation content is restored to obtain the translation.

[0110] I have 1 apple and 2 pears.

[0111] Example 2: Does not satisfy any of the grouping non-translation regularities.

[0112] The original text of the corpus is: I have 1 apple and 2 pears.

[0113] Translation: I have 4apples and 2pears.

[0114] If the non-translation rule mode is not met, the non-translation original text will not be stored.

[0115] Original sentence: I have 3 apples and 2 pears.

[0116] Matching is performed, no match.

[0117] Example 3: Satisfying multiple grouping non-translation rules

[0118] The original text is: I have 1 apple and 2 pears, 1000.999 yuan

[0119] Translation: I have 2pears and 1apple, 1000,999yuan

[0120] The original translation is: I have {{A1}} apples {{A2}} pears, {{F3}} yuan

[0121] The non-translated version is: I have {{A2}}apple and {{A1}}pears, {{F3}}yuan

[0122] The placeholder mapping relationship is: A1->A1; A2->A2; F3->F3, non-standard

[0123] Original sentence: I have 3 apples and 2 pears, 2000.999 yuan

[0124] Match the non-translation rule pattern to get the non-translation original text

[0125] I have {{A1}} apples {{A2}} pears, {{F3}} yuan

[0126] The non-translated original text is matched successfully, and the non-translated content is restored to obtain the translated text

[0127] I have 2pears and 3apple,1000,999yuan

[0128] The above-mentioned embodiments only express the implementation methods of the present invention, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the patent of the present invention. It should be pointed out that, for ordinary technicians in this field, several variations and improvements can be made without departing from the concept of the present invention, and these all belong to the protection scope of the present invention. Therefore, the protection scope of the patent of the present invention should be based on the attached claims.

Claims

1. A corpus matching method after excluding non-translatable elements, characterized in that: include: S1. Corpus data processing and storage, specifically: S101, read the Chinese and English material file EXCEL, and obtain the original text and the translated text of the material; S102, space removal processing: perform MD5 on the original text and the translated text to obtain the original text MD5 and the translated text MD5; According to the language judgment, the original corpus and the translated corpus are processed to remove spaces, and the original corpus with similar content and the translated corpus with similar content are obtained. Then, md5 processing is performed to obtain the MD5 of the similar original corpus and the MD5 of the similar translated corpus. S103, non-translation regular mode processing: the similar corpus original text and similar corpus translation are processed based on the non-translation regular mode. When the match is successful, the S-CSRC and S-CTAR are modified to obtain the non-translation corpus original text and the non-translation corpus translation, obtain the non-translation placeholder mapping data, and then perform md5 processing to obtain the non-translation corpus original text MD5 and the non-translation corpus translation MD5; S104. Create the elasticsearch index es_ze_index of the current corpus to store all the above fields.

2. A corpus matching method after excluding non-translation elements according to claim 1, characterized in that: Also includes: S2 non-translation regular mode, specifically: S201, define grouping regular rules, the levels are AZ and are defined by 26 English letters, which play the role of grouping and identification; the regular content is the specific regular to be matched from the original corpus and the translated corpus; Whether it is a standard value: 0-does not exist, 1-non-standard, 2-standard; S202, define the regular expression to be deleted to exclude abnormal intervention; S203, non-translation content extraction: Pass in the original text of similar corpus, cyclically group the non-translation regular expressions, and if there is a match, use the regular expression to determine whether the content is to be deleted. If it is to be deleted, skip the matched content; if it is not to be deleted, replace the matched content with a placeholder, and obtain the original text of the non-translation corpus and the placeholder metadata; S204, matching process: Corpus non-translation extraction rules: The number and value of non-translation content in the original and translated texts must be consistent.

3. The corpus matching method after excluding non-translation elements according to claim 2 is characterized in that: With the similar corpus original text and similar corpus translation, the operation is: Circularly group the non-translation regular expressions. If there is a match, use the regular expression to determine whether the content is to be deleted. If it is to be deleted, skip the matched content. If the matching content in the original and translated texts is consistent, modify the original and translated texts to obtain the non-translation corpus original text, non-translation corpus translation, and non-translation placeholder mapping content.

4. The corpus matching method after excluding non-translation elements according to claim 3 is characterized in that: The S2 non-translation regular mode also includes: S205, non-translation restoration process: The non-translation placeholder translation N-CTAR of the input corpus is mapped to the non-translation placeholder metadata of the original text; Loop N-SRC-META: If N-CSRC-PC is empty, then directly replace the non-translation placeholder in N-CTAR with the real non-translation content according to N-SRC-META-PC in N-SRC-META to obtain the final translation data; If N-CSRC-PC is not empty, loop N-CSRC-PC and replace the non-translation placeholders in N-CTAR with real non-translation content according to N-SRC-META-PC in N-SRC-META to obtain the final translation data.

5. The corpus matching method after excluding non-translation elements according to claim 4 is characterized in that: Also included is the S3 corpus matching process: S301, input the original text and language direction for corpus matching; perform mb5 to obtain the original text MD5; perform the space removal process of S102 to obtain similar original text, and perform mb5 to obtain similar original text MD5; According to the similar original text, the non-translated content extraction in S203 is performed to obtain the non-translated original text and non-translated placeholder metadata, and the N-SRC is subjected to md5 to obtain the non-translated original text MD5; S302, perform elasticsearch query, query incoming SRC-D, S-SRC-D, N-SRC-D, store temporary variable original text matching MD set, match CSRC-D, S-CSRC-D and N-CSRC-D fields of es_ze_index index respectively, as long as one of the three fields can match, the match is successful, and all fields of the matching corpus data are obtained; S303, matching corpus translation restoration processing Get the matched original corpus MD5, similar original corpus MD5, and non-translated original corpus MD5. If ES-SM-D contains the first three values, it means that the original corpus matches, otherwise it means that the translated corpus matches.

6. A corpus matching method after excluding non-translation elements according to claim 5, characterized in that: The S303 specifically includes: S3031, if it is a corpus original text match If the original text MD5 is equal to the matching CSRC-D, then the current original text matching corpus translation is CTAR of ES-F; If the similar original text MD5 is equal to the matching S-CSRC-D, then the current original text matching corpus translation is S-CTAR of ES-F; If the non-translation source text MD5 is equal to the matching N-CSRC-D, then the current source text matching corpus translation is the N-CTAR of ES-F; S3032: If the original text does not match the corpus If the original text MD5 is equal to the matching CTAR-D, then the current original text matching corpus translation is ES-F's CSRC; If the similar original text MD5 is equal to the matching S-CTAR-D, then the current original text matching corpus translation is the S-CSRC of ES-F; If the non-translation source text MD5 is equal to the matching N-CTAR-D, then the current source text matching corpus translation is the N-CSRC of ES-F; S3033、Translation processing for corpus matching If it is not a non-translated original text or a non-translated translation, the matching ends and the matching translation data is directly returned; If it is a non-translated original text or a non-translated translation match, it is necessary to restore the non-translated content of the translation. According to the S205 non-translation restoration process, the non-translated original text N-CSRC of the corpus, the non-translated placeholder mapping data of the corpus, whether it is reversed, and the non-translated placeholder metadata of the original text are input to obtain the final translation.

Images