Method for acquiring translation memory process for multiple languages
By training the multilingual connection model to repair and align sentences in PDF files, the problem of low translation memory quality is solved, and the effect of efficient extraction of complete sentences is achieved. It is suitable for multilingual pairs and improves the quality and yield of translation memory.
Patent Information
- Application Number
- CN202411662276.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-11-20
- Publication Date
- 2025-03-14
AI Technical Summary
The prior art is difficult to effectively extract complete and separate sentences from translated PDF files, resulting in poor quality of translation memory (TMs), affecting the training of machine translation models and the completion of translation tasks.
By training a multilingual "connection" model, identifying and repairing broken sentences and words, removing unnecessary newlines and hyphens, and aligning source and target text through the alignment model to form accurate TMs.
It enables quick and efficient extraction of complete sentences from PDF files, improves the quality and yield of translation memory, is suitable for any language pair, and can be fine-tuned using existing tools and libraries.
Smart Images
Figure CN119623420A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of multilingual translation memory, in particular to a method for obtaining a translation memory process for multiple languages. Background Art
[0002] Translated PDFs can be a very beneficial source of translation memories (TMs), which can be used to assist in translation or train machine translation models. High-quality TMs need to be complete and separate sentences. However, PDFs store text as positioned visual elements rather than logical sentences, which causes extraction tools to misinterpret line breaks and text flow as separate blocks of content rather than continuous text. Words are also often separated by hyphens. This hinders the step of "aligning the source code to the target text", resulting in low yields. By using a novel "Joiner" model to reconstruct broken sentences and words, the process aligns the source and target texts to create accurate TMs. This approach leverages existing tools for scalability and speed, providing a more efficient solution for developing machine translation models or assisting with translation tasks. Summary of the invention
[0003] (I) Purpose of the invention
[0004] In view of this, the purpose of the present invention is to provide a method for obtaining translation memory for multiple languages, which solves the problems raised in the above background.
[0005] (II) Technical solution
[0006] A method for obtaining a translation memory process for multiple languages, the method comprising the following steps:
[0007] 1) Batch download PDF files translated into multiple languages;
[0008] 2) Use the “Faiss” method from Bitextizer / Neural Document Aligner to identify and match the translated versions of the PDF files;
[0009] 3) Prepare the data, including converting the PDF file into text blocks according to the visual blocks and removing the headers, footers and unnecessary lines;
[0010] 4) Implement sentence repair methods, including:
[0011] a) Train a join model: [Only once] train a model (e.g. mBart) to recognize that consecutive lines are part of the same sentence. mBart is a multilingual model, so a single model will be trained for all language pairs. This is the joiner model. The model has two tasks, namely removing line breaks that it deems unnecessary and removing “hyphens” if words are separated at line breaks. The model is trained by using a synthetic dataset of broken sentences as input and “reconstructed” sentences as output. This dataset is created by taking multilingual sentences, splitting them at random positions, and inserting a hyphen as a word.
[0012] b) Annotation: For each block in step 3, run the concatenation model to join the lines together to form complete sentences when necessary. For better performance, batch processing uses batches containing multiple text blocks;
[0013] c) Recover minor edits that were not necessary during the join process by comparing with the original rows;
[0014] d) Segmentation: for the corresponding language (e.g. sentence segmentation library);
[0015] E) Use the sentence segmentation library to segment lines containing multiple sentences into individual sentences;
[0016] 5) Use the alignment model to align sentences from the source and target ends;
[0017] 6) Perform post-processing including removing duplicate sentences and eliminating wrong alignments using duplexer or other QE evaluation tools.
[0018] Preferably, the connection model in the sentence repair method is a multilingual model, applicable to multiple language pairs, and connects lines by removing unnecessary line breaks and hyphens.
[0019] Preferably, the sentence repair method further comprises a recovery step for recovering unnecessary minor edits by comparing the original line with the model output after connection model processing.
[0020] Preferably, the sentence segmentation step uses a rule-based EOL sentence segmenter to separate a line containing multiple sentences into individual sentences.
[0021] Preferably, the training of the Joiner model is also included:
[0022] a) Prepare a high-quality multilingual sentence dataset with each entry as a separate row;
[0023] b) For each training dataset entry, construct an input text block in which some lines have newlines randomly inserted, and construct an output text block in which there are no additional newlines;
[0024] c) During training, convert all line breaks to <mask> tokens and train the model to recognize when to convert a <mask> token to a line break or remove it.
[0025] Preferably, the text blocks of the PDF file are input into the model in parallel in the connection step of the sentence repair method to increase the processing speed.
[0026] Preferably, the method is applicable to any language pair and can correctly extract complete sentences from PDF without relying on existing PDF text extraction libraries.
[0027] Preferably, the method uses simple and popular tools available on the market, and for specialized texts, the model can be fine-tuned to improve throughput.
[0028] Preferably, the method is highly scalable and fast and is capable of creating translation memories (TM) from PDF files.
[0029] It can be seen from the above technical solutions that the present application has the following beneficial effects:
[0030] 1. The invention solves this problem by training a "connection" model to learn to repair broken sentences and words. Then it performs alignment and quality estimation. The method is highly scalable and can quickly create translation memory (TM) from PDF files. It is applicable to any language pair and can produce a large amount of correct transcranial magnetic stimulation. It is highly scalable and fast, and can correctly extract complete sentences from PDF. It can use simple and popular tools available on the market. For specialized texts, the model can be fine-tuned to increase production. BRIEF DESCRIPTION OF THE DRAWINGS
[0031] Figure 1 It is a schematic diagram of the process of the present invention; DETAILED DESCRIPTION
[0032] The following description is merely exemplary in nature and is not intended to limit the present disclosure, application, and use. It should be understood that in all of these figures, the same or similar reference numerals indicate the same or similar parts and features. The various drawings only schematically represent the concepts and principles of the embodiments of the present disclosure, and do not necessarily show the specific dimensions and proportions of the various embodiments of the present disclosure. Specific parts in specific drawings may be exaggerated to illustrate the relevant details or structures of the embodiments of the present disclosure.
[0033] See also Figure 1 , an embodiment provided by the present invention:
[0034] A method for obtaining a translation memory process for multiple languages, the method comprising the following steps:
[0035] 1) Batch download PDF files translated into multiple languages;
[0036] 2) Use the “Faiss” method from Bitextizer / Neural Document Aligner to identify and match the translated versions of the PDF files;
[0037] 3) Prepare the data, including converting the PDF file into text blocks according to the visual blocks and removing the headers, footers and unnecessary lines;
[0038] 4) Implement sentence repair methods, including:
[0039] a) Train a join model: [Only once] train a model (e.g. mBart) to recognize that consecutive lines are part of the same sentence. mBart is a multilingual model, so a single model will be trained for all language pairs. This is the joiner model. The model has two tasks, namely removing line breaks that it deems unnecessary and removing “hyphens” if words are separated at line breaks. The model is trained by using a synthetic dataset of broken sentences as input and “reconstructed” sentences as output. This dataset is created by taking multilingual sentences, splitting them at random positions, and inserting a hyphen as a word.
[0040] b) Annotation: For each block in step 3, run the concatenation model to join the lines together to form complete sentences when necessary. For better performance, batch processing uses batches containing multiple text blocks;
[0041] c) Recover minor edits that were not necessary during the join process by comparing with the original rows;
[0042] d) Segmentation: for the corresponding language (e.g. sentence segmentation library);
[0043] E) Use the sentence segmentation library to segment lines containing multiple sentences into individual sentences;
[0044] 5) Use the alignment model to align sentences from the source and target ends;
[0045] 6) Perform post-processing including removing duplicate sentences and eliminating wrong alignments using duplexer or other QE evaluation tools.
[0046] Furthermore, the connection model in the sentence repair method is a multilingual model that is applicable to multiple language pairs and connects lines by removing unnecessary line breaks and hyphens.
[0047] Furthermore, the sentence repair method also includes a recovery step for recovering unnecessary small edits by comparing the original line with the model output after the connection model processing.
[0048] Furthermore, the sentence segmentation step uses a rule-based EOL sentence segmenter to separate lines containing multiple sentences into individual sentences.
[0049] Furthermore, it also includes the training of the Joiner model:
[0050] a) Prepare a high-quality multilingual sentence dataset with each entry as a separate row;
[0051] b) For each training dataset entry, construct an input text block in which some lines have newlines randomly inserted, and construct an output text block in which there are no additional newlines;
[0052] c) During training, convert all line breaks to <mask> tokens and train the model to recognize when to convert a <mask> token to a line break or remove it.
[0053] Furthermore, text chunks of the PDF file are fed into the model in parallel during the concatenation step of the sentence repair method to increase the processing speed.
[0054] Furthermore, the method is applicable to any language pair and can correctly extract complete sentences from PDF without relying on existing PDF text extraction libraries.
[0055] Furthermore, our approach uses simple and popular tools available on the market, and for specialized texts, the model can be fine-tuned to improve throughput.
[0056] Furthermore, the method is highly scalable and fast and is able to create translation memories (TM) from PDF files.
[0057] It will be apparent to those skilled in the art that the invention is not limited to the details of the exemplary embodiments described above and that the invention can be implemented in other specific forms without departing from the spirit or essential features of the invention. Therefore, the embodiments should be considered exemplary and non-limiting in all respects, and the scope of the invention is defined by the appended claims rather than the foregoing description, and it is intended that all variations falling within the meaning and scope of the equivalent elements of the claims be included in the invention. Any reference numeral in a claim should not be considered as limiting the claim to which it relates.
Claims
1. A method for obtaining a translation memory process for multiple languages, characterized in that: The method comprises the following steps: 1) Batch download PDF files translated into multiple languages; 2) Use the "Faiss" method in Bitextizer / Neural Document Aligner to identify and match the translated versions of the PDF files; 3) Prepare the data, including converting the PDF file into text blocks according to the visual blocks and removing the headers, footers and unnecessary lines; 4) Implement sentence repair methods, including: a) Train a join model: [only once] train a model (e.g. mBart) to recognize that consecutive lines are part of the same sentence. mBart is a multilingual model, so a single model will be trained for all language pairs. This is the joiner model. The model has two tasks, namely removing line breaks that it deems unnecessary and removing "hyphens" if words are separated at line breaks. The model is trained by using a synthetic dataset of broken sentences as input and "reconstructed" sentences as output. This dataset is created by taking multilingual sentences, splitting them at random positions, and inserting a hyphen where a word is. b) Annotation: For each block in step 3, run the concatenation model to join the lines together to form complete sentences when necessary. For better performance, batch processing uses batches containing multiple text blocks; c) Recover minor edits that were not necessary during the join process by comparing with the original rows; d) Segmentation: for the corresponding language (e.g. sentence segmentation library); E) Use the sentence segmentation library to segment lines containing multiple sentences into individual sentences; 5) Use the alignment model to align sentences from the source and target ends; 6) Perform post-processing including removing duplicate sentences and eliminating wrong alignments using duplexer or other QE evaluation tools.
2. A method for obtaining translation memory for multiple languages according to claim 1, characterized in that: The connection model in the sentence repair method is a multilingual model that is applicable to multiple language pairs and connects lines by removing unnecessary line breaks and hyphens.
3. A method for obtaining translation memory for multiple languages according to claim 1, characterized in that: The sentence repair method also includes a recovery step for recovering unnecessary small edits by comparing the original line and the model output after the connection model processing.
4. A method for obtaining translation memory for multiple languages according to claim 1, characterized in that: The sentence segmentation step uses a rule-based EOL sentence segmenter to separate a line containing multiple sentences into individual sentences.
5. The method for obtaining translation memory for multiple languages according to claim 1 is characterized in that: It also includes the training of the Joiner model: a) Prepare a high-quality multilingual sentence dataset with each entry as a separate row; b) For each training dataset entry, construct an input text block in which some lines have newlines randomly inserted, and construct an output text block in which there are no additional newlines; c) During training, convert all line breaks to <mask> tokens and train the model to recognize when to convert a <mask> token to a line break or remove it.
6. A method for obtaining translation memory for multiple languages according to claim 1, characterized in that: The text blocks of the PDF file are fed into the model in parallel during the concatenation step of the sentence repair method to increase processing speed.
7. A method for obtaining translation memory for multiple languages according to claim 1, characterized in that: The method is applicable to any language pair and can correctly extract complete sentences from PDF without relying on existing PDF text extraction libraries.
8. The method for obtaining translation memory for multiple languages according to claim 1 is characterized in that: The method uses simple and popular tools available on the market, and for specialized texts, the model can be fine-tuned to improve yield.
9. A method for obtaining translation memory for multiple languages according to claim 1, characterized in that: The method is highly scalable and fast and is able to create translation memories (TM) from PDF files.