Methods and systems for multilingual document translation

Abstract

Methods and systems for multilingual document translation are described herein. A method implemented by a server system includes receiving input documents in one or more file formats. Furthermore, the method includes preprocessing the input documents to parse complex expressions present in them. Preprocessing is performed to generate parsed documents. Furthermore, the method includes translating the parsed documents into one or more target languages ​​to generate one or more translated documents.

Description

Technical Field

[0001] The embodiments of this specification generally relate to multilingual document translation, and more particularly, to systems and methods for translating an input document into one or more languages.

Background Art

[0002] Document translation is an important application of machine translation that enables the automatic conversion of a document from one language to another. In today's increasingly interconnected world, the ability to effectively communicate and share information across language barriers has become important for businesses, organizations, and individuals operating on a global scale. Despite significant advances in machine translation technology, existing document translation methods still face a number of challenges that limit their effectiveness, accuracy, and usefulness.

[0003] Conventional automatic translation systems are designed to translate an input sentence from a first language to a second language using a translation dictionary, translation rules, translation patterns, and statistical translation information. However, the translation results of conventional systems are often unnatural and the overall translation quality is low. This is mainly due to the ambiguity in vocabulary, structure, meaning, and style in the translation rules, patterns, or statistical information used by such systems.

[0004] One significant problem with conventional document translation systems is their inability to maintain the original formatting and layout of the source document. For example, specialized and academic documents often rely on complex elements such as tables, graphs, images, formulas, and special symbols to convey information clearly and effectively. If such elements are lost or distorted during the translation process, it can lead to confusion, misunderstanding, and a diminished user experience. Conventional translation methods often focus solely on text and struggle to handle such content elements, resulting in translated documents that lack visual consistency and fail to accurately represent the original content.

[0005] Another significant challenge in traditional document translation is the accurate translation of complex technical terms, domain-specific language, and idiomatic expressions. Documents in specialized fields such as law, medicine, technology, or science contain vocabulary and phrases that require a deep understanding of their subject matter and cultural nuances. Existing machine translation systems, which primarily rely on statistical or rule-based methods, often fail to grasp the precise meaning and context of these terms. This can further lead to inaccurate, confusing, or meaningless translations, posing a significant risk in situations where accuracy is paramount.

[0006] Furthermore, traditional document translation methods often struggle to produce high-quality translations across a wide range of languages. While some conventional systems may function adequately for a limited set of language pairs, they often fail to maintain consistency, fluency, and idiomatic correctness when dealing with diverse target languages. This is particularly problematic for globally operating organizations, as they may require translations in dozens of languages ​​to effectively reach their target markets and communicate with international stakeholders. The lack of a comprehensive, high-quality translation solution can lead to miscommunication, cultural errors, and damage to an organization's reputation.

[0007] Furthermore, the efficiency and speed of document translation are critical factors that present significant challenges for many organizations. Existing methods often involve time-consuming manual work, such as document pre- and post-processing, formatting adjustments, and quality assurance checks. This can lead to significant delays and increased costs, especially when dealing with large volumes of documents or tight deadlines. Organizations often face a trade-off between quality and speed, sacrificing one for the other. This is particularly problematic in industries where both accuracy and timeliness are crucial, such as legal litigation, medical diagnosis, and financial reporting.

[0008] Furthermore, the lack of a rational automated document translation process can hinder an organization's ability to scale its activities and adapt to changing market conditions. As companies expand into new areas and target new customer segments, they need to translate their documents and communications quickly and efficiently to meet the growing needs of their users. However, relying on manual processes and disparate translation tools can create bottlenecks and inefficiencies that slow growth and limit organizational agility. [Overview of the project] [Problems that the invention aims to solve]

[0009] In light of these challenges, there is an urgent need for a sophisticated, comprehensive, and efficient multilingual document translation system. [Means for solving the problem]

[0010] In view of the foregoing, embodiments of this specification provide a method for multilingual document translation. The method includes the step of receiving input documents in one or more file formats by a server system. Furthermore, the method includes the step of preprocessing the input documents by the server system to parse and extract one or more content elements present in the input documents. Parsing is performed based on a parsing algorithm. The preprocessing step is performed to generate a parsed document. Furthermore, the method includes the step of translating the parsed document into one or more target languages ​​by the server system via an implementation of at least one neural machine translation (NMT) engine to generate one or more translated documents. Each translated document corresponds to a machine-translated document in each language.

[0011] In one embodiment, the preprocessing step includes a step in which the server system creates a hierarchical tree structure of the input document. The hierarchical tree structure represents a Document Object Model (DOM) tree. The preprocessing step further includes a step in which the server system parses the hierarchical tree structure to parse and extract one or more content elements along with their associated metadata. Furthermore, the preprocessing step includes a step in which the server system generates a parsed document based on at least the parsing step.

[0012] In one embodiment, during the parsing step, the method includes a step in which the server system identifies one or more content elements present in the input document. Furthermore, the method includes a step in which the server system replaces one or more content elements with one or more predefined placeholders. Each content element is replaced with each predefined placeholder.

[0013] In one embodiment, prior to the translation step, the method includes a step in which a server system optimizes the parsed document based on one or more simplification techniques. The one or more simplification techniques include one of sentence splitting techniques, syntactic restructuring techniques, and lexical normalization techniques. The optimization is performed prior to the translation step.

[0014] In one embodiment, after the translation step, the method includes a step of post-processing one or more translated documents by a server system to preserve one or more content elements of the input document in one or more translated documents.

[0015] In one embodiment, in order to perform a post-processing step, the method includes the steps of a server system traversing a hierarchical tree structure and the server system replacing one or more placeholders with one or more content elements.

[0016] In one embodiment, the post-processing step includes the steps of: having the server system store the translations performed during the translation step as segments in a repository; and having the server system utilize the stored segments to perform the translation step.

[0017] In one embodiment, one or more content elements include at least one of the following: an image, an expression, an inline image, a header and footer literal, a character class, a boundary matcher, a quantifier, a group, and an OR operator.

[0018] Another embodiment of this specification provides a system for multilingual document translation. The system includes an input unit configured to receive input documents in one or more file formats. The system further includes a preprocessing unit configured to preprocess the input documents to parse and extract one or more content elements present in the input documents. The parsing step is performed based on at least a parsing algorithm. The preprocessing step is performed to generate a parsed document. Furthermore, the system includes a translation unit configured to translate the parsed document into one or more target languages ​​to generate one or more translated documents via an implementation of at least one neural machine translation (NMT) engine. Each translated document corresponds to a machine-translated document in each language.

[0019] In one embodiment, to perform the preprocessing step, the preprocessing unit is instructed to create a hierarchical tree structure of the input document. The hierarchical tree structure represents a document object model (DOM) tree. The preprocessing unit is then instructed to parse the hierarchical tree structure to parse and extract one or more content elements along with their associated metadata. Furthermore, the preprocessing unit is instructed to generate a parsed document based on at least the parsing step.

[0020] In one embodiment, during the parsing step, the preprocessing unit is instructed to identify one or more content elements present in the input document. Furthermore, the preprocessing unit is instructed to replace one or more content elements with one or more predefined placeholders. Each content element is replaced with a predefined placeholder.

[0021] In one embodiment, the system includes a simplification engine configured to optimize a parsed document based on one or more simplification techniques. These simplification techniques include one of sentence splitting techniques, syntactic restructuring techniques, and lexical normalization techniques. The optimization step is performed before the translation step.

[0022] In one embodiment, the system includes a post-processing unit configured to post-process one or more translated documents in order to retain one or more content elements of the input documents in one or more translated documents. The post-processing step is performed after the translation step.

[0023] In one embodiment, to perform the post-processing step, the post-processing unit is configured to traverse a hierarchical tree structure and to replace one or more placeholders with one or more content elements.

[0024] In one embodiment, during the post-processing step, the post-processing unit is configured to store the translations performed during the translation step as segments in a repository, and to use the stored segments to perform the translation step.

[0025] In one embodiment, one or more content elements include at least one of the following: an image, an expression, an inline image, a header and footer literal, a character class, a boundary matcher, a quantifier, a group, and an OR operator.

[0026] These and other embodiments of the present invention will be better understood and appreciated when considered together with the following description and accompanying drawings. However, it should be understood that the following description is given as examples and not as an limitation, while showing preferred embodiments and many specific details thereof. Many changes and modifications can be made within the scope of the embodiments herein without departing from the spirit thereof, and the embodiments herein include all such modifications.

[0027] Examples of this specification are better understood from the following detailed description with reference to the drawings.

Brief Description of the Drawings

[0028] [Figure 1] FIG. 1 is a block diagram of a system for performing multilingual document translation according to an embodiment of the present disclosure. [Figure 2] FIG. 2 is a diagram showing an exemplary English mathematics worksheet according to an embodiment of the present disclosure. [Figure 3] FIG. 3 is a diagram showing an exemplary Hindi mathematics worksheet according to an embodiment of the present disclosure.

Modes for Carrying Out the Invention

[0029] Referring to the non-limiting examples shown in the accompanying drawings and detailed in the following description, the examples of this specification and their various features and advantageous details are more fully described. Descriptions of well-known components and processing techniques are omitted so as not to obscure the examples of this specification unnecessarily. The examples used herein are only for facilitating the understanding of how the examples of this specification can be implemented and further enabling those skilled in the art to implement the examples of this specification. Therefore, they should not be construed as limiting the scope of the examples of this specification.

[0030] FIG. 1 shows a block diagram of a system 100 for performing multilingual document translation according to an embodiment of the present disclosure. The system 100 includes an input unit 102, a preprocessing unit 104, a simplification engine 106, a translation unit 108, and a postprocessing unit 110. The system 100 may also include an encoder and a decoder, and an optimizer (not shown in the figure).

[0031] In one embodiment, system 100 is a server system that implements a method for performing multilingual document translation. First, input unit 102 is configured to receive input documents in one or more file formats. In some examples, one or more file formats may include Word documents, Portable Document Format (PDF) documents, image files, video files, spreadsheets, presentations, database files, etc.

[0032] Upon receiving an input document, the preprocessing unit 104 is configured to preprocess the input document to parse and extract one or more content elements present in the input document. Parsing is performed based on at least a parsing algorithm. Content elements include at least one of the following: images, expressions, inline images, header and footer literals, character classes, boundary matchers, quantifiers, groups, OR operators, etc.

[0033] To perform the preprocessing steps, system 100 creates a hierarchical tree structure of the input document. The hierarchical tree structure corresponds to the document object model (DOM) of the input document. System 100 is then configured to parse the hierarchical tree structure to parse and extract one or more content elements along with their associated metadata. Furthermore, system 100 is configured to generate a parsed document based on at least the parsing steps.

[0034] In one implementation, system 100 is configured to intelligently parse the hierarchical tree structure of the input document. Furthermore, system 100 is configured to structure and extract various elements (i.e., content elements) in the input document. Content elements may include images, tables, formulas, and formatting information. In one exemplary implementation, system 100 utilizes open-source libraries to handle the document format and preserve the original layout and styling.

[0035] In one implementation, system 100 is configured to allow a library to read an input document and represent it as a hierarchical tree structure (also called a DOM tree). The DOM tree organizes the content and formatting of the input document as a series of nested elements.

[0036] System 100 is configured to parse the DOM tree. System 100 is configured to identify and extract content elements based on their location and attributes within the hierarchy. For example, while traversing the DOM tree, System 100 is configured to encounter various content elements such as paragraphs, tables, images, and formulas.

[0037] System 100 is configured to treat each such element as a separate chunk. Furthermore, System 100 is configured to identify such individual chunks in the DOM tree along with their formatting information (e.g., font style, color, alignment properties, etc.).

[0038] For example, when system 100 encounters a paragraph element in the DOM tree, system 100 is configured to extract the text content along with its formatting attributes, such as font family, size, color, and line spacing. This ensures that the parsed output maintains the original styling and layout of the paragraph.

[0039] Similarly, when system 100 encounters a table element, system 100 is configured to identify the table structure, including the number of rows and columns, cell content, and cell formatting. The table is extracted as a separate chunk, preserving its original structure and layout. System 100 also handles nested elements within table cells, such as paragraphs or images, by processing them recursively and maintaining their relative positions within the table.

[0040] Similarly, system 100 is configured to identify images and expressions as separate chunks. Furthermore, system 100 is configured to store metadata associated with such chunks (e.g., location, dimensions, etc.).

[0041] In one embodiment, system 100 is configured to identify content elements using byte stream analysis. In another embodiment, system 100 is configured to identify content elements using object detection.

[0042] In one implementation, system 100 is also configured to perform intelligent chunking of the input document based on logical delimiters (such as paragraph boundaries and significant whitespace). System 100 is further configured to parse a DOM tree to identify these structural elements and use them to divide the input document into smaller, manageable chunks.

[0043] After the parsing step, system 100 is configured to output a parsed document. The parsed document is a structured representation of the input document, comprising a series of chunks, each chunk containing extracted text, formatting information, and any other relevant content elements. The structured representation serves as input for subsequent stages of the translation pipeline, such as the simplification engine 106 and at least one neural machine translation (NMT) engine.

[0044] In one implementation, system 100 is configured to perform chunking and parsing as a single, integrated step, achieving a more efficient and accurate analysis of the input document structure. System 100 is configured to handle input documents in one or more formats. Furthermore, system 100 is configured to preserve the original layout and styling of the input document, which leads to a high-quality translation that faithfully represents the input document.

[0045] During the parsing step, system 100 is configured to identify one or more text elements present in the input document. System 100 is then configured to replace one or more text elements with one or more predefined placeholders. Each text element is replaced with each predefined placeholder.

[0046] For example, during the parsing step, system 100 is configured to utilize an encoding algorithm to handle one or more content elements, such as images, figures, and formulas. System 100 is configured to identify such content elements and isolate them from the surrounding text. For example, each unique content element is assigned a separate placeholder in the format "x_1", "x_2", ... "x_i", where i is a natural number representing the total number of content elements in the input document.

[0047] For example, consider a scientific document that includes multiple equations such as "E=mc^2", "F=ma", and "PV=nRT" along with an image showing a graph. System 100 is configured to identify and extract these elements and replace them with placeholders such as "x_1", "x_2", "x_3", and "x_4", respectively. This encoding scheme allows System 100 to process the remaining text independently while preserving the original position and formatting of the content elements.

[0048] It should be noted that the use of the "x_i" encoding scheme offers several advantages. Firstly, it ensures that placeholders are unique, unlikely to appear naturally in text, and can be easily identified and replaced during post-processing. Secondly, neural machine translation (NMT) engines are designed to recognize and retain these placeholders during the translation process. NMT engines treat placeholders as distinct objects within sentences, effectively translating the surrounding text while preserving the placeholders.

[0049] It should be noted that preserving the "x_i" encoding is important to maintain the integrity and accuracy of the translated document. By treating placeholders as immutable objects, system 100 ensures that complex elements are not altered or lost during the translation process, regardless of the target language or the complexity of the translation task.

[0050] In one implementation, system 100 includes a simplification engine 106. The simplification engine 106 is configured to optimize the parsed document based on one or more simplification techniques. One or more simplification techniques include one of sentence splitting techniques, syntactic restructuring techniques, and lexical normalization techniques. Optimization is performed before the translation step.

[0051] Furthermore, the "x_i" encoding scheme is maintained throughout the entire translation pipeline, including the text simplification and translation steps. In one implementation, the simplification engine 106 is configured to employ advanced machine learning techniques to streamline and optimize the text for translation, and is designed to recognize and retain placeholders. This ensures that the simplified text retains the original structure and arrangement of content elements, even when the surrounding text is modified for improved translatability.

[0052] Next, the translation unit 108 is configured to translate the parsed document into one or more target languages ​​in order to generate one or more translated documents via the implementation of at least one neural machine translation (NMT) engine. Each translated document corresponds to a machine-translated document in each language.

[0053] The post-processing unit 110 is configured to post-process one or more translated documents in order to retain one or more content elements of the input documents in one or more translated documents. To perform the post-processing steps, the post-processing unit 110 is configured to traverse a hierarchical tree structure. The post-processing unit 110 is then configured to replace one or more placeholders with one or more content elements again.

[0054] During translation, the post-processing unit 110 is configured to post-process one or more translated documents in order to preserve elements of the input documents in the translated documents. Content elements may include images, formulas, tables, etc.

[0055] In one implementation, at least one NMT engine is trained to seamlessly handle "x_i" placeholders. The NMT engine understands that these placeholders represent distinct content elements and translates the surrounding text accordingly without modifying or omitting those placeholders. This ensures that the translated text maintains the same level of accuracy and fidelity as the input document, with the content elements preserved in their original positions.

[0056] For example, consider the following sentence: "Equation x_1 represents the relationship between energy and mass, as shown in graph x_2." During the simplification and translation processes, the placeholders "x_1" and "x_2" remain unchanged, but the surrounding text is simplified and translated as needed. The resulting translation reads as follows: JPEG2026521489000002.jpg14157 (i.e., a French translation). Please note that the placeholders "x_1" and "x_2" have been preserved exactly as they appear in the original text to ensure the accuracy and completeness of the translated document.

[0057] In short, the "x_i" encoding scheme is a critical component of the parsing and translation process. By replacing content elements with unique placeholders, system 100 can efficiently process text while maintaining the original structure and formatting of the input document. Retaining these placeholders throughout the simplification and translation steps ensures that all content elements remain intact and properly positioned, resulting in a final output that is an accurate and faithful representation of the original document (i.e., the input document).

[0058] After the encoding step (where complex elements such as expressions, symbols, or special characters are replaced with unique placeholders (e.g., "x_1", "x_2", etc.)), the simplification engine 106 is configured to perform a simplification process on the extracted text. The simplification engine 106 employs advanced machine learning techniques (e.g., a finely tuned version of a machine learning model) to streamline and optimize the text for translation while preserving the integrity of the placeholders. The machine learning model is finely tuned on a large, publicly available simplification dataset. The simplification process is a critical subroutine that improves the accuracy and fluency of the final translated output by making the text more suitable for machine translation.

[0059] For example, consider the following complex sentence, which contains mathematical formulas and symbols.

number

[0060] Once the encoding step is performed, system 100 is configured to replace expressions and symbols with placeholders.

[0061] "The Schrödinger equation, given by x_1, is a fundamental equation in quantum mechanics that describes the time-dependent behavior of a quantum mechanical system, and its solution, known as the wave function x_3, contains all the information about the state of that system."

[0062] In the first implementation, the simplification engine 106 is configured to perform sentence decomposition on the description. For example, the simplification engine 106 transforms the sentence into: "The Schrödinger equation is given by x_1. The Schrödinger equation is a fundamental equation in quantum mechanics. It describes the time-dependent behavior of a quantum mechanical system. The solution to the Schrödinger equation is known as the wave function x_3. It contains all the information about the state of that system."

[0063] In the second implementation, the simplification engine 106 is configured to perform syntactic reconstruction on the description. For example, the simplification engine 106 transforms the sentence into: "x_1 is the Schrödinger equation. The Schrödinger equation is a fundamental equation in quantum mechanics. The Schrödinger equation describes the time-dependent behavior of a quantum mechanical system. The solution to the Schrödinger equation is known as the wave function. The wave function is represented by x_3. It contains all the information about the state of the system."

[0064] In the third implementation, the simplification engine 106 is configured to perform lexical normalization on the description. For example, the simplification engine 106 transforms the sentence into: "x_1 represents the Schrödinger equation, which is a fundamental equation in quantum mechanics. It describes the time-dependent behavior of a quantum mechanical system. The solution to the Schrödinger equation is called the wave function x_3. They contain all the information about the conditions of that system."

[0065] In this way, the simplification engine 106 is configured to transform complex, formulaic sentences into a simpler, more translatable format while retaining placeholders. The simplified text is easier for the NMT engine to process because complex mathematical formulas and symbols are replaced with placeholders and the sentence structure is streamlined.

[0066] In another example, consider the following sentence: "The reaction between H2SO4 and NaOH produces Na2SO4 and H2O, as described by the balanced chemical equation: H2SO4 + 2NaOH → Na2SO4 + 2H2O, which proves the law of conservation of mass."

[0067] After the encoding step, system 100 is configured to output: "The reaction of x_1 and x_2 produces x_3 and x_4, as described by the equilibrium chemical equation: x_5, which proves the law of conservation of mass." After the simplification step, system 100 is configured to output: "x_1 reacts with x_2 to produce x_3 and x_4. The equilibrium chemical equation x_5 describes this reaction. x_5 demonstrates the law of conservation of mass."

[0068] It should be noted that the simplification engine 106 is capable of handling complex technical content while maintaining the integrity of placeholders. By simplifying the text and making it more suitable for machine translation, the simplification engine 106 significantly improves the quality and accuracy of translations produced by subsequent NMT engines. The simplification engine 106 ensures that the translated text is faithful to the original content, including formulas, symbols, and equations, and is fluent in the target language.

[0069] Once simplification is complete, the NMT engine is configured to parse and translate the simplified text, with placeholders intact. In one example, the NMT engine may utilize a deep learning architecture (e.g., a transformer model) to perform the translation. Generally, transformers rely entirely on self-attention mechanisms to grasp the dependencies between words in a sentence. This allows the NMT engine to parallelize computations and process the input sequence as a whole rather than sequentially, which leads to faster training times and improved translation quality.

[0070] Please note that the translation step is performed after the simplification step.

[0071] The NMT engine is trained on one or more multilingual datasets. For example, an NMT engine might be trained on a dataset of European languages ​​and another on a dataset of Indian languages. These datasets contain millions of parallel sentences across multiple language pairs, allowing the model to learn the intricacies and nuances of language translation.

[0072] By exposing the model to a wide variety of texts, ranging from news articles and government documents to literary works and social media posts, the NMT engine develops a deep understanding of the language structures, idioms, and cultural contexts specific to each language.

[0073] In one implementation, during translation, the NMT engine employs techniques such as byte-pair encoding (BPE) and subword regularization to handle out-of-vocabulary words and reduce computational complexity. Generally, BPE is a data compression algorithm that iteratively replaces the most frequent pairs of bytes in a sequence with a single unused byte, effectively creating a vocabulary in subword units. This allows the NMT model to break down rare or unseen words into smaller, more manageable fragments, improving the NMT model's ability to produce accurate translations.

[0074] In general, subword regularization is a technique that probabilistically segments words into smaller units during training, helping NMT models learn more robust and generalizable representations of language. By introducing noise into the segmentation process, subword regularization prevents NMT models from overfitting to specific word forms and allows them to better handle variations and misspellings in the input text.

[0075] In one implementation, the NMT engine also incorporates advanced techniques such as multi-head attention, residual connection, and layer normalization to further improve translation quality and fluency. For example, multi-head attention allows the NMT model to pay attention to different parts of the input sequence simultaneously and grasp multiple types of dependencies and relationships between words. Furthermore, residual connection helps mitigate the vanishing gradient problem and enables deeper network training, while layer normalization stabilizes the training process and promotes faster convergence.

[0076] The NMT engine takes into account the context and semantics of the source text (i.e., the input document) to produce translations that sound natural and consistent in the target language, even when dealing with complex sentence structures or idiomatic expressions.

[0077] For example, consider a simplified sentence with placeholders: "The cat represented by x_1 is sitting on the mat indicated by x_2." When passed through the NMT engine, the NMT engine produces the following fluent and idiomatic translation into French: JPEG2026521489000004.jpg11115 This translation not only accurately conveys the meaning of the original sentence, but also preserves placeholders and maintains the natural flow of the language.

[0078] By leveraging advanced deep learning architectures, large-scale multilingual datasets, and state-of-the-art techniques such as transformer models, byte-pair coding, and subword regularization, the NMT engine can generate translations that accurately capture the meaning, context, and nuances of the source text while preserving the integrity of placeholders introduced during the coding and simplification steps.

[0079] Once the translation process is complete, system 100 performs a post-processing step to reintegrate the previously extracted content elements into the translated document. This decryption ensures that the final output maintains the visual structure and integrity of the original document while seamlessly incorporating the translated text.

[0080] In some examples, one or more target languages ​​include Indian languages ​​such as Assamese (as), Hindi (hi), Marathi (mr), Tamil (ta), Bengali (bn), Kannada (kn), Odiya (or), Telugu (te), Gujarati (gu), Malayalam (ml), and Punjabi (pa). In some examples, one or more target languages ​​include foreign languages ​​such as French, German, Dutch, Spanish, Chinese, Arabic, Japanese, and Russian.

[0081] In one embodiment, system 100 may include a processor and memory for storing instructions (not shown in the figure). When the processor executes an instruction, it causes system 100 to perform a method for performing multilingual document translation. As used herein, “processor” or “processing unit” includes one or more processors, and processor refers to any logic circuit for processing instructions. A processor may be a general-purpose processor, a dedicated processor, a conventional processor, a digital signal processor, a plurality of microprocessors, one or more microprocessors associated with a DSP core, a controller, a microcontroller, a low-end microcontroller, an application-specific integrated circuit, a field-programmable gate array circuit, or any other type of integrated circuit. A processor may perform signal coding data processing, input / output processing, and / or any other functions that enable the operation of system 100 as disclosed herein. More specifically, a processor or processing unit is a hardware processor.

[0082] As used herein, “database,” “memory unit,” “storage unit,” and / or “memory” refer to a machine or computer-readable medium that includes any mechanism for storing information in a format readable by a computer or similar machine. For example, computer-readable medium includes read-only memory ("ROM"), random access memory ("RAM"), magnetic disk storage media, optical storage media, flash memory devices, or other types of machine-accessible storage media.

[0083] Also disclosed is a method for multilingual document translation (i.e., translating one or more input documents into one or more languages). The method includes the step of receiving one or more input documents in file formats by a server system (e.g., system 100). The method further includes the step of preprocessing the input documents by the server system to parse and extract one or more content elements present in the input documents. Parsing is performed based on at least a parsing algorithm. The preprocessing step is performed to generate a parsed document.

[0084] In one embodiment, the preprocessing unit 104 preprocesses the input document. The preprocessing unit 104 parses content elements from the input document (including, but not limited to, images, expressions, inline images, headers, footers, etc.). Content elements may further include literals, character classes, boundary matchers, quantifiers, groups, and OR operators. The preprocessing unit 104 parses the input document so that non-text attributes are handled simultaneously, thereby improving translation quality.

[0085] Therefore, system 100 retains most of the format and layout of the input document and / or file, and thus human translators do not need to spend time replacing images, restoring font styles, or reconstructing paragraphs in the input document. In one embodiment, the input document may include a batch of documents.

[0086] During the parsing of the input document, non-text elements such as images, figures, and formulas are identified and replaced with unique placeholders such as "x_1", "x_2", ..., "x_i". Encoding the input document with such placeholders serves two essential purposes: (1) it allows system 100 to process text independently of non-text elements, facilitating the simplification and translation process; and (2) the unique nature of the placeholders ensures that they can be easily identified during the decoding stage and replaced with actual text without ambiguity or confusion.

[0087] It should be noted that the choice to use "x" followed by an underscore and a numerical identifier (e.g., x_1) is intentional, as it minimizes the likelihood of these placeholders occurring naturally in the text. This encoding scheme also adapts to a wide range of languages ​​and writing systems, making it suitable for multilingual document translation.

[0088] This method includes the step of translating a parsed document into one or more target languages ​​in order for a server system to generate one or more translated documents via an implementation of at least one neural machine translation (NMT) engine. Each translated document corresponds to a machine-translated document in each language.

[0089] It should be noted that System 100 translates input documents and / or files received in different formats, including, but not limited to, Word documents, PDF files, PowerPoint presentations, Excel spreadsheets, Outlook files, etc. The parsed documents are passed to Translation Unit 108.

[0090] The preprocessing step includes a step by the server system to create a hierarchical tree structure of the input document. The hierarchical tree structure represents a Document Object Model (DOM) tree. The preprocessing step further includes a step by the server system to parse the hierarchical tree structure in order to parse and extract one or more content elements along with their associated metadata. Furthermore, the preprocessing step includes a step by the server system to generate a parsed document based on at least the parsing step.

[0091] During the parsing step, the method includes a step in which the server system identifies one or more content elements present in the input document. The method also includes a step in which the server system replaces one or more content elements with one or more predefined placeholders. Each content element is replaced with each predefined placeholder.

[0092] Prior to the step of translating the parsed document, this method includes a step in which the server system optimizes the parsed document based on one or more simplification techniques. One or more simplification techniques include one of sentence splitting techniques, syntactic restructuring techniques, and lexical normalization techniques.

[0093] The simplification and translation processes are designed to maintain the integrity of placeholders throughout each of their respective processes. During the simplification step, the system employs advanced machine learning techniques, such as fine-tuned machine learning models, to streamline and optimize the text for translation. The simplification step includes techniques such as sentence splitting, syntactic restructuring, and lexical normalization, aimed at improving the readability and translatability of the text while preserving placeholders in their original positions.

[0094] NMT models utilize techniques such as multi-head attention and subword regularization to enable them to process text while preserving placeholders, ensuring that the translated output maintains the same placeholder structure and order as the original input.

[0095] In one implementation, system 100 utilizes a parsing algorithm to analyze the structure of an input document and extract its content and formatting information. The parsing process creates a hierarchical representation of the input document, where each content element (e.g., paragraphs, tables, images, and expressions) is represented as a node in an XML tree.

[0096] During the decryption or post-processing step, system 100 traverses the XML tree and replaces placeholders in the translated text with their corresponding content elements. The hierarchical structure of the XML tree ensures that these elements are reintegrated in the correct locations within the document, maintaining the original layout and formatting.

[0097] For example, consider a placeholder "x_1" representing an expression in the original document. During the parsing stage, the expression is extracted and its location in the XML tree is recorded while it is replaced with "x_1". After the translation process, system 100 identifies the location of the "x_1" placeholder in the translated text and replaces it with the original expression based on its location in the XML tree. This process ensures that the expression is reintegrated into the translated document in the exact same location as in the original document, preserving the visual structure and integrity of the document.

[0098] Furthermore, XML tree parsing enables the system to handle complex formatting and layout elements, such as tables, headers, footers, and nested elements, with high accuracy. By maintaining the document's hierarchical structure throughout the translation process, the decryption method can accurately reconstruct the original formatting and layout, even when dealing with intricate designs and multiple levels of nesting.

[0099] Following the translation step, the method includes a step in which the server system post-processes one or more translated documents to retain one or more content elements of the input documents in one or more translated documents. To perform the post-processing step, the method includes a step in which the server system traverses a hierarchical tree structure. The method further includes a step in which the server system replaces one or more placeholders with one or more content elements.

[0100] When the translation unit 108 translates the input document into the required target language, the post-processing unit 110 post-processes one or more translated documents to preserve content elements (e.g., images, formulas, etc.). Translation can be performed without a network or internet connection, i.e., offline. Because the system 100 does not require an internet connection to operate, the system 100 provides high security for highly secure documents. Although the system 100 performs translation offline, it achieves highly accurate translated documents.

[0101] During the post-processing step, the method includes a step in which the server system stores the translations performed during the translation step as segments in a repository (for example, a database (not shown in the diagram)). The method also includes a step in which the server system uses the stored segments to perform translation steps in the future.

[0102] One or more translated documents received from System 100 are high-quality translated documents that guarantee all linguistic and cultural differences. Furthermore, System 100 accepts multiple file formats. A commendable feature is its ability to accept various file formats and filter content. System 100 can translate a single file, a batch of files, an email, or a paragraph of text. According to one embodiment, System 100 learns words and phrases while editing translations. System 100 stores translations as segments in a bilingual repository. These segments are used for future translations of documents, thus avoiding the need to translate the same word or phrase twice. Some examples of such include spare parts catalogs, organizational codes of conduct, and employment contracts. This is particularly useful for organizations that use specialized vocabulary.

[0103] System 100 may also be used in any of the following broad sectors, but is not limited to: banking and financial services, corporations, e-commerce, education, energy and utilities, gaming services, IT company software, legal services, life sciences and healthcare services, literature and publishing, manufacturing and automotive, marketing and advertising, media and entertainment, pharmaceuticals, retail, science, travel and hospitality.

[0104] Furthermore, System 100 is a multilingual document translation system. System 100 can be an Indian language document translator that translates input documents into 12 Indian languages, including, but not limited to, Assamese (as), Hindi (hi), Marathi (mr), Tamil (ta), Bengali (bn), Kannada (kn), Odiya (or), Telugu (te), Gujarati (gu), Malayalam (ml), and Punjabi (pa). In one implementation, System 100 is trained on at least the Samantar dataset, which is the largest publicly available set of Indian languages. System 100 can be used to translate any English printed document, web page, or test paper into 12 Indian regional languages ​​with high accuracy and high security.

[0105] In some embodiments, system 100 can be used in fields such as artificial intelligence (AI), neural machine translation (NMT), document translation, web page translation, and print translation.

[0106] Accordingly, a system 100 for translating one or more input documents into one or more languages ​​is provided herein, the system 100 preprocesses the input documents to parse at least one of content elements, including, but not limited to, images, formulas, etc., passes the parsed documents to a translation unit and a post-processing engine, the post-processing engine post-processes the translated documents to retain the elements (e.g., images, formulas, etc.).

[0107] In one embodiment, one or more input documents are translated into one or more languages, and the system 100 may be able to operate without an internet connection.

[0108] Figure 2 shows an exemplary English mathematics problem sheet 200 according to one embodiment of the present disclosure.

[0109] Figure 3 shows an exemplary Hindi mathematics problem sheet 300 according to one embodiment of the present invention. Problem sheet 200 is converted to problem sheet 300 based on an implementation of a multilingual document translation method carried out by system 100.

[0110] As shown in Figure 2, the problem sheets are in English, but they have been translated into Hindi (as shown in Figure 3). Note that despite the language translation process, the formulas and numbers in both problem sheets 200 and 300 remain unchanged. Note that System 100 translates mathematical problem sheets from one language to another without changing the numbers in those problem sheets.

[0111] The above descriptions of specific embodiments are so sufficient to reveal the general nature of the embodiments herein that others may readily modify and / or adapt such specific embodiments for various applications without deviating from the general concept by applying their current knowledge, and such adaptations and modifications should and shall be understood within the meaning and scope of the equivalents of the disclosed embodiments. It should be understood that the phrasing or terminology used herein is for illustrative purposes only and not limitation. Thus, although the embodiments herein have been described in relation to preferred embodiments, those skilled in the art will recognize that the embodiments herein can be carried out by modifications within the spirit and scope of the appended claims.

Claims

1. A method for translating multilingual documents, The server system receives input documents in one or more file formats, A server system preprocesses an input document to parse and extract one or more content elements present in the input document, wherein the parsing is performed based on at least a parsing algorithm, and the preprocessing step is performed to generate a parsed document. The server system, through the implementation of at least one neural machine translation (NMT) engine, translates the parsed document into one or more target languages ​​in order to generate one or more translated documents, wherein each translated document corresponds to a machine-translated document in each language. Methods that include...

2. The aforementioned pre-processing step is: The server system performs a step of creating a hierarchical tree structure of the input documents, wherein the hierarchical tree structure represents a document object model (DOM) tree. The server system parses the hierarchical tree structure in order to parse and extract one or more content elements along with their associated metadata, The server system performs at least the step of generating the parsed document based on the analysis step and The method according to claim 1, including the method described in claim 1.

3. During the parsing step, the method The server system includes the step of identifying one or more content elements present in the input document, The server system performs a step of replacing one or more content elements with one or more predefined placeholders, wherein each content element is replaced with each predefined placeholder. The method according to claim 2, including the method described in claim 2.

4. Before translating the parsed document, the method, The server system optimizes the parsed document based on one or more simplification techniques, wherein the one or more simplification techniques include one of sentence splitting techniques, syntactic restructuring techniques, and lexical normalization techniques. The method according to claim 1, including the method described in claim 1.

5. After the translation step, the method The server system performs a post-processing step of the one or more translated documents in order to preserve the one or more content elements of the input document in the one or more translated documents. The method according to claim 1, including the method described in claim 1.

6. In order to carry out the post-processing step described above, the method The server system performs the step of traversing a hierarchical tree structure, The server system then performs the step of replacing the one or more placeholders with the one or more content elements. The method according to claim 5, including the method described in claim 5.

7. During the post-processing step, the method The server system stores the translations performed during the translation step as segments in a repository. The server system uses the stored segments to perform the translation step. The method according to claim 5, including the method described in claim 5.

8. The method according to claim 1, wherein the one or more content elements include at least one of the following: an image, an expression, an inline image, a header and footer literal, a character class, a bounding matcher, a quantifier, a group, and an OR operator.

9. A system for translating multilingual documents, An input unit configured to receive input documents in one or more file formats, A preprocessing unit configured to preprocess an input document for parsing and extracting one or more content elements present in the input document, wherein the parsing step is performed based on at least a parsing algorithm, and the preprocessing step is performed to generate a parsed document; Translating a parsed document into one or more target languages ​​to generate one or more translated documents via an implementation of at least one neural machine translation (NMT) engine, wherein each translated document corresponds to a machine-translated document in each language. A system equipped with these features.

10. In order to carry out the aforementioned preprocessing step, the system Creating a hierarchical tree structure of the input documents, wherein the hierarchical tree structure represents a document object model (DOM) tree. Parsing the hierarchical tree structure in order to parse and extract one or more of the aforementioned content elements along with their associated metadata, The parsed document is generated based on at least the analysis step. The system according to claim 9, which causes the following to be performed.

11. During the parsing step, the preprocessing unit, Identifying one or more content elements present in the input document, Replacing one or more content elements with one or more predefined placeholders, wherein each content element is replaced by each predefined placeholder. The system according to claim 10, which causes the following to be performed.

12. A simplification engine configured to optimize the parsed document based on one or more simplification techniques, wherein the one or more simplification techniques include one of sentence splitting techniques, syntactic restructuring techniques, and lexical normalization techniques, and the optimization step is performed before the translation step. The system according to claim 9, comprising:

13. A post-processing unit configured to post-process one or more translated documents in order to retain one or more content elements of the input document in the one or more translated documents, wherein the post-processing step is performed after the translation step. The system according to claim 9, comprising:

14. In order to carry out the aforementioned post-processing step, the post-processing unit, Traversing a hierarchical tree structure, Replacing the one or more placeholders with the one or more content elements again The system according to claim 13, configured to perform the following:

15. During the post-processing step, the post-processing unit The translations performed during the aforementioned translation step are stored in the repository as segments, To perform the aforementioned translation step, the stored segment is used. The system according to claim 13, configured to perform the following:

16. The system according to claim 9, wherein the one or more content elements include at least one of an image, an expression, an inline image, a header and footer literal, a character class, a boundary matcher, a quantifier, a group, and an OR operator.

Images