Methods and systems for multilingual document translation

Abstract

Described herein are methods and systems for multilingual document translation. The method, performed by a server system, includes receiving an input document in one or more file formats. In addition, the method includes pre-processing the input document to parse complex expressions present in the input document. The pre-processing is performed to generate a parsed document. Further, the method includes translating the parsed document in one or more target languages to generate one or more translated documents.

Description

METHODS AND SYSTEMS FOR MULTILINGUAL DOCUMENT TRANSLATIONTECHNICAL FIELD

[0001] The embodiments herein generally relate to multilingual document translation, and more particularly, relate to a system and a method for translating an input document into one or more languages.BACKGROUND

[0002] Document translation is an essential application of machine translation, enabling the automatic conversion of documents from one language to another. In today's increasingly interconnected world, the ability to effectively communicate and share information across language banners has become crucial for businesses, organizations, and individuals operating on a global scale. Despite significant advancements in machine translation technology, existing document translation methods still face numerous challenges that limit their effectiveness, accuracy, and usability.

[0003] The conventional automatic translation system is designed to translate input sentences from a first language into a second language using translation dictionaries, translation rules, translation patterns, and statistical translation information. However, the translation results of conventional systems are often unnatural, and the overall translation quality is low. This is primarily due to the ambiguity in vocabulary, structure, meaning, and style in the translation rules, patterns, or statistical information used by such systems.

[0004] One significant issue with conventional document translation systems is their inability to maintain the original formatting and layout of the source document. For example, professional and academic documents often rely on complex elements such as tables, graphs, images, equations, and special symbols to convey information clearly and effectively. Any loss or distortion of such components during the translation process can lead to confusion, misinterpretation, and a diminished user experience. Conventional translation methods, whichoften focus solely on the text, struggle to handle such content elements, resulting in translated documents that lack visual coherence and fail to accurately represent the original content.

[0005] Another major challenge in conventional document translation is the accurate translation of complex terminology, domain- specific language, and idiomatic expressions. Documents in specialized fields such as legal, medical, technical, or scientific domains contain vocabulary and phrases that require a deep understanding of the subject matter and cultural nuances. Existing machine translation systems, which primarily rely on statistical or rule-based approaches, often fail to capture the precise meaning and context of these terms. This can further lead to translations that are inaccurate, misleading, or nonsensical, posing significant risks in situations where accuracy is paramount.

[0006] Moreover, conventional document translation methods often struggle to produce high-quality translations across a wide range of languages. While some conventional systems may perform adequately for a limited set of language pairs, they often fail to maintain consistency, fluency, and idiomatic correctness when dealing with diverse target languages. This is particularly problematic for organizations operating globally, as they may require translations in dozens of languages to effectively reach their target markets and communicate with international stakeholders. The lack of a comprehensive, high-quality translation solution can lead to miscommunication, cultural faux pas, and damage to an organization's reputation.

[0007] Moreover, the efficiency and speed of document translation are also critical factors that pose significant challenges for many organizations. Existing methods often involve timeconsuming manual work, such as pre- and post-processing of documents, formatting adjustments, quality assurance checks, and so on. This can lead to significant delays and increased costs, especially when dealing with large volumes of documents or tight deadlines. Organizations often face a trade-off between quality and speed, sacrificing one for the other. This is particularly problematic in industries where both accuracy and timeliness are essential, such as legal proceedings, medical diagnoses, financial reporting, and the like.

[0008] Also, the lack of a streamlined, automated document translation process can hinder an organization's ability to scale its operations and adapt to changing market conditions. As businesses expand into new regions and target new demographics, they need to quickly andefficiently translate their documents and communications to meet the needs of their growing audience. However, reliance on manual processes and disparate translation tools can create bottlenecks and inefficiencies that slow down growth and limit an organization's agility.

[0009] In light of these challenges, there is an urgent need for an advanced, comprehensive, and efficient multilingual document translation system.SUMMARY

[0010] In view of the foregoing, an embodiment herein provides a method for multilingual document translation. The method includes receiving, by a server system, an input document in one or more file formats. In addition, the method includes pre-processing, by the server system, the input document for parsing and extracting one or more content elements present in the input document. The parsing is performed based on a parsing algorithm. The pre-processing is performed to generate a parsed document. Further, the method includes translating, by the server system via implementation of at least one neural machine translation (NMT) engine, the parsed document in one or more target languages to generate one or more translated documents. Each translated document corresponds to machine-translated document in each language.

[0011] In an embodiment, the pre-processing comprises: creating, by the server system, a hierarchical tree structure of the input document. The hierarchical tree structure represents a Document Object Model (DOM) tree. The pre-processing further comprises analyzing, by the server system, the hierarchical tree structure for parsing and extracting the one or more content elements along with their associated metadata. Furthermore, the pre-processing comprises generating, by the server system, the parsed document based at least on the analyzing step.

[0012] In an embodiment, during the parsing step, the method comprises: identifying, by the server system, the one or more content elements present in the input document. Moreover, the method comprises replacing, by the server system, the one or more content elements with one or more pre-defined placeholders. Each content element is replaced with each pre-defined placeholder.

[0013] In an embodiment, prior to translating step, the method comprises; optimizing, by the server system, the parsed document based on one or more simplification techniques. The one or more simplification techniques comprises one of: sentence split technique, syntactic restructure technique, and vocabulary normalize technique. The optimization is performed prior to the translating step.

[0014] In an embodiment, subsequent to the translating step, the method comprises: postprocessing, by the server system, the one or more translated documents to preserve the one or more content elements of the input document in the one or more translated documents.

[0015] In an embodiment, for performing the post-processing, the method comprises: traversing, by the server system, the hierarchical tree structure; and re-replacing, by the server system, the one or more placeholders with the one or more content elements.

[0016] In an embodiment, during the post-processing, the method comprises: storing, by the server system, translations performed during the translating step as segments, in a repository; and utilizing, by the server system, the stored segments to perform the translating step.

[0017] In an embodiment, the one or more content elements comprise at least one of images, equations, inline images, headers and footers literals, character classes, boundary matchers, quantifiers, groups and OR operator.

[0018] Another embodiment herein provides a system for multilingual document translation. The system includes an input unit configured to receive an input document in one or more file formats. The system further includes a pre-processing unit configured to pre- process the input document to parse and extract one or more content elements present in the input document. The parse step is performed based at least on a parsing algorithm. The pre- process step is performed to generate a parsed document. Furthermore, the system includes a translating unit configured to translate, via implementation of at least one neural machine translation (NMT) engine, the parsed document in one or more target languages to generate one or more translated documents. Each translated document corresponds to machine- translated document in each language.

[0019] In an embodiment, to perform the pre-process step, the pre-processing unit is caused to create a hierarchical tree structure of the input document. The hierarchical tree structure represents a Document Object Model (DOM) tree. The pre-processing unit is then caused to analyze the hierarchical tree structure to parse and extract the one or more content elements along with their associated metadata. Furthermore, the pre-processing unit is caused to generate the parsed document based at least on the analyze step.

[0020] In an embodiment, during the parse step, the pre-processing unit is caused to identify the one or more content elements present in the input document. Moreover, the pre-processing unit is caused to replace the one or more content elements with one or more pre-defined placeholders. Each content element is replaced with each pre-defined placeholder.

[0021] In an embodiment, the system comprises a simplification engine configured to optimize the parsed document based on one or more simplification techniques. The one or more simplification techniques comprises one of: sentence split technique, syntactic restructure technique, and vocabulary normalize technique. The optimize step is performed prior to the translate step.

[0022] In an embodiment, the system comprises a post-processing unit configured to postprocess the one or more translated documents to preserve the one or more content elements of the input document in the one or more translated documents. The post-process step is performed subsequent to the translate step.

[0023] In an embodiment, to perform the post-process step, the post-processing unit is configured to traverse the hierarchical tree structure; and re-replace the one or more placeholders with the one or more content elements.

[0024] In an embodiment, during the post-process step, the post-processing unit is configured to store translations performed during the translate step as segments, in a repository; and utilize the stored segments to perform the translate step.

[0025] In an embodiment, the one or more content elements include at least one of images, equations, inline images, headers and footers literals, character classes, boundary matchers, quantifiers, groups and OR operator.

[0026] These and other aspects of the embodiments herein will be better appreciated and understood when considered in conjunction with the following description and the accompanying drawings. It should be understood, however, that the following descriptions, while indicating preferred embodiments and numerous specific details thereof, arc given by way of illustration and not of limitation. Many changes and modifications may be made within the scope of the embodiments herein without departing from the spirit thereof, and the embodiments herein include all such modifications.BRIEF DESCRIPTION OF THE DRAWINGS

[0027] The embodiments herein will be better understood from the following detailed description with reference to the drawings, in which:

[0028] FIG. 1 illustrates a block diagram of a system for performing multilingual document translation, in accordance with an embodiment of the present disclosure;

[0029] FIG. 2 illustrates an exemplary mathematics question paper in English language, in accordance with an embodiment of the present disclosure; and

[0030] FIG. 3 illustrates an exemplary mathematics question paper in Hindi language, in accordance with an embodiment of the present disclosure.DETAILED DESCRIPTION OF PREFERRED EMBODIMENTS

[0031] The embodiments herein and the various features and advantageous details thereof are explained more fully with reference to the non-limiting embodiments that are illustrated in the accompanying drawings and detailed in the following description. Descriptions of well- known components and processing techniques are omitted so as to not unnecessarily obscure the embodiments herein. The examples used herein are intended merely to facilitate an understanding of ways in which the embodiments herein may be practiced and to further enable those of skill in the ail to practice the embodiments herein. Accordingly, the examples should not be construed as limiting the scope of the embodiments herein.

[0032] FIG. 1 illustrates a block diagram of a system 100 for performing multilingual document translation, in accordance with an embodiment of the present disclosure. The system100 includes an input unit 102, a pre-processing unit 104, a simplification engine 106, a translating unit 108, and a post-processing unit 110. The system 100 may also include an encoder and a decoder, and an optimizer (not shown in figures).

[0033] In an embodiment, the system 100 is a server system that performs a method for performing multilingual document translation. Initially, the input unit 102 is configured to receive an input document in one or more file formats. In some examples, the one or more file formats may include a word document, a portable document format (pdf) document, an image file, a video file, a spreadsheet, a presentation, a database file, and the like.

[0034] Upon receiving the input document, the pre-processing unit 104 is configured to pre- process the input document to parse and extract one or more content elements present in the input document. The parsing is performed based at least on a parsing algorithm. The content elements include at least one of images, equations, inline images, headers and footers literals, character classes, boundary matchers, quantifiers, groups, OR operator, and the like.

[0035] To perform the pre-process step, the system 100 creates a hierarchical tree structure of the input document. The hierarchical tree structure corresponds to a Document Object Model (DOM) of the input document. The system 100 is then configured to analyze the hierarchical tree structure to parse and extract the one or more content elements along with their associated metadata. Further, the system 100 is configured to generate the parsed document based at least on the analyze step.

[0036] In one implementation, the system 100 is configured to intelligently analyze the hierarchical tree structure of the input document. In addition, the system 100 is configured to structure and extract various elements (i.e., content elements) in the input document. The content elements may include images, tables, equations, and formatting information. In one exemplary implementation, the system 100 utilizes an open-source library to handle document formats and preserve the original layout and styling.

[0037] In one implementation, the system 100 is configured to enable the library to read the input document and represent it as the hierarchical tree structure (also referred to as DOM tree). The DOM tree organizes the content and formatting of the input document as a series of nested elements.

[0038] The system 100 is configured to analyze the DOM tree. The system 100 is configured to identify and extract the content elements based on their position and attributes within the hierarchy. For example, during traversal of the DOM tree, the system 100 is configured to encounter various content elements such as paragraphs, tables, images, equations, and the like.

[0039] The system 100 is configured to treat each such element as a distinct chunk. In addition, the system 100 is configured to identify such discrete chunks in the DOM tree along with their formatting information (e.g., font style, color, alignment properties, etc.).

[0040] For instance, when the system 100 encounters a paragraph element in the DOM tree, the system 100 is configured to extract the text content along with its formatting attributes, such as font family, size, color, and line spacing. This ensures that the parsed output maintains the original styling and layout of the paragraph.

[0041] In a similar manner, when the system 100 encounters a table element, the system 100 is configured to identify the table structure, including the number of rows and columns, cell contents, and cell formatting. The table is extracted as a separate chunk, preserving its original structure and layout. The system 100 also handles nested elements within table cells, such as paragraphs or images, by recursively processing them and maintaining their relative positions within the table.

[0042] In a similar manner, the system 100 is configured to identify the images and equations as distinct chunks. In addition, the system 100 is configured to store the metadata (e.g., position, dimensions, etc.) associated with such chunks.

[0043] In an embodiment, the system 100 is configured to identify the content elements using byte-stream analysis. In another embodiment, the system 100 is configured to identify the content elements using object detection.

[0044] In one implementation, the system 100 is also configured to perform intelligent chunking of the input document based on logical breaks (such as paragraph boundaries, significant whitespace, etc.). The system 100 is further configured to analyze the DOM tree to identify these structural elements and uses them to split the input document into smaller, manageable chunks.

[0045] After the parse step, the system 100 is configured to output the parsed document. The parsed document is the structured representation of the input document, including a series of chunks, wherein each chunk includes the extracted text, formatting information, and any other associated content elements. The structured representation serves as the input for the subsequent stages of the translation pipeline, such as the simplification engine 106 and at least one neural machine translation (NMT) engine.

[0046] In one implementation, the system 100 is configured to perform the chunking and parsing as a single, integrated step, achieving a more efficient and accurate analysis of the input document structure. The system 100 is configured to handle the input document in the one or more formats. In addition, the system 100 is configured to preserve the original layout and styling of the input document, leading to quality translations that faithfully represent the input document.

[0047] During the parse step, the system 100 is configured to identify the one or more textual elements present in the input document. The system 100 is then configured to replace the one or more textual elements with one or more pre-defined placeholders. Each textual element is replaced with each pre-defined placeholder.

[0048] For example, during the parse step, the system 100 is configured to utilize an encoding algorithm to handle the one or more content elements such as images, diagrams, and equations. The system 100 is configured to identify and isolate such content elements from the surrounding text. For example, each unique content element is assigned a distinct placeholder in the format of “x_l”, “x_2”, .... “x_i”, wherein i is a natural number that represents the total number of the content elements in the input document.

[0049] For example, consider a scientific document containing multiple equations, such as "E = mcA2", "F = ma”, and "PV = nRT", along with an image depicting a graph. The system 100 is configured to identify and extract these elements, replacing them with placeholders like "x_l", "x_2", "x_3", and "x_4", respectively. This encoding scheme allows the system 100 to process the remaining text independently while preserving the original position and formatting of the content elements.

[0050] It is noted that the use of "x_i" encoding scheme offers several advantages. First, the placeholders are unique and unlikely to appear naturally in the text, ensuring that they can be easily identified and replaced during the post-process stage. Second, Neural Machine Translation (NMT) engines are designed to recognize and preserve these placeholders during the translation process. The NMT engines treat the placeholders as distinct objects within the sentence, effectively translating the surrounding text while leaving the placeholders intact.

[0051] It is noted that the preservation of the "x_i" encoding is crucial for maintaining the integrity and accuracy of the translated document. By treating the placeholders as immutable objects, the system 100 ensures that the complex elements are not altered or lost during the translation process, regardless of the target language or the complexity of the translation task.

[0052] In one implementation, the system 100 includes the simplification engine 106. The simplification engine 106 is configured to optimize the parsed document based on one or more simplification techniques. The one or more simplification techniques include one of: sentence split technique, syntactic restructure technique, and vocabulary normalize technique. The optimization is performed prior to the translation step.

[0053] Moreover, the "x_i" encoding scheme is maintained throughout the entire translation pipeline, including the text simplification and translation steps. In one implementation, the simplification engine 106 is configured to employ advanced machine learning techniques to streamline and optimize the text for translation, is designed to recognize and preserve the placeholders. This ensures that the simplified text retains the original structure and positioning of the content elements, even as the surrounding text is modified for improved translatability.

[0054] Then, the translating unit 108 is configured to translate, via implementation of at least one neural machine translation (NMT) engine, the parsed document in one or more target languages to generate one or more translated documents. Each translated document corresponds to machine-translated document in each language.

[0055] The post-processing unit 110 is configured to post-process the one or more translated documents to preserve the one or more content elements of the input document in the one or more translated documents. To perform the post-process step, the post-processing unit 110 isconfigured to traverse the hierarchical tree structure. The post-processing unit 110 is then configured to re-replace the one or more placeholders with the one or more content elements.

[0056] Upon translation, the post-processing unit 110 is configured to post-process the one or more translated documents to preserve elements of the input document in the translated document. The content elements may include images, equations, tables, and the like.

[0057] In one implementation, the at least one NMT engine is trained to handle the “x_i” placeholders seamlessly. The NMT engine understands that these placeholders represent distinct content components and translate the surrounding text accordingly, without modifying or omitting the placeholders. This ensures that the translated text maintains the same level of accuracy and fidelity as the input document, with the content elements preserved in their original positions.

[0058] For example, consider the sentence: "The equation x_l represents the relationship between energy and mass, as shown in the graph x_2. During the simplification and translation processes, the placeholders "x_l" and "x_2" remains unchanged, while the surrounding text is simplified and translated as required. The output translation might read like: “L'equation x_l represente la relation entre 1'energie et la masse, comme illustre dans le graphique x_2”. (i.e., French translation). It is noted that the placeholders "x_l" and "x_2" are preserved exactly as they appear in the original text, ensuring the accuracy and integrity of the translated document.

[0059] In a nutshell, the "x_i" encoding scheme is a critical component of the parsing and translation process. By replacing the content elements with unique placeholders, the system 100 can efficiently process the text while maintaining the original structure and formatting of the input document. The preservation of these placeholders throughout the simplification and translation steps ensures that the final output is an accurate and faithful representation of the original document (i.e., input document), with all content components intact and properly positioned.

[0060] After the encoding step (where complex elements like equations, symbols, or special characters are replaced with unique placeholders (e.g., "x_l", "x_2", etc.)), the simplification engine 106 is configured to perform simplification process on the extracted text. The simplification engine 106 employs advanced machine learning techniques (for example, a fine-tuned version of a machine learning model), to streamline and optimize the text for translation while preserving the integrity of the placeholders. The machine learning model is fine-tuned on large scale publicly available simplification datasets. The simplification process is a crucial subroutine that enhances the accuracy and fluency of the final translation output by making the text more amenable to machine translation.

[0061] For example, consider the following complex sentence containing mathematical equations and symbols: "The Schrodinger equation, given by i?Z(?P(r,t) / 3t = HT(r,t), is a fundamental equation in quantum mechanics that describes the time-dependent behavior of a quantum-mechanical system, and its solutions, known as wave functions P(r,t), contain all the information about the system's state."

[0062] Once the encoding step is performed, the system 100 is configured to replace the equations and symbols with the placeholders:

[0063] " The Schrodinger equation, given by x_1, is a fundamental equation in quantum mechanics that describes the time-dependent behavior of a quantum- mechanical system, and its solutions, known as wave functions x_3, contain all the information about the system's state."

[0064] In a first implementation, the simplification engine 106 is configured to perform sentence splitting on the statement. For example, the simplification engine 106 converts the sentence into: "The Schrodinger equation is given by x_l. The Schrodinger equation is a fundamental equation in quantum mechanics. It describes the time-dependent behavior of a quantum-mechanical system. The solutions of the Schrodinger equation are known as wave functions x_3. IT contains all the information about the system's state."

[0065] In a second implementation, the simplification engine 106 is configured to perform syntactic restructuring on the statement. For example, the simplification engine 106 converts the sentence into: "x_l is the Schrodinger equation. The Schrodinger equation is a fundamental equation in quantum mechanics. The Schrodinger equation describes the time-dependent behavior of a quantum-mechanical system. The solutions of the Schrodinger equation are known as wave functions. Wave functions are represented by x_3. It contains all the information about the system's state."

[0066] In a third implementation, the simplification engine 106 is configured to perform vocabulary normalization on the statement. For example, the simplification engine 106 converts the sentence into: "x_l represents the Schrodinger equation. The Schrodinger equation is a basic equation in quantum mechanics. It explains the time -dependent behavior of a quantum-mechanical system. The answers to the Schrodinger equation are called wave functions x_3. They hold all the information about the system's condition."

[0067] In this manner, the simplification engine 106 is configured to transform the complex, equation-laden sentence into a more straightforward, translation-friendly format while preserving the placeholders. The simplified text is easier for NMT engine to process, as the complex mathematical equations and symbols are replaced with placeholders, and the sentence structure is streamlined.

[0068] In another example, consider the sentence: "The reaction of H2SO4 with NaOH produces Na2SC>4 and H2O, as described by the balanced chemical equation: H2SO4 + 2NaOHNa2SC>4 + 2H2O, which demonstrates the law of conservation of mass."

[0069] After the encoding step, the system 100 is configured to output: "The reaction of x_l with x_2 produces x_3 and x_4, as described by the balanced chemical equation: x_5, which demonstrates the law of conservation of mass". After the simplification step, the system 100 is configured to output: "x_l reacts with x_2 to produce x_3 and x_4. The balanced chemical equation x_5 describes this reaction. x_5 shows the law of conservation of mass".

[0070] It is noted that the simplification engine 106 is able to handle complex technical content while preserving the integrity of placeholders. By simplifying the text and making it more amenable to machine translation, the simplification engine 106 significantly improves the quality and accuracy of the translations produced by the subsequent NMT engines. The simplification engine 106 ensures that the translated text is both faithful to the original content, including the equations, symbols, and formulas, and fluent in the target language.

[0071] Once the simplification is complete, the NMT engine is configured to analyze and translate the simplified text, with the placeholders intact. In one example, the NMT engine may utilize deep learning architecture (e.g., transformer model) to perform the translation. In general, transformers rely entirely on self-attention mechanisms to capture the dependenciesbetween words in a sentence. This allows the NMT engine to parallelize the computation and process the input sequence as a whole, rather than sequentially, leading to faster training times and improved translation quality.

[0072] It is noted that the translating step is performed subsequent to the simplification step.

[0073] The NMT engine is trained on one or more multilingual datasets. For example, the NMT engine is trained on a dataset for European languages and trained on another dataset for Indic languages. These datasets include millions of parallel sentences across multiple language pairs, allowing the models to learn the intricacies and nuances of language translation.

[0074] By exposing the models to a wide variety of texts, ranging from news articles and government documents to literary works and social media posts, the NMT engine develop a deep understanding of the linguistic structures, idioms, and cultural contexts specific to each language.

[0075] In one implementation, during the translation, the NMT engine employ techniques, such as byte-pair encoding (BPE) and sub-word regularization to handle out-of-vocabulary words and reduce the computational complexity. In general, BPE is a data compression algorithm that iteratively replaces the most frequent pair of bytes in a sequence with a single, unused byte, effectively creating a vocabulary of sub-word units. This allows the NMT model to break down rare or unseen words into smaller, more manageable pieces, improving its ability to generate accurate translations.

[0076] In general, sub-word regularization is a technique that probabilistically segments words into smaller units during training, helping the NMT model learn a more robust and generalizable representation of the language. By introducing noise into the segmentation process, sub-word regularization prevents the NMT model from overfitting to specific word forms and enables it to better handle variations and misspellings in the input text.

[0077] In one implementation, the NMT engine also incorporates advanced techniques such as multi-head attention, residual connections, and layer normalization to further improve the quality and fluency of the translations. For instance, multi-head attention allows the NMT model to attend to different pails of the input sequence simultaneously, capturing multiple types of dependencies and relationships between words. Further, residual connections helpmitigate the vanishing gradient problem and enable the training of deeper networks, while layer normalization stabilizes the training process and promotes faster convergence.

[0078] The NMT engine takes into account the context and semantics of the source text (i.e., the input document) to produce translations that sound natural and coherent in the target language, even when dealing with complex sentence structures or idiomatic expressions.

[0079] For example, consider the simplified sentence with placeholders: "The cat, represented by x_l, is sitting on the mat, denoted by x_2." When passed through the NMT engine, the NMT engine generates the following fluent and idiomatic translation in French: "Le chat, represents par x_l, est assis sur le tapis, designe par x_2". This translation not only accurately conveys the meaning of the original sentence but also preserves the placeholders and maintains the natural flow of the language.

[0080] By leveraging advanced deep learning architectures, massive multilingual datasets, and cutting-edge techniques like transformer models, byte-pair encoding, and sub-word regularization, the NMT engine can generate translations that accurately capture the meaning, context, and nuances of the source text while preserving the integrity of the placeholders introduced during the encoding and simplification steps.

[0081] Once the translation process is complete, the system 100 performs a post-processing step to reintegrate the previously extracted content elements back into the translated document. This decoding ensures that the final output maintains the original document's visual structure and integrity while seamlessly incorporating the translated text.

[0082] In some examples, the one or more target languages include Indic languages such as: Assamese (as), Hindi (hi), Marathi (mr), Tamil (ta), Bengali (bn), Kannada (kn), Odia (or), Telugu (te), Gujarati (gu), Malayalam (ml), and Punjabi (pa). In some examples, the one or more target languages include foreign languages such as: French, German, Dutch, Spanish, Chinese, Arabic, Japanese, Russian, and the like.

[0083] In one embodiment, the system 100 may include a processor and a memory storing instructions (not shown in figures). The processor, when executes the instructions, cause the system 100 to perform the method to perform multilingual document translation. As used herein, a “processor” or “processing unit” includes one or more processors, wherein processorrefers to any logic circuitry for processing instructions. A processor may be a general-purpose processor, a special-purpose processor, a conventional processor, a digital signal processor, a plurality of microprocessors, one or more microprocessors in association with a DSP core, a controller, a microcontroller, a low-end microcontroller, Application Specific Integrated Circuits, Field Programmable Gate Array circuits, any other type of integrated circuits, etc. The processor may perform signal coding data processing, input / output processing, and / or any other functionality that enables the working of the system 100 according to the present disclosure. More specifically, the processor or processing unit is a hardware processor.

[0084] As used herein, “database” “memory unit”, “storage unit” and / or “memory” refers to a machine or computer-readable medium including any mechanism for storing information in a form readable by a computer or similar machine. For example, a computer-readable medium includes read-only memory (“ROM”), random access memory (“RAM”), magnetic disk storage media, optical storage media, flash memory devices or other types of machine- accessible storage media.

[0085] A method for multilingual document translation (i.e., translating one or more input documents into one or more languages) is also disclosed. The method includes receiving, by a server system (e.g., the system 100), the input document in one or more file formats. The method further includes pre-processing, by the server system, the input document for parsing and extracting one or more content elements present in the input document. The parsing is performed based at least on a parsing algorithm. The pre-processing is performed to generate a parsed document.

[0086] In one embodiment, the pre-processing unit 104 pre-processes the input document. The pre-processing unit 104 parses the content elements (including but not limited to images, equations, inline images, headers, footers, etc.) from the input document. The content elements may further include literals, character classes, boundary matchers, quantifiers, groups, and the OR operator. The pre-processing unit 104 parses the input document such that non-text attributes are handled at the same time, thereby improving the translation quality.

[0087] Therefore, the system 100 preserves most of the format and layout of the input document and / or file so that human translator need not spend time replacing the images in theinput document, reverting the font style, or reconstructing the paragraphs. In one embodiment, the input document may include a batch of documents.

[0088] During parsing of the input document, non-textual elements such as images, diagrams, and equations are identified and replaced with unique placeholders such as “x_l”, “x_2”, ..., “x_i”. The encoding of the input document with such placeholders serve two essential purposes: (1) It allows the system 100 to process the text independently of the nontextual elements, facilitating the simplification and translation process; and (2) the unique nature of the placeholders ensures that they can be easily identified and replaced during the decoding stage, without ambiguity or confusion with the actual text.

[0089] It is noted that the choice of using "x" followed by an underscore and a numeric identifier (e.g., x_l) is intentional, as it minimizes the likelihood of these placeholders occurring naturally in the text. This encoding scheme is also compatible with a wide range of languages and writing systems, making it suitable for multilingual document translation.

[0090] The method includes translating, by the server system via implementation of at least one neural machine translation (NMT) engine, the parsed document in one or more target languages to generate one or more translated documents. Each translated document corresponds to machine-translated document in each language.

[0091] It is noted that the system 100 translates the input document and / or files received in different formats which may include but are not limited to Word documents, PDF files, PowerPoint presentations, Excel spreadsheets, Outlook files, and the like. The parsed document is passed into the translating unit 108.

[0092] The pre-processing includes creating, by the server system, the hierarchical tree structure of the input document. The hierarchical tree structure represents a Document Object Model (DOM) tree. The pre-processing further includes analyzing, by the server system, the hierarchical tree structure for parsing and extracting the one or more content elements along with their associated metadata. Furthermore, the pre-processing includes generating, by the server system, the parsed document based at least on the analyzing step.

[0093] During the parsing step, the method includes identifying, by the server system, the one or more content elements present in the input document. The method includes replacing,by the server system, the one or more content elements with one or more pre-defined placeholders. Each content element is replaced with each pre-defined placeholder.

[0094] Prior to translating the parsed document, the method includes optimizing, by the server system, the parsed document based on one or more simplification techniques. The one or more simplification techniques includes one of: sentence split technique, syntactic restructure technique, and vocabulary normalize technique.

[0095] The simplification and translation processes are designed to preserve the integrity of the placeholders throughout their respective processes. During the simplification step, the system employs advanced machine learning techniques, such as fine-tuned machine learning model, to streamline and optimize the text for translation. The simplification step includes techniques such as sentence splitting, syntactic restructuring, and vocabulary normalization, which aim to improve the text's readability and translatability while maintaining the placeholders in their original positions.

[0096] The NMT model utilizes techniques such as multi-head attention, and sub-word regularization, allowing them to process the text while keeping the placeholders intact, ensuring that the translated output maintains the same structure and order of the placeholders as the original input.

[0097] In one implementation, the system 100 utilizes the parsing algorithm to analyze the structure of the input document and extract its content and formatting information. The parsing process creates a hierarchical representation of the input document, where each content element (e.g., paragraphs, tables, images, and equations) is represented as a node in XML tree.

[0098] During the decoding stage or post-processing step, the system 100 traverses the XML tree, replacing the placeholders in the translated text with their corresponding content elements. The hierarchical structure of the XML tree ensures that the reintegration of these elements occurs at the correct positions within the document, maintaining the original layout and formatting.

[0099] For instance, consider a placeholder "x_l" that represents an equation in the original document. During the parsing stage, the equation is extracted and replaced with "x_l", while its position in the XML tree is recorded. After the translation process, the system 100 locatesthe "x_l" placeholder in the translated text and replaces it with the original equation, based on its position in the XML tree. This process ensures that the equation is reintegrated into the translated document at the exact same location as in the original document, preserving the document's visual structure and integrity.

[0100] Furthermore, the XML tree parsing allows the system to handle complex formatting and layout elements, such as tables, headers, footers, and nested elements, with high precision. By maintaining the hierarchical structure of the document throughout the translation process, the decoding scheme can accurately reconstruct the original formatting and layout, even when dealing with intricate designs and multiple levels of nesting.

[0101] Subsequent to the translating step, the method includes post-processing, by the server system, the one or more translated documents to preserve the one or more content elements of the input document in the one or more translated documents. For performing the post-processing, the method includes traversing, by the server system, the hierarchical tree structure. The method further includes re-replacing, by the server system, the one or more placeholders with the one or more content elements.

[0102] Once the translating unit 108 translates the input document in required target languages, the post-processing unit 110 post-processes the one or more translated documents to preserve the content elements (e.g., images, equations, and the like). The translation can be performed without any network or internet connection i.e., offline. Since the system 100 does not require any internet connection to operate, the system 100 provides high security for the highly secured documents. Although the system 100 performs the translation offline, but the system 100 achieves highly accurate translated document.

[0103] During the post-processing step, the method includes storing, by the server system, translations performed during the translating step as segments, in a repository (e.g., a database (not shown in figures)). The method includes utilizing, by the server system, the stored segments to perform the translating step in future.

[0104] The one or more translated documents received from the system 100 are high-quality translation documents that ensure all the linguistic and cultural differences. Further, the system 100 accepts multiple file formats. The critical feature to look up to is the ability to acceptvarious file formats and filter the content. The system 100 may translate a single file or batch of files and even an email or a paragraph of text. According to an embodiment, the system 100 learns the words and phrases while editing translations. The system 100 stores the translations as segments in a bilingual repository. These segments may be used for future translation of documents, thus avoiding the need to translate the same word or phrase twice. Some examples of such materials include spare parts catalogs, organizational code of conduct and employment contracts. This is particularly useful for organizations using specialized vocabulary.

[0105] The system 100 may further be used in any of the wide range of sector including but not limited to banking and financial services, corporate, e-commerce, educational, energy and utility, gaming services, IT companies’ software, legal services, Life science and healthcare services, literature and publication houses, manufacturing and automotive, marketing and advertising, media and entertainment, pharmaceutical, retail, scientific, travel and hospitality etc.

[0106] Further, the system 100 is a multi-lingual document translation system. The system 100 may be an Indic document translator that translates the input document in 12 Indic languages, which may include but not limited to Assamese (as), Hindi (hi), Marathi (mr), Tamil (ta), Bengali (bn), Kannada (kn), Odia (or), Telugu (te), Gujarati (gu), Malayalam (ml), Punjabi (pa) etc. In one implementation, the system 100 is trained at least on Samanantar dataset, which is the largest publicly available collection for Indic languages. The system 100 may be used for translating any English print document, webpage, test-paper into 12 local Indian languages with high accuracy and high security.

[0107] In some embodiments, the system 100 may be used in the field of Artificial Intelligence (Al), neural machine translation (NMT), document translation, web page translation, print translation, and the like.

[0108] Therefore, the system 100 for translating one or more input documents into one or more languages is provided herein, wherein the system 100 pre-processes the input document to parse at least one of content elements including but not limited to images, equations, and the like; passing the parsed document to a translating unit; and a post-processing engine that postprocesses the translated documents to preserve elements (e.g., images, equations, etc.).

[0109] In an embodiment, the one or more input documents are translated to one or more languages, wherein the system 100 can be operable without internet connection.

[0110] FIG. 2 illustrates an exemplary mathematics question paper 200 in English language, in accordance with an embodiment of the present disclosure.

[0111] FIG. 3 illustrates an exemplary mathematics question paper 300 in Hindi language, in accordance with an embodiment of the present invention. The question paper 200 has been converted into the question paper 300 based on implementation of the multilingual document translation method performed by the system 100.

[0112] As shown in FIG. 2, the question paper is in English language; however, the question paper has been changed to Hindi language (as shown in FIG. 3). It is noted that the equations and numbers in both question papers 200 and 300 have remain unchanged despite the language translation process. It is thus noted that the system 100 converts the mathematical question paper from one language to another without any change in the numbers in the question paper.

[0113] The foregoing description of the specific embodiments will so fully reveal the general nature of the embodiments herein that others can, by applying current knowledge, readily modify and / or adapt for various applications such specific embodiments without departing from the generic concept, and, therefore, such adaptations and modifications should and are intended to be comprehended within the meaning and range of equivalents of the disclosed embodiments. It is to be understood that the phraseology or terminology employed herein is for the purpose of description and not of limitation. Therefore, while the embodiments herein have been described in terms of preferred embodiments, those skilled in the ail will recognize that the embodiments herein can be practiced with modification within the spirit and scope of the appended claims.

Claims

I / We claim:

1. A method for multilingual document translation, comprising: receiving, by a server system, an input document in one or more file formats; pre-processing, by the server system, the input document for parsing and extracting one or more content elements present in the input document, wherein the parsing is performed based at least on a parsing algorithm, wherein the pre-processing is performed to generate a parsed document; and translating, by the server system via implementation of at least one neural machine translation (NMT) engine, the parsed document in one or more target languages to generate one or more translated documents, wherein each translated document corresponds to machine-translated document in each language.

2. The method as claimed in claim 1, wherein the pre-processing comprises: creating, by the server system, a hierarchical tree structure of the input document, wherein the hierarchical tree structure represents a Document Object Model (DOM) tree; analyzing, by the server system, the hierarchical tree structure for parsing and extracting the one or more content elements along with their associated metadata; and generating, by the server system, the parsed document based at least on the analyzing step.

3. The method as claimed in claim 2, wherein during the parsing step, the method comprises: identifying, by the server system, the one or more content elements present in the input document; and replacing, by the server system, the one or more content elements with one or more pre-defined placeholders, wherein each content element is replaced with each predefined placeholder.

4. The method as claimed in claim 1, wherein prior to translating the parsed document, the method comprises: optimizing, by the server system, the parsed document based on one or more simplification techniques, wherein the one or more simplification techniques comprises one of: sentence split technique, syntactic restructure technique, and vocabulary normalize technique.

5. The method as claimed in claim 1, wherein subsequent to the translating step, the method comprises: post-processing, by the server system, the one or more translated documents to preserve the one or more content elements of the input document in the one or more translated documents.

6. The method as claimed in claim 5, wherein for performing the post-processing, the method comprises: traversing, by the server system, the hierarchical tree structure; and re-replacing, by the server system, the one or more placeholders with the one or more content elements.

7. The method as claimed in claim 5, wherein during the post-processing, the method comprises: storing, by the server system, translations performed during the translating step as segments, in a repository; and utilizing, by the server system, the stored segments to perform the translating step.

8. The method as claimed in claim 1, wherein the one or more content elements comprise at least one of images, equations, inline images, headers and footers literals, character classes, boundary matchers, quantifiers, groups and OR operator.

9. A system for multilingual document translation, comprising:an input unit configured to receive an input document in one or more file formats; a pre-processing unit configured to pre-process the input document to parse and extract one or more content elements present in the input document, wherein the parse step is performed based at least on a parsing algorithm, wherein the pre-process step is performed to generate a parsed document; and translate, via implementation of at least one neural machine translation (NMT) engine, the parsed document in one or more target languages to generate one or more translated documents, wherein each translated document corresponds to machine- translated document in each language.

10. The system as claimed in claim 9, wherein to perform the pre-process step, the system is caused to: create a hierarchical tree structure of the input document, wherein the hierarchical tree structure represents a Document Object Model (DOM) tree; analyze the hierarchical tree structure to parse and extract the one or more content elements along with their associated metadata; and generate the parsed document based at least on the analyze step.

11. The system as claimed in claim 10, wherein during the parse step, the pre-processing unit is caused to: identify the one or more content elements present in the input document; and replace the one or more content elements with one or more pre-defined placeholders, wherein each content element is replaced with each pre-defined placeholder.

12. The system as claimed in claim 9, comprising: a simplification engine configured to optimize the parsed document based on one or more simplification techniques, wherein the one or more simplification techniques comprises one of: sentence split technique, syntactic restructure technique, and vocabulary normalize technique, wherein the optimize step is performed prior to the translate step.

13. The system as claimed in claim 9, comprising: a post-processing unit configured to post-process the one or more translated documents to preserve the one or more content elements of the input document in the one or more translated documents, wherein the post-process step is performed subsequent to the translate step.

14. The system as claimed in claim 13, wherein to perform the post-process step, the postprocessing unit is configured to: traverse the hierarchical tree structure; and re-replace the one or more placeholders with the one or more content elements.

15. The system as claimed in claim 13, wherein during the post-process step, the postprocessing unit is configured to: store translations performed during the translate step as segments, in a repository; and utilize the stored segments to perform the translate step.

16. The system as claimed in claim 9, wherein the one or more content elements comprise at least one of images, equations, inline images, headers and footers literals, character classes, boundary matchers, quantifiers, groups and OR operator.

Images