Translation method, device and platform oriented to DITA document

Abstract

The invention provides a DITA document-oriented translation method, device and platform, and the method, device and platform guarantee the integrity of semantics as much as possible through paragraph-level semantic unit segmentation, avoid semantic truncation, and effectively improve the accuracy of subsequent translation processing. Meanwhile, the self-training large language model is obtained through parameter fine tuning optimization and training of a training data set with layered content, the training data set with the layered content comprises different types of training corpora set according to a preset proportion, and through the design of the training corpora with the layered content, the self-training large language model is obtained. According to the method, the translation result can be effectively improved to better meet the requirements of industry specifications and enterprise styles, the accuracy rate of professional terms and the accuracy rate of the DITA format are improved, the original DITA document format is reserved in the translation process, the processing process of extraction, translation and reconstruction in a conventional scheme is not needed, the accuracy rate of format keeping is higher, and the post-processing workload is reduced.

Description

Technical Field

[0001] This application relates to the field of information technology, and in particular to a method, device and platform for translating DITA documents. Background Technology

[0002] In the software industry, DITA (Darwin Information Typing Architecture) documents are a commonly used structured document format. Users typically use DITA documents to write various important documents, such as product technical documentation. When translating DITA documents, the following methods are usually adopted: a. Traditional machine translation solutions. These solutions suffer from complex terminology management and high maintenance costs, weak contextual understanding capabilities, and cumbersome processing workflows for structured documents.

[0003] b. Directly calling a general-purpose LLM (Large Language Model). This approach, due to the use of a general-purpose model, suffers from insufficient accuracy in terminology, high costs, and a lack of workflow integration and quality assurance mechanisms. Choosing a third-party general-purpose LLM carries data security risks, while local deployment presents issues such as unstable translation quality, insufficient long document processing capabilities, and poor support for structured documents.

[0004] c. AWS (Amazon Web Services) Solution. This solution utilizes model training services provided by cloud platforms such as AWS SageMaker, employs efficient parameter fine-tuning techniques like LoRA to fine-tune open-source models, prepares parallel corpora for supervised learning, deploys the fine-tuned model to provide API services, and optimizes translation output through prompt word engineering. The problem with this solution is: 1. Document segmentation lacking semantic awareness: When performing document segmentation, using fixed-length or simple sentence segmentation can easily destroy semantic integrity.

[0005] 2. Insufficient processing of structured documents: Not optimized for DITA, XML and other format documents.

[0006] 3. The quality assurance mechanism is too simplistic: it mainly relies on word-based reasoning results to ensure translation quality, and lacks multi-level retry and appropriate checking mechanisms.

[0007] 4. Weak workflow integration capabilities: It focuses on providing API interfaces and lacks a one-stop platform for corpus management, task scheduling and result management. Summary of the Invention

[0008] One objective of this application is to provide a method, apparatus, and CI / CD platform for translating DITA documents.

[0009] To achieve the above objectives, this application provides a translation method for DITA documents, the method comprising: Determine if the number of DITA documents to be translated exceeds the token limit; If the token limit is exceeded, the DITA document to be translated will be split into paragraphs according to the document structure. The DITA document to be translated will be split into multiple paragraphs to be translated, and each paragraph to be translated will be treated as a separate corpus to be translated. The self-trained large language model is called to translate each piece of text to be translated, and the original DITA document format is preserved during the translation process to obtain the translation result that retains the original DITA document format. The self-trained large language model is optimized by parameter fine-tuning and is trained using a content-layered training dataset. The content-layered training dataset includes different types of training texts set according to a preset ratio. Each translation result is validated; If the verification is successful, the translation results corresponding to each piece of text to be translated will be concatenated into the target translation result according to the original DITA document format.

[0010] Furthermore, if the token limit is exceeded, the DITA document to be translated is split into paragraphs according to its document structure. Each paragraph is then treated as a separate corpus for translation, including: If the token limit is exceeded, the DITA document to be translated will be split into multiple paragraphs according to the document structure. Determine whether the text length of the paragraph to be translated is less than the merging threshold. If it is less than the merging threshold, merge the paragraph to be translated with the adjacent paragraphs to be translated into a corpus that does not exceed the upper limit of the number of tokens.

[0011] Furthermore, based on the document structure, the DITA document to be translated is split into paragraphs, including: Identify tags related to the document structure starting from the root directory of the DITA document to be translated; The DITA document to be translated is split into paragraphs based on the tags mentioned.

[0012] Furthermore, a self-trained large language model is invoked to translate each piece of text to be translated, preserving the original DITA document format during the translation process, to obtain translation results that retain the original DITA document format, including: Construct prompt words, which include content that retains the original DITA document format during the translation process, as well as the title and summary of the DITA document to be translated; Using the aforementioned prompt words, a self-trained large language model is invoked to translate each piece of text to be translated, preserving the original DITA document format during the translation process, in order to obtain translation results that retain the original DITA document format.

[0013] Furthermore, each translation result is validated, including: Each translation result is subjected to format verification and / or language verification; wherein, the format verification is used to detect whether the format of the translation result is consistent with that of the DITA document to be translated, and the language verification is used to detect whether the translation result contains text content in a non-target language.

[0014] Furthermore, the method also includes: If the verification fails, adjust the temperature parameters of the self-trained large language model; The self-trained large language model is invoked to translate each piece of text to be translated. During the translation process, the original DITA document format is preserved to obtain translation results that retain the original DITA document format, until the verification is passed or the preset number of attempts is reached.

[0015] When the preset number of attempts is reached, the DITA document to be translated is re-splittered according to the document structure, into multiple paragraphs to be translated, and each paragraph to be translated is treated as a separate corpus to be translated. The self-trained large language model is called to translate each piece of text to be translated, and the original DITA document format is preserved during the translation process to obtain translation results that retain the original DITA document format; If the re-splitting verification fails, a third-party large language model is called and a preset terminology list is used to translate each piece of text to be translated. During the translation process, the original DITA document format is preserved to obtain translation results that retain the original DITA document format.

[0016] Furthermore, the method also includes: If one or more translation results corresponding to the corpus to be translated fail the verification, the original text corresponding to the translation result that failed the verification is concatenated with the translation result that failed the verification to obtain the translation result to be verified, and the DITA document to be translated is marked as translation failure.

[0017] Furthermore, the training dataset for the content layering includes: 30% industry-standard documents and enterprise glossaries, 30% UI product interface text, 30% core DITA format documents, and 10% DITA format training corpus pairs.

[0018] This application also provides an integrated translation platform for DITA documents, the platform comprising: The corpus management module is used to obtain a content-layered training dataset, which includes different types of training corpora set according to a preset ratio. The model training model is used to train a self-trained large language model using a content-layered training dataset and to fine-tune and optimize the parameters. The translation processing module is used to process the DITA document to be translated using the aforementioned translation method for DITA documents, and obtain the corresponding target translation result; The interaction module is used to obtain relevant input information and provide the target translation results.

[0019] Based on another aspect of this application, a computing device is also provided, which includes a memory for storing computer program instructions and a processor for executing the computer program instructions, wherein when the computer program instructions are executed by the processor, the device is triggered to execute the translation method for DITA documents.

[0020] This application also provides a computer-readable medium having stored thereon computer program instructions that can be executed by a processor to implement the translation method for DITA documents.

[0021] Compared with existing technologies, this application provides a translation scheme for DITA documents. The scheme first determines whether the DITA document to be translated exceeds the token limit. If it exceeds the token limit, the DITA document to be translated is split into paragraphs according to the document structure, and each paragraph is treated as a separate corpus to be translated. A self-trained large language model is called to translate each corpus to be translated, while preserving the original DITA document format during the translation process to obtain a translation result that retains the original DITA document format. Then, each translation result is verified. If the verification passes, the translation results corresponding to each corpus to be translated are concatenated according to the original DITA document format to form the target translation result.

[0022] This solution employs paragraph-level semantic unit segmentation to ensure semantic integrity and avoid semantic truncation, effectively improving the accuracy of subsequent translation processing. Simultaneously, the self-trained large language model undergoes parameter fine-tuning and optimization, and is trained using a content-layered training dataset. This content-layered training dataset includes different types of training corpora arranged in a preset proportion. This content-layered training corpus design effectively improves the translation results, making them more in line with industry standards and corporate style requirements. It enhances the accuracy of professional terminology and DITA formatting, and preserves the original DITA document format during translation, eliminating the need for the "extraction, translation, and reconstruction" processes found in conventional solutions. This results in higher accuracy in format preservation and reduced post-processing workload.

[0023] In addition, when the translation result verification fails, a triple adaptive retry quality assurance mechanism is adopted, which includes adjusting temperature parameters, re-segmenting the document, and calling a third-party LLM, thereby improving the success rate of translation and effectively reducing human intervention.

[0024] In addition, this application also provides an integrated translation platform for DITA documents, realizing a full-stack platform that integrates corpus management, model training, translation execution, quality inspection, and business integration, rather than simply providing APIs to the outside world. This enables the document workflow to be deeply integrated with business, greatly improving efficiency. Attached Figure Description

[0025] Other features, objects, and advantages of this application will become more apparent from the following detailed description of non-limiting embodiments with reference to the accompanying drawings: Figure 1 A flowchart illustrating a translation method for DITA documents provided in this application embodiment; Figure 2 A schematic diagram of the structure of an integrated translation platform for DITA documents provided in this application embodiment; Figure 3 This is a schematic diagram of the architecture of an integrated AI translation platform implemented using the solution provided in the embodiments of this application; Figure 4 A flowchart illustrating the construction process of the self-trained large language model used by the AI ​​translation integration platform in this application embodiment when performing translation; Figure 5 A flowchart illustrating the process of translating DITA documents using a self-trained large language model. The same or similar reference numerals in the accompanying drawings represent the same or similar parts. Detailed Implementation

[0026] The present application will now be described in further detail with reference to the accompanying drawings.

[0027] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0028] In a typical configuration of this application, the terminal and the service network devices each include one or more processors (CPUs), input / output interfaces, network interfaces, and memory.

[0029] Memory may include non-persistent storage in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM. Memory is an example of computer-readable media.

[0030] Computer-readable media include both permanent and non-permanent, removable and non-removable media, which can store information using any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, read-only optical disc (CD-ROM), digital versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other non-transfer medium that can be used to store information accessible by a computing device.

[0031] This application provides a translation method for DITA documents. This method employs paragraph-level semantic unit segmentation to ensure semantic integrity and avoid semantic truncation, effectively improving the accuracy of subsequent translation processing. Simultaneously, the self-trained large language model undergoes parameter fine-tuning and is trained using a content-layered training dataset. This content-layered training dataset includes different types of training corpora set according to a preset ratio. This content-layered training corpus design effectively improves the translation results, making them more consistent with industry standards and corporate style requirements. It enhances the accuracy of professional terminology and DITA formatting, and preserves the original DITA document format during translation, eliminating the need for the conventional "extraction, translation, and reconstruction" process. This results in higher accuracy in format preservation and reduced post-processing workload.

[0032] In practical scenarios, the executing entity of the method may include user equipment, network equipment, or a device composed of user equipment and network equipment integrated through a network, or it may be an application running on the aforementioned devices. The user equipment includes, but is not limited to, various terminal devices such as computers, mobile phones, and tablets; the network equipment includes, but is not limited to, network hosts, single network servers, servers aggregated from multiple network servers, or servers in a distributed cloud network. Here, the distributed cloud network consists of a large number of hosts or network servers based on cloud computing.

[0033] Figure 1 The present application illustrates a processing flow for a DITA document translation method provided in an embodiment of this application. This method includes at least the following processing steps: Step S101: Determine whether the DITA document to be translated exceeds the token limit.

[0034] In real-world scenarios, due to the constraint of the maximum number of tokens that a large language model can process at once, DITA documents exceeding a certain length cannot be processed by the large language model in one go. Therefore, during translation, it is first necessary to determine whether the DITA document to be translated exceeds the token limit. If it does not exceed the token limit, it means that the large language model can process the DITA document to be translated at once, and the document can be directly input into the large language model as the translation corpus. If it exceeds the token limit, it cannot be processed directly and needs to be segmented using certain methods while ensuring semantic and structural integrity.

[0035] Step S102: If the token quantity limit is exceeded, the DITA document to be translated is split into paragraphs according to the document structure. The DITA document to be translated is split into multiple paragraphs to be translated, and each paragraph to be translated is treated as a separate corpus to be translated.

[0036] For DITA documents, especially those with an XML-like (Extensible Markup Language) structure, when splitting the DITA document to be translated into paragraphs based on its document structure, tags related to the document structure can be identified starting from the root directory of the DITA document to be translated. Then, the document can be split into paragraphs based on these tags. Taking a DITA document D1 as an example, identification can start from the root directory of the main content and split level by level, for example, the tags within it... … Indicates two tags andThe content between these tags belongs to the same paragraph. Therefore, when splitting, it's necessary to avoid splitting the content within these tags to ensure that the split content still maintains its semantic and structural integrity.

[0037] In real-world scenarios, after splitting a DITA document to be translated into paragraphs, there may be paragraphs with too few words. In this case, these paragraphs with too few words can be merged with adjacent paragraphs. While ensuring that the merged content does not exceed the token limit, the entire DITA document to be translated is split into as few corpora as possible to preserve the semantic and structural integrity as much as possible, while improving the efficiency of subsequent processing.

[0038] Specifically, the scheme in this embodiment can split the DITA document to be translated into paragraphs according to the document structure. After splitting the DITA document to be translated into multiple paragraphs, it is determined whether the text length of each paragraph is less than a merging threshold. If it is less than the merging threshold, the paragraph to be translated is merged with adjacent paragraphs into a single corpus that does not exceed the token limit. For example, the merging threshold can be set to 'a', and the token limit can be set to 'b'. For example, after splitting the DITA document D1 into 10 paragraphs p1~p10, the text length of paragraph p4 is less than 'a'. In this case, it can be merged with adjacent paragraphs p3 or p5. During the merging process, it is ensured that the merged content does not exceed the token limit. Thus, a DITA document exceeding the token limit will be intelligently split into multiple corpora.

[0039] In practical scenarios, a recursive approach can be used to implement the merging process after splitting the DITA document to be translated. For example, for the 10 segments p1~p10 obtained from splitting the DITA document D1 to be translated in the aforementioned scenario, we can first determine whether the text length of segment p1 is less than the merging threshold 'a'. If it is less than 'a', then merge it with its next adjacent segment p2. For the merged segment p1+2, we can continue to determine whether it is less than the merging threshold 'a'. If it is still less than 'a', then continue to merge it with its next adjacent segment p3, until the text length of the merged segment is greater than or equal to the merging threshold 'a', at which point we stop merging. At this point, we can start determining whether the text length of the next segment to be translated (such as segment p4) is less than the merging threshold 'a', and use the same recursive logic as before to complete the recursive processing of subsequent segments to be translated. This allows the DITA document to be split into as few segments as possible, thus better preserving the semantic and structural integrity.

[0040] Step S103: Call the self-trained large language model to translate each piece of text to be translated, preserving the original DITA document format during the translation process, so as to obtain translation results that retain the original DITA document format.

[0041] The self-trained large language model mentioned above is the large model used for translating DITA documents in this application, obtained through pre-training. The main difference between this self-trained large language model and the large models in conventional solutions is that it is trained using a content-layered training dataset and undergoes parameter fine-tuning optimization. The content-layered training dataset includes different types of training corpora set according to a preset ratio.

[0042] In some embodiments of this application, the content-layered training dataset may specifically include training corpora in the following proportions: 30% industry-standard documents and enterprise terminology lists, 30% UI (User Interface) product interface text, 30% core DITA format documents, and 10% DITA format training corpus pairs. Taking cloud computing as an example, the industry-standard documents and enterprise terminology lists may include cloud computing general glossaries, enterprise terminology lists compiled by cloud computing companies, etc. UI product interface text may include UI interface text of internal company products, plain text, including short words, sentences, paragraphs, etc. Core DITA format documents may include DITA format text such as enterprise user manuals, requiring a high degree of text consistency. DITA format training corpus pairs are format-related training corpora in DITA documents, used to strengthen the large model's ability to recognize DITA document formats and reduce misidentification of various tag information such as tag pairs in DITA documents. Through this content-layered training corpus design, the translation results can be effectively improved to better conform to industry standards and corporate style requirements, thereby improving the accuracy of professional terminology and DITA format.

[0043] In practical scenarios, any suitable LLM can be chosen for self-trained large language models. Theoretically, the larger the number of parameters in an LLM model, the better. However, a larger number of parameters increases the deployment cost for training and inference. Therefore, this solution selects an LLM with a parameter size of around 7 bytes, which can be trained and inferred using a single conventional GPU with moderate performance. After deployment, the parameters can be fine-tuned and optimized during training using LoRA (Low-Rank Adaptation).

[0044] To ensure that the original DITA document format is preserved during the translation process, the solution in this application specifically constructs prompt words (Prompt) that are input to the self-trained large language model. By explicitly restricting the content that retains the original DITA document format during the translation process in the prompt words, the original DITA document format can be effectively preserved during the translation process.

[0045] For example, when calling a self-trained large language model to translate each piece of text, the following prompt can be used: "You are an AI assistant specializing in translating from {sourceLanguage} to {targetLanguage}. For this translation task: 1. Strictly preserve all Markdown syntax (###, *, -, `, etc.). 2. Preserve all XML / HTML tags (<...>) as is. 3. Do not translate the following: code blocks, commands, file paths, variable names, technical parameters. 4. The following need to be translated: comments, descriptions, error messages, document text. 5. Completely preserve all line breaks and spacing. 6. If applicable, prioritize the use of computer / technical industry terminology. 7. Only provide the translation content, no additional explanation is required." This ensures that the original DITA document format is effectively preserved during the translation process.

[0046] Furthermore, when a DITA document to be translated is split into multiple corpora not exceeding the token limit, each corpus needs to be separately input into a self-trained large language model for translation. To ensure the translation results do not deviate from the core content of the original text, the title and description of the DITA document to be translated can be appended to the prompt words to further improve the accuracy of the translation results. For example, the following content can be added to the aforementioned prompt words: "Translation Context - Topic Domain: {Title}{Description}".

[0047] The Title can be taken from the tag pairs of each DITA document. <title>…< / title> The content within the heading is used for the description, which is taken from the first paragraph of the non-section tag pair under the heading of each DITA document. For example, a portion of the content of a DITA document might look like this: <title> Network interface card (NIC) normalization (optional)< / title> <conbody> "conbody"": Unknown word. This chapter primarily introduces the method for configuring network interface cards (NICs) using normalization. If you are installing and deploying a cloud platform in a real production environment, it is recommended to configure the network using NIC normalization. If you are only conducting a proof-of-concept (POC) test, you can skip this chapter. The following describes two configuration methods for network card normalization: command line method and graphical interface method. <section id="'section jhd tdf_cdb"> The title can be "NIC Normalization (Optional)", while the description can be "This section mainly introduces the NIC normalization configuration method. If you are installing and deploying a cloud platform in a real production environment, it is recommended to use the NIC normalization method to configure the network. If you are only doing POC testing, you can choose to skip this section." Therefore, the solution in this application embodiment can first construct prompt words when calling a self-trained large language model to translate each corpus to be translated. These prompt words include content that retains the original DITA document format during translation, as well as the title and summary of the DITA document to be translated. Then, using these prompt words, the self-trained large language model is called to translate each corpus to be translated, preserving the original DITA document format during the translation process. This yields translation results that retain the original DITA document format, while also preventing the translation results from deviating from the core content of the original text. Furthermore, the entire translation process does not require the complex "extraction-translation-reconstruction" process; it only requires automated intelligent splitting of extremely long DITA documents to complete the format-preserving translation.

[0048] Step S104: Verify each translation result. In this embodiment, the verification may include two aspects: format verification and language verification. The format verification is used to detect whether the format of the translation result is consistent with the format of the DITA document to be translated, and the language verification is used to detect whether the translation result contains text content in a language other than the target language.

[0049] When performing format validation, the following validation methods can be used: Ensure consistency through structural checks, primarily checking whether node levels, node names, node attributes, and the number of nodes are consistent. Extract structural summaries from the original and translated content, and recursively compare the node names, attributes, and child nodes in the structural summaries of the original and translated content for consistency. Inconsistencies in the structure of some non-textual content, such as different image positions corresponding to the Fig tags, can be ignored.

[0050] When performing language validation, regular expression matching can be used to detect whether the translated content contains non-target language content. Taking Chinese to English translation as an example, the following function can be used to detect whether the translated content contains Chinese characters: function containsChinese(text) { return / [\u4e00-\u9fa5] / .test(text); } The regular expression "[\u4e00-\u9fa5]" is used to match Chinese characters.

[0051] Step S105: If the verification passes, the translation results corresponding to each piece of text to be translated are concatenated into the target translation result according to the original DITA document format. For example, for the DITA document D2 to be translated, after splitting it into 5 paragraphs p1~p5, each paragraph p1~p5 is a piece of text to be translated. After calling the self-trained large language model to translate it, 5 corresponding translation results r1~r5 can be obtained. If these translation results all pass the verification, these split translation results r1~r5 can be concatenated into a complete target translation result according to the original DITA document format.

[0052] To further improve translation quality, this application also provides a three-level adaptive retry quality assurance mechanism. By adjusting temperature parameters, re-segmenting documents, and invoking a third-party LLM, the success rate of translation is improved, and the probability of human intervention is reduced. The specific details of this quality assurance mechanism are as follows: If the verification fails, the temperature parameter of the self-trained large language model is adjusted. The self-trained large language model is then used to translate each piece of text to be translated, preserving the original DITA document format during the translation process to obtain translation results that retain the original DITA document format, until the verification passes or the preset number of attempts is reached. For example, in this embodiment, the number of attempts can be set to two. When adjusting the temperature parameter for the first time, the value can be decreased from the original temperature parameter. When adjusting the temperature parameter for the second time, the value can be increased from the original temperature parameter. This improves the translation success rate through different methods. For example, under normal circumstances, the temperature parameter of the self-trained large language model can be set to 0.1. If the verification fails, the temperature parameter can be adjusted to decrease it to 0.01 to reduce the randomness of the LLM output. If the translation result still fails the verification, the temperature parameter can be adjusted again to increase it to 0.2 to increase the randomness of the LLM output. By trying different approaches, translation failures caused by different reasons can be resolved, thereby improving the translation success rate and reducing the probability of human intervention.

[0053] If the verification still fails after reaching the preset number of attempts, the DITA document to be translated can be re-splittered according to its document structure, dividing it into multiple paragraphs and treating each paragraph as a separate corpus. During this splitting process, the parameters can be adjusted, such as performing more detailed splitting to obtain different results. Then, the self-trained large language model is called again to translate each corpus, preserving the original DITA document format to obtain translation results that retain the original DITA document format.

[0054] If the re-splitting verification fails, a third-party large language model can be invoked in conjunction with a pre-defined terminology list to translate each piece of text. During the translation process, the original DITA document format is preserved to obtain translation results that retain the original DITA document format. This three-level adaptive retry quality assurance mechanism, compared to existing word-based reasoning-based quality assurance mechanisms, results in a higher translation success rate and requires less human intervention.

[0055] If one or more translations of the corpus to be translated fail the verification, the corresponding source text of the failed translation can be concatenated with the failed translation to obtain the translation result to be verified, and the DITA document to be translated is marked as a translation failure. At this time, the DITA document marked as "translation failure" can be processed manually to obtain the correct translation result. At the same time, the reasons for the failure can be investigated and the parameters of the scheme can be adjusted to optimize the subsequent processing.

[0056] Furthermore, this application also provides an integrated translation platform for DITA documents, the structure of which is as follows: Figure 2 As shown, the system includes a corpus management module 210, a model training module 220, a translation processing module 230, and an interaction module 240. The corpus management module 210 acquires a content-layered training dataset, which includes different types of training corpora arranged according to a preset ratio. The model training module 220 trains a self-trained large language model using the content-layered training dataset and performs parameter fine-tuning. The translation processing module 230 processes the DITA document to be translated using the aforementioned translation method for DITA documents, obtaining the corresponding target translation result. The interaction module 240 acquires relevant input information and provides the target translation result.

[0057] Figure 3 This diagram illustrates the architecture of an integrated AI translation platform implemented using the solution provided in this application. The platform's infrastructure is based on an AI Infra platform, including a model repository, dataset management, model fine-tuning, and inference services. It is primarily used for LoRA fine-tuning of LLM models and deployment of inference services, providing LLM-related capabilities, as well as basic cloud services related to computing, storage, and networking.

[0058] The integrated AI translation platform includes the following functionalities: corpus management, content translation, document translation, application interface layer, and others. Corpus management is used for full lifecycle management of the trained corpus. Content translation supports real-time translation via a web interface, allowing selection of a self-trained large language model or an external model, and supports result comparison and high-quality translation feedback. Document translation provides complete task management and result preview capabilities, supports various document formats, including DITA documents, and has built-in intelligent processing capabilities for DITA documents to implement the translation methods described in the foregoing embodiments of this application.

[0059] Figure 4 The process of building the self-trained large language model used by this AI translation platform during translation is shown, including the following steps: Step S401, Data preparation.

[0060] To improve the translation quality of LLM in specialized fields, the training dataset is structured as follows: 30% industry-standard documents and enterprise terminology lists, 30% UI (User Interface) product interface text, 30% core DITA format documents, and 10% DITA format training corpus pairs. Taking cloud computing as an example, the industry-standard documents and enterprise terminology lists can include general cloud computing vocabularies and enterprise terminology lists compiled by cloud computing companies. UI product interface text can include UI interface text from internal company products, in plain text format, including short words, sentences, and paragraphs. Core DITA format documents can include DITA format text such as enterprise user manuals, ensuring high text consistency. DITA format training corpus pairs are format-related training data from DITA documents, used to strengthen the large model's ability to recognize DITA document formats and reduce misidentification of label pairs and other label information within DITA documents.

[0061] Before fine-tuning, the training corpus needs to be reviewed, mainly for accuracy, style consistency, and scene coverage.

[0062] Step S402, Model selection.

[0063] In theory, the larger the number of parameters in an LLM model, the better. However, a larger number of parameters increases the deployment cost for training and inference. Therefore, this solution selects an LLM with a parameter size of around 7 bytes, which can be trained and inferred using a single, moderately powerful conventional GPU. Furthermore, the selection prioritizes LLMs with general multilingual capabilities, strong Chinese language support, and robust instruction compliance. Additionally, the chosen LLM needs to support local deployment and LoRA fine-tuning.

[0064] Step S403, model fine-tuning.

[0065] LoRA was used to fine-tune the parameters of the LLM, and the training corpus structure was sharedgpt. The prompt words were designed as follows: "You are an AI assistant, responsible for translating language from {sourceLanguage} to {targetLanguage}. For this translation task: 1. Preserve all Markdown syntax (###, *, -, \`, etc.) as is. 2. Preserve the original form of all XML / HTML tags <...>. 3. Do not translate: code blocks, commands, file paths, variable names, technical parameters. 4. Please translate: comments, descriptions, error messages, and document text. 5. Keep all line breaks and spacing consistent. 6. Prioritize computer / technical industry terminology (if applicable). 7. Only the translation content is provided, without any explanation." Step S404, Model Evaluation. This can be done through manual quality control, BLEU (Bilingual Evaluation Replacement) algorithm script scoring, or AI quality control. Manual quality control primarily assesses the accuracy of content / terminology, fluency, style, and format accuracy.

[0066] Step S405, model iteration.

[0067] The process involves corpus selection and accumulation. Specifically, high-quality corpora can be extracted from the newly translated results and added to the database either manually with a single input or using AI to analyze logs. A second manual quality check is performed on newly added corpora. Then, the process returns to the data preparation step S301 for iteration. During iteration, incremental training is performed if the basic model remains unchanged; otherwise, full retraining is conducted if the basic model needs upgrading.

[0068] Figure 5 The process of translating DITA documents using a self-trained large language model is presented, including the following steps: Step S501: Obtain the DITA document to be translated.

[0069] Step S502: Determine whether the number of tokens in the DITA document to be translated exceeds the upper limit of the self-trained LLM. If the determination result is yes, proceed to step S503; if the determination result is no, proceed to step S507.

[0070] Step S503: Split paragraphs according to the DITA document structure.

[0071] Step S504: Determine whether the split content exceeds the maximum number of tokens in the self-trained LLM. If the result is yes, proceed to step S503 to continue splitting; if the result is no, proceed to step S505.

[0072] Step S505: Intelligently merge the content into the split content according to the upper limit of the number of tokens of the self-trained LLM, so that each merged content is as close as possible to the upper limit of the number of tokens of the LLM.

[0073] Step S506: Add the title and brief description of the DITA document to be translated to the prompt words.

[0074] Step S507: Call the self-trained LLM to perform translation and obtain the translation result.

[0075] Step S508: Verify the format and language of the translation result. If the first two verifications fail, proceed to step S509; if the third verification fails, proceed to step S503; if the fourth verification fails, proceed to step S510; if the fifth verification fails, proceed to step S511. If the verification passes, proceed to step S512.

[0076] Step S509: Adjust the temperature parameters of the self-trained LLM and proceed to step S507.

[0077] Step S510: Call a third-party LLM to translate the terminology list, obtain the translation results, and then execute step S508. Step S511: If the document is not split, output the original DITA document content and mark it as failed. If the document is split, concatenate the translation results of the verified part and the original text of the verified part according to the original DITA document format, and mark the whole document as failed.

[0078] Step S512: Determine whether merging is necessary. If merging is necessary, proceed to step S513; otherwise, proceed to step S514.

[0079] Step S513: Combine and translate the original text according to the original DITA document format, and output the result.

[0080] Step S514: Directly output the translation result.

[0081] Based on the same inventive concept, this application also provides a computing device corresponding to the translation method for DITA documents in the aforementioned embodiments, and its problem-solving principle is similar to that of the method. The computing device provided in this application includes a memory for storing computer program instructions and a processor for executing the computer program instructions, wherein when the computer program instructions are executed by the processor, the device is triggered to implement the methods and / or technical solutions of the aforementioned multiple embodiments of this application.

[0082] The specific implementation of the device may include user equipment, network equipment, or a device composed of user equipment and network equipment integrated through a network, or it may be an application running on the aforementioned device. The user equipment includes, but is not limited to, various terminal devices such as computers, mobile phones, and tablets; the network equipment includes, but is not limited to, network hosts, single network servers, servers aggregated from multiple network servers, or servers in a distributed cloud network. Here, the distributed cloud network consists of a large number of hosts or network servers based on cloud computing.

[0083] In particular, the methods and / or embodiments in this application can be implemented as computer software programs. For example, embodiments of this disclosure include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowchart. When the computer program is executed by a processing unit, it performs the functions defined in the methods of this application.

[0084] It should be noted that the computer-readable medium described in this application can be a computer-readable signal medium or a computer-readable storage medium, or any combination thereof. A computer-readable medium can be, for example,—but not limited to—an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this application, a computer-readable medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device.

[0085] In this application, a computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. The computer-readable signal medium may also be any computer-readable medium other than a computer-readable storage medium, capable of transmitting, propagating, or transmitting a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium may be transmitted using any suitable medium, including but not limited to: wireless, wireline, optical fiber, RF, etc., or any suitable combination thereof.

[0086] Computer program code for performing the operations of this application can be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, and C++, and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0087] The flowcharts or block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of devices, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-specific system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0088] In another aspect, this application also provides a computer-readable medium, which may be included in the device described in the above embodiments; or it may exist independently and not assembled into the device. The aforementioned computer-readable medium carries one or more computer program instructions, which may be executed by a processor to implement the methods and / or technical solutions of the various embodiments of this application.

[0089] It should be noted that this application can be implemented in software and / or a combination of software and hardware, for example, using an application-specific integrated circuit (ASIC), a general-purpose computer, or any other similar hardware device. In some embodiments, the software program of this application can be executed by a processor to implement the steps or functions described above. Similarly, the software program of this application (including related data structures) can be stored in a computer-readable recording medium, such as RAM memory, a magnetic or optical drive, a floppy disk, or similar devices. Furthermore, some steps or functions of this application can be implemented in hardware, for example, as circuitry that cooperates with a processor to perform the various steps or functions.

[0090] It will be apparent to those skilled in the art that this application is not limited to the details of the exemplary embodiments described above, and that this application can be implemented in other specific forms without departing from the spirit or essential characteristics of this application. Therefore, the embodiments should be considered exemplary and non-limiting in all respects, and the scope of this application is defined by the appended claims rather than the foregoing description. Thus, all variations falling within the meaning and scope of equivalents of the claims are intended to be included within this application. No reference numerals in the claims should be construed as limiting the scope of the claims. Furthermore, it is clear that the word "comprising" does not exclude other units or steps, and the singular does not exclude the plural. Multiple units or devices recited in the apparatus claims may also be implemented by a single unit or device through software or hardware. The terms "first," "second," etc., are used to indicate names and do not indicate any particular order. The numerical order of the sequence numbers corresponding to the steps does not indicate any particular execution order; the steps may be executed in any combination of orders provided that the execution logic is met.< / section> < / conbody>

Claims

1. A translation method for DITA documents, characterized in that, The method includes: Determine if the number of DITA documents to be translated exceeds the token limit; If the token limit is exceeded, the DITA document to be translated will be split into paragraphs according to the document structure. The DITA document to be translated will be split into multiple paragraphs to be translated, and each paragraph to be translated will be treated as a separate corpus to be translated. The self-trained large language model is called to translate each piece of text to be translated, and the original DITA document format is preserved during the translation process to obtain the translation result that retains the original DITA document format. The self-trained large language model is optimized by parameter fine-tuning and is trained using a content-layered training dataset. The content-layered training dataset includes different types of training texts set according to a preset ratio. Each translation result is validated; If the verification is successful, the translation results corresponding to each piece of text to be translated will be concatenated into the target translation result according to the original DITA document format.

2. The method according to claim 1, characterized in that, If the token limit is exceeded, the DITA document to be translated will be split into paragraphs according to its document structure. Each paragraph will be treated as a separate corpus for translation, including: If the token limit is exceeded, the DITA document to be translated will be split into multiple paragraphs according to the document structure. Determine whether the text length of the paragraph to be translated is less than the merging threshold. If it is less than the merging threshold, merge the paragraph to be translated with the adjacent paragraphs to be translated into a corpus that does not exceed the upper limit of the number of tokens.

3. The method according to claim 1 or 2, characterized in that, The DITA document to be translated is split into paragraphs according to its document structure, including: Identify tags related to the document structure starting from the root directory of the DITA document to be translated; The DITA document to be translated is split into paragraphs based on the tags mentioned.

4. The method according to claim 1, characterized in that, The self-trained large language model is invoked to translate each piece of text to be translated, preserving the original DITA document format during the translation process, to obtain translation results that retain the original DITA document format, including: Construct prompt words, which include content that retains the original DITA document format during the translation process, as well as the title and summary of the DITA document to be translated; Using the aforementioned prompt words, a self-trained large language model is invoked to translate each piece of text to be translated, preserving the original DITA document format during the translation process, in order to obtain translation results that retain the original DITA document format.

5. The method according to claim 1, characterized in that, Each translation result is validated, including: Each translation result is subjected to format verification and / or language verification; wherein, the format verification is used to detect whether the format of the translation result is consistent with that of the DITA document to be translated, and the language verification is used to detect whether the translation result contains text content in a non-target language.

6. The method according to claim 1, characterized in that, The method further includes: If the verification fails, adjust the temperature parameters of the self-trained large language model; The self-trained large language model is called to translate each piece of text to be translated. During the translation process, the original DITA document format is preserved to obtain translation results that retain the original DITA document format until the verification is passed or the preset number of attempts is reached. When the preset number of attempts is reached, the DITA document to be translated is re-splittered according to the document structure, divided into multiple paragraphs to be translated, and each paragraph to be translated is treated as a separate corpus to be translated. The self-trained large language model is called to translate each piece of text to be translated, and the original DITA document format is preserved during the translation process to obtain translation results that retain the original DITA document format; If the re-splitting verification fails, a third-party large language model is called and a preset terminology list is used to translate each piece of text to be translated. During the translation process, the original DITA document format is preserved to obtain translation results that retain the original DITA document format.

7. The method according to claim 6, characterized in that, The method further includes: If one or more translation results corresponding to the corpus to be translated fail the verification, the original text corresponding to the translation result that failed the verification is concatenated with the translation result that failed the verification to obtain the translation result to be verified, and the DITA document to be translated is marked as translation failure.

8. The method according to claim 1, characterized in that, The content-layered training dataset includes: 30% industry-standard documents and enterprise glossaries, 30% UI product interface text, 30% core DITA format documents, and 10% DITA format training corpus pairs.

9. An integrated translation platform for DITA documents, characterized in that, The platform includes: The corpus management module is used to obtain a content-layered training dataset, which includes different types of training corpora set according to a preset ratio. The model training model is used to train a self-trained large language model using a content-layered training dataset and to fine-tune and optimize the parameters. The translation processing module is used to process the DITA document to be translated using the method described in any one of claims 1 to 8, and obtain the corresponding target translation result; The interaction module is used to obtain relevant input information and provide the target translation results.

10. A computing device, the device comprising a memory for storing computer program instructions and a processor for executing the computer program instructions, wherein, When the computer program instructions are executed by the processor, the device is triggered to perform the method of any one of claims 1 to 8.

11. A computer-readable medium having stored thereon computer program instructions that can be executed by a processor to implement the method as described in any one of claims 1 to 8.

Images