A medical literature translation method based on a large language model and a terminology library

Abstract

The application relates to the technical field of medical literature translation, in particular to a medical literature translation method based on a large language model and a terminology database; the method comprises the following steps: S1, reading a to-be-translated document to obtain a document object model (DOM); S2, screening special formats in the document object model; S3, constructing a translation database, wherein the translation database comprises a terminology database, a memory database and an abbreviation database; the to-be-translated document is matched with the translation database by using an Aho-Corasick automatic machine and a vector index technology, and a corresponding matching result is obtained for each translation database; S4, the matching result of the step S3 is provided to an AI system as a reference by using the AI system, the translation of the to-be-translated document is completed, and a translation text is obtained; and S5, the special formats screened in the step S2 are subjected to format restoration in the translation text translated in the step S4, and the application can improve the accuracy and professionalism of medical literature translation.

Description

Technical Field

[0001] This invention relates to the field of medical literature translation technology, and in particular to a medical literature translation method based on a large language model and terminology database. Background Technology

[0002] In daily life and academic communication, there is an increasing demand for translation, with ever-higher requirements. This is especially true in medical communication, where a large amount of medical literature exists in English, often necessitating the translation of English medical documents into Chinese, and vice versa. Accurate document translation enables unified solutions, data exchange, and the integration of results; it holds irreplaceable core value in medical research and practice.

[0003] Existing technology CN109241539B discloses a method for updating a machine learning-based artificial intelligence translation database, including building a vocabulary database using machine learning; establishing an artificial intelligence speech prediction model; converting the speech to be translated into text; and determining the target text based on the vocabulary database and the speech prediction model. This method utilizes a large-scale foreign-language-Chinese dictionary obtained through machine learning, and performs semantic and contextual matching based on a sixth-order deep probabilistic analysis method, reducing computational load and improving translation and update efficiency while ensuring translation accuracy. However, the above method still has a relatively high error rate for translating medical literature terminology, and its accuracy and professionalism are insufficient.

[0004] Therefore, there is an urgent need to provide a medical literature translation method based on a large language model and terminology database, which can improve the accuracy and professionalism of medical literature translation compared with existing technologies. Summary of the Invention

[0005] This invention aims to solve the technical problems existing in the prior art and provides a method for translating medical literature based on a large language model and terminology database.

[0006] To achieve the above objectives, the technical solution adopted by the present invention is as follows:

[0007] A method for translating medical literature based on a large language model and terminology database includes the following steps:

[0008] S1. Read the document to be translated and obtain the Document Object Model (DOM).

[0009] S2. Filter out special formats in the document object model;

[0010] S3. Construct a translation database, which includes a terminology database, a terminology memory database, and an abbreviation database. Use the Aho-Corasick automaton to match the document to be translated with the terminology database and the abbreviation database. Use vector indexing technology to match the document to be translated with the terminology memory database to obtain the matching results.

[0011] S4. Using an AI system, provide the matching results from step S3 to the AI ​​system to complete the translation of the document to be translated, obtaining the translated text; including the following steps:

[0012] S41. Perform structured processing on the paragraphs of the document to be translated, and construct a structured data body for each paragraph containing paragraph identifiers, source text content, and translation fields;

[0013] S42. Construct a structured prompt word containing paragraph merging instructions. The structured prompt word defines the merging rules for the paragraphs to be merged. The merging rules indicate that the merged translation is stored in the translation field of the paragraph with the smallest paragraph identifier in the merged paragraph group, and the translation fields of the remaining merged paragraphs are set to empty.

[0014] S43. Based on the matching results of step S3, the structured data body containing all paragraphs and the structured prompt words of the merging instructions are sent to the main model through a single API call to obtain the translation containing the paragraph merging processing results.

[0015] S5. In the translation of step S4, restore the special formatting selected in step S2.

[0016] Furthermore, the terminology database constructed in step S3 is used to include words and phrases with professional semantics, covering language pairs between Chinese and foreign languages, as well as between foreign languages ​​and Chinese; the abbreviation database constructed is used to process the conversion of abbreviations in translated documents.

[0017] Furthermore, in step S3, the Aho-Corasick automaton is used to match the document to be translated with the terminology database and the abbreviation database to calculate the terms and abbreviations contained in the document. When matching with the abbreviation database, the entire text is scanned for abbreviations. When the abbreviation appears for the first time, the full spelling is used, and the abbreviation is used for all subsequent occurrences.

[0018] Furthermore, the memory constructed in step S3 is used to store specialized text segments.

[0019] Furthermore, in step S3, a pre-trained language model is used to convert the text segments in the historical documents of the memory into vector representations. At the same time, the specialized text segments in the document to be translated are also subjected to the same embedding process and converted into vector representations. Using vector indexing technology, the vectors converted from the historical documents are added to a pre-selected vector index to establish an index structure. The vectors converted from the document to be translated and the vectors converted from the historical documents are matched for text segment similarity. When the text segment matching degree reaches 90% or above, the existing translation corresponding to the specialized text segment in the historical document is retrieved.

[0020] Furthermore, in step S4, the AI ​​system sets the large language model as the primary model and first uses the primary model for translation; the matching results from step S3 are provided to the primary model using prompt engineering.

[0021] Furthermore, the structured prompts constructed in step S42 include: a paragraph data structure description section, which defines the data format of paragraph identifiers and source text content; a merging rule instruction section, which explicitly specifies which paragraphs need to be merged for translation and the storage location of the merged translation; and a list of paragraphs to be translated section, which contains all the structured paragraph data that needs to be translated.

[0022] Furthermore, the special formats selected in step S2 include, but are not limited to, fields and subscripts / superscripts.

[0023] Furthermore, the specific method for filtering special formats in step S2 is as follows:

[0024] S21. Traverse the root node tree, collect multiple paragraph trees, then parse each paragraph tree and collect the minimum format unit tree. Each paragraph tree includes multiple minimum format unit trees.

[0025] S22. Based on the tag name corresponding to the last minimum format unit tree of each paragraph tree, obtain the type of the paragraph tree;

[0026] S23. Based on the minimum format unit tree corresponding to each paragraph tree, and based on the content of the number of w:rPr nodes in the minimum format unit tree, determine the style of the paragraph tree;

[0027] S24. Based on the minimum format unit tree corresponding to each paragraph tree, read the content of the node tree within it, and represent it as follows: <w:t>XXX <w:t>, where XXX represents the content of the node tree;

[0028] S25. Based on the determined type, style, and content, parse each paragraph, determine the location of the field, and collect the content of paragraphs outside the field; sort and store the content of the field and other paragraphs in chronological order to form a reading order;

[0029] S26. Traverse the node array of the reading order, process the subscript and superscript nodes, identify the leading character before the found vertAlign string, and perform logical segmentation and aggregation.

[0030] S27. The position of the summary field and the position of the subscripts and superscripts.

[0031] Furthermore, step S5 specifically includes the following steps:

[0032] S51. Set each special format as an object that needs to be restored, sort each object that needs to be restored, and rearrange the order according to the order in which they appear in the translation.

[0033] S52. Find the text nodes within the paragraph, use them as translation containers, and set them as translation content. Clear the text from nodes other than the translation containers, keeping the DOM tree structure unchanged.

[0034] S53. Iterate through each object that needs to be restored to its original format and output the matching result for each object.

[0035] S54. When the match result is "none", remove the object from the DOM tree; when the match result is "yes", restore the format according to the case.

[0036] S55. Repeat S51-S54 to complete the format restoration.

[0037] Furthermore, in step S54, when the matching result is "yes", the specific method for restoring the format according to different cases is as follows: when the object is a field, it is replaced with the translation; when the object is a subscript or superscript, the content remains unchanged, but the target language settings are adapted; at the same time, on the parent node of the translation container, the matching position is split, and an object tree with the original format is inserted, and the remaining content is stored in the new translation container.

[0038] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0039] This invention establishes a terminology database, an abbreviation database, and an abbreviation database. It uses the Aho-Corasick automaton to match the terminology database and the abbreviation database, and employs vector indexing technology to match the abbreviation database. Matching results are obtained in each database and provided to a large language model as a reference, offering more accurate matching content and aiding in the model's translation. Simultaneously, it filters out special formats in the documents to be translated, and after obtaining the corresponding translation, restores the original formatting, thus improving the accuracy and professionalism of medical literature translation. Attached Figure Description

[0040] Figure 1 This is a flowchart of the method of the present invention. Detailed Implementation

[0041] The technical solution of the present invention will be clearly described below with reference to the accompanying drawings. Obviously, the described embodiments are not all embodiments of the present invention. All other embodiments obtained by those skilled in the art without creative effort are within the protection scope of the present invention.

[0042] like Figure 1 As shown, this invention provides a method for translating medical literature based on a large language model and terminology database, comprising the following steps:

[0043] S1. Read the docx format document to be translated, use Python's zipfile library to decompress the document to be translated, and read the content corresponding to word / document.xml in the decompressed document to obtain the Document Object Model (DOM).

[0044] The Document Object Model (DOM) displays document content in a hierarchical structure, with each node representing a part of the document:

[0045] Root node: usually a label, located at the top level of the tree.

[0046] Child nodes: elements, attributes, text, comments, etc. Elements include, for example, ``, ``, ``, `<c>`, `<c>`, `<p ... .

[0047] Parent-child relationship: can be understood as the parent node of the sum, which is... The parent node.

[0048] Sibling relationship: This can be understood as child nodes of the same parent node being sibling nodes, such as and .

[0049] The node types in the DOM include:

[0050] Element node: corresponding HTML tag (e.g.) , ).

[0051] Attribute nodes: Attributes of an element (such as class="example").

[0052] Text node: Plain text content within an element (e.g., ...) text in the "text" in

[0053] Comment Node: Comments in html <!--Comment-->

[0054] Document Node: The root node of the entire document (document object).

[0055] S2. Screen out special formats in the document object model. The special formats include but are not limited to fields, superscripts, and subscripts. The specific steps are as follows:

[0056] S21. Traverse the root node tree to collect multiple paragraph trees. Each paragraph tree corresponds to a paragraph of the document to be translated. Each paragraph tree can be represented as a tag <w:p>Then, each paragraph tree is parsed, and the minimum format unit tree is collected. <w:r>Each paragraph tree consists of multiple minimal format unit trees.

[0057] S22. Based on the minimum format unit tree of each paragraph tree, obtain the type of that paragraph tree. Based on the tag name corresponding to the last minimum format unit tree of each paragraph tree, obtain the specific type of that paragraph tree. Paragraph tree types include: text type, whitespace, field instructions, field opening, separator, end, page break, image, line break, non-breaking line break, comment, etc. For example, when the tag name of the last minimum format unit tree of a paragraph tree is w:t, the paragraph tree is a text type; when the tag name of the last minimum format unit tree of a paragraph tree is w:tab, the paragraph tree is a whitespace type; when the tag name of the last minimum format unit tree of a paragraph tree is w:instrText, the paragraph tree is a field instruction type.

[0058] S23. Based on the minimum format unit tree corresponding to each paragraph tree, obtain the style corresponding to that paragraph tree. Based on the content of the w:rPr node number in the minimum format unit tree, read the style value and determine the style of the paragraph tree based on the style value. The style includes color, italics, bold, quotation style, etc.

[0059] S24. Based on the minimum format unit tree corresponding to each paragraph tree, read the content of the node tree, represented as follows: <w:t>XXX <w:t>, where XXX represents the content of the node tree.

[0060] S25. Based on the determined type, style, and content, parse each paragraph and determine the position of the fields. The field position includes the opening, splitting, and closing of fields, specifically corresponding to functions such as figure / table references, table of contents, and EndNote bibliographic references in Word. Simultaneously, collect the normal paragraph content outside the fields. Sort and store the field content and other paragraph content in chronological order to form a reading sequence.

[0061] S26. Traverse the node array of the reading order, process the subscript and superscript nodes, and identify the leading character before the found vertAlign string. For example, identify E... max E, 98.5 th The 98.5% accuracy was identified, and logical segmentation and aggregation were performed. The purpose of this step is to split words with subscripts and superscripts, and to perform logical and DOM tree-based segmentation with other parts. For example, for "significant effect on k..." A2 k needs to be removed A2 It is split from the preceding word, while kA2 is treated as a whole.

[0062] S27. Parse the original text of the paragraph, removing non-breaking hyphens and whitespace. Concatenate superscript and subscript words, and extract visible content from the fields, such as... Figure 1 Hyperlinks and cross-references such as Table 1 and Appendix Table 1. Skip the table of contents, bibliographic citations, page number citations, and content copied and pasted from the EndNote plugin; summarize the content that needs to be formatted, i.e., special formatting.

[0063] S3. Construct a translation database, which includes a terminology database, a translation memory database, and an abbreviation database. Match the document to be translated against the terminology database, the translation memory database, and the abbreviation database respectively to obtain matching results. The matching results include terms, technical passages, and abbreviations. The terms, technical passages, and abbreviations matched from the terminology database, the translation memory database, and the abbreviation database are provided to the AI ​​system as a reference for translation.

[0064] Terminology database: This database contains words and phrases with specific professional meanings, covering language pairs between Chinese and foreign languages, as well as between foreign languages ​​(foreign languages ​​may include English, Japanese, French, and other languages).

[0065] Abbreviation Library: Used to handle the conversion of abbreviations in translated documents. For example, an abbreviation appearing for the first time in a Chinese document is usually in the form of "Full Chinese Name (English Abbreviation)," which translates to "Full English Name (English Abbreviation)." Following professional writing conventions, only the full name before the parentheses is translated, while the abbreviation within the parentheses remains unchanged. The AI ​​system will identify the first occurrence of an abbreviation and retain its abbreviated form in subsequent content, avoiding repeated translation to conform to the writing style of professional documents.

[0066] Translation Memory: This stores specialized texts with fixed translations, i.e., previously translated terms and their translations. It is suitable for translating professional documents with relatively fixed content. When the content to be translated matches the text in the memory with a match rate of 90% or higher (match rate is optional), the system can automatically retrieve the corresponding existing translation, make appropriate adjustments according to the current context, and then output the translation, thereby maintaining the existing translation style and ensuring translation accuracy.

[0067] The Aho-Corasick automaton is used to efficiently compute terms and abbreviations contained in the document to be translated. Combining the ideas of trie and finite-state automata, the Aho-Corasick automaton constructs the automaton by preprocessing the pattern string, making the matching time complexity close to linear O(n + m + z), where n represents the text length, m represents the total length of the pattern string, and z represents the number of matches. When matching with the abbreviation database, the entire document is scanned for abbreviations; the full spelling is used only for the first occurrence, and the abbreviation is used for all subsequent occurrences. Each abbreviation in the document content is traversed, and the Aho-Corasick automaton is used to find and match it, replacing the full spelling descriptions that do not appear for the first time.

[0068] A pre-trained language model is used to convert text segments from historical documents in the translation memory into vector representations. Simultaneously, the text segments in the document to be translated undergo the same embedding process, also converting them into vector representations. The pre-trained language model can capture semantic relationships between text segments. Using vector indexing technology, the vectors converted from historical documents are added to a pre-selected vector index to establish a fast retrieval index structure. The vectors converted from the document to be translated are compared with the vectors converted from historical documents based on text segment similarity. When the text segment similarity reaches 90% or higher, the corresponding existing translation from the historical document is retrieved.

[0069] The pre-trained language model can be BERT, Sentence-BERT, OpenAI GPT-3, or other suitable text embedding models. The similarity between text segments can be calculated using cosine similarity.

[0070] S4. The AI ​​system supports multiple model selection. After pre-setting model priorities, the large language model is prioritized as the primary model. When the primary model experiences high load or congestion, the system automatically switches to the next available model to ensure translation efficiency and processing speed. The primary model translates the document to be translated based on the matching results from step S3, obtaining the corresponding translation. Prompt engineering is then used to inject the matching results from step S3 into the primary model. This includes the following steps:

[0071] S41. Perform structured processing on the paragraphs of the document to be translated, and construct a structured data body for each paragraph containing paragraph identifiers, source text content, and translation fields.

[0072] S42. Construct a structured prompt word containing paragraph merging instructions. The structured prompt word defines the merging rules for the paragraphs to be merged. The merging rules indicate that the merged translation is stored in the translation field of the paragraph with the smallest paragraph identifier in the merged paragraph group, and the translation fields of the remaining merged paragraphs are set to empty.

[0073] Specifically, the structured prompts consist of three main components. The first part is a description of the paragraph data structure, explicitly informing the large language model of the input data format, including paragraph IDs and the organization of the original text content. The second part is the merging rule instructions. The merging rules explicitly specify the processing logic for merging paragraphs: when multiple paragraphs need to be merged, the merged complete translation is stored in the `translation_text` field of the paragraph with the smallest paragraph ID in the merged paragraph group, while the `translation_text` fields of the other merged paragraphs are set to empty strings. The third part is a list of paragraphs to be translated, containing all the structured paragraph data that needs to be translated.

[0074] Let's take a specific technical document translation example. Suppose there's a technical manual containing 50 paragraphs that needs to be translated. The system will generate the following structured prompts: First, it will state that the system should translate the following structured paragraph data, informing the large model to maintain overall structural consistency, ensuring the returned translation array has the same length as the input array, and if two translations are merged into one sentence, to retain the merged paragraph_id, and setting the translation to an empty string. Then, it will list the structured data for all 50 paragraphs, with each data item containing a paragraph_id, original_text, and an initially empty translation_text field.

[0075] S43. Based on the matching results of step S3, the structured data body containing all paragraphs and the structured prompt words of the merging instructions are sent to the main model through a single API call to obtain the translation containing the paragraph merging processing results.

[0076] S5. In the translated text from step S4, restore the special formats selected in step S2. This includes the following steps:

[0077] S51. Sort the objects whose format needs to be restored, and rearrange their order according to the order in which they appear in the translation.

[0078] S52. Find the text nodes within the paragraph, use them as translation containers, and set them as translation content. Clear the text from nodes other than the translation containers, keeping the DOM tree structure unchanged.

[0079] S53. Iterate through each object that needs to be restored to its original format and output the matching result for each object.

[0080] S54. When the match result is "none", remove the object from the DOM tree. When the match result is "yes", restore the format according to the case. When the object is a field, replace it with the translation. When the object is a subscript or superscript, the content remains unchanged, but it is adapted according to the characteristics of the target language. For example, spaces need to be added when translating into English.

[0081] Meanwhile, on the parent node of the translation container, the matching position is split, an object tree with the original format is inserted, and the remaining content is stored in the new translation container.

[0082] S55. Repeat S51-S54 to complete the format restoration.

[0083] This invention establishes a terminology database, an abbreviation database, and an abbreviation database. It uses the Aho-Corasick automaton to match the terminology database and the abbreviation database, and employs vector indexing technology to match the abbreviation database. Matching results are obtained in each database and provided to a large language model as a reference, offering more accurate matching content and aiding in the model's translation. Simultaneously, it filters out special formats in the documents to be translated, and after obtaining the corresponding translation, restores the original formatting, thus improving the accuracy and professionalism of medical literature translation.

[0084] Finally, it should be noted that the above content is only used to illustrate the technical solution of the present invention, and is not intended to limit the scope of protection of the present invention. Simple modifications or equivalent substitutions made by those skilled in the art to the technical solution of the present invention do not depart from the essence and scope of the technical solution of the present invention.< / w:t> < / w:t> < / w:r> < / w:p> < / w:t> < / w:t>

Claims

1. A medical literature translation method based on a large language model and a terminology library, characterized in that, The method comprises the following steps: S1, reading a document to be translated to obtain a document object model (DOM); S2, screening special formats in the document object model (DOM); the special formats screened in step S2 include but are not limited to superscripts and subscripts; the specific method for screening the special formats in step S2 is as follows: S21, traversing a root node tree, collecting a plurality of paragraph trees, and then parsing each paragraph tree to collect a minimum format unit tree; each paragraph tree comprises a plurality of minimum format unit trees; S22, obtaining the type of the paragraph tree according to the tag name corresponding to the last minimum format unit tree corresponding to each paragraph tree; S23, determining the style of the paragraph tree according to the minimum format unit tree corresponding to each paragraph tree and the content of the w:rPr node number in the minimum format unit tree; S24, according to the minimum format unit tree corresponding to each paragraph tree, read the content of the node tree therein, denoted as: <w:t>XXX <w:t>wherein, XXX represents the content of the node tree;< / w:t> < / w:t> S25, parsing each paragraph according to the determined type, style and content, determining the position of the field, and collecting the paragraph content outside the field at the same time; the content of the field and the other paragraph content are sorted and stored in sequence according to the order of appearance to form a reading sequence; S26, traversing the node array of the reading sequence, processing the superscript and subscript nodes, identifying the leading characters in front of the found vertAlign string, and logically cutting and aggregating; S27, summarizing the positions of the field and the superscript and subscript; S3, constructing a translation database, the translation database comprising a term library, a memory library and an abbreviation library; using an Aho-Corasick automatic machine to match the document to be translated with the term library and the abbreviation library; using a vector index technology to match the document to be translated with the memory library to obtain a matching result; S4, using an AI system to provide the matching result of step S3 to the AI system to complete the translation of the document to be translated to obtain a translation; comprising the following steps: S41, structuring the paragraphs of the document to be translated, and constructing a structured data body comprising a paragraph identifier, original content and a translation field for each paragraph; S42, constructing a structured prompt word comprising a paragraph merging instruction, the structured prompt word defining a merging rule for the paragraphs to be merged, the merging rule indicating that the merged translation is stored in the translation field of the paragraph with the smallest paragraph identifier in the merged paragraph group, and the translation fields of the remaining merged paragraphs are set to empty; S43, according to the matching result of step S3, sending the structured prompt word comprising all paragraph structured data bodies and merging instructions to the main model through a single API call to obtain a translation comprising a paragraph merging processing result; S5, in the translation of step S4, the special formats screened in step S2 are restored to their original formats; step S5 specifically comprises the following steps: S51, setting each special format as an object that needs to be restored to its original format, sorting each object that needs to be restored to its original format, and reordering according to the order of appearance in the translation; S52, finding the text nodes within the paragraph as a translation container and setting them as translation content, emptying the text nodes other than the translation container, and keeping the DOM tree structure unchanged; S53, traversing each object that needs to be restored to its original format, and outputting the matching result of each object; S54, when the matching result is "no", the object is removed from the DOM tree, and when the matching result is "yes", the format restoration is performed in different cases; S55, repeat S51-S54 to complete the entire format restoration.

2. The medical literature translation method based on a large language model and a terminology library according to claim 1, characterized in that, The term library constructed in step S3 is used to collect words and phrases with professional semantics, covering language pairs in Chinese-foreign and foreign-Chinese translation; the abbreviation library is used to process the conversion of abbreviations in the translated document. 3.The medical literature translation method based on a large language model and a terminology library according to claim 2, characterized in that, In step S3, the Aho-Corasick automaton is used to match the translated document with the term library and the abbreviation library, and calculate the terms and abbreviations contained in the translated document; when matching with the abbreviation library, the full-text overall abbreviation scanning is performed, and the full spelling is used for introduction at the first occurrence, and the abbreviations are used for the latter.

4. The medical literature translation method based on a large language model and a terminology library according to claim 1, characterized in that, The memory library constructed in step S3 is used to store professional text segments.

5. The medical literature translation method based on a large language model and a terminology library according to claim 4, characterized in that, In step S3, the pre-trained language model is used to convert the text segments in the historical documents in the memory library into vector representations, and at the same time, the professional text segments in the translated document are also embedded in the same way to convert them into vector representations; using vector indexing technology, the vectors converted from the historical documents are added to the pre-selected vector index to establish an index structure, and the vectors converted from the translated document and the vectors converted from the historical documents are matched in terms of text segment similarity, when the text segment matching degree reaches 90% or above, the corresponding existing translation of the professional text segment in the historical document is retrieved.

6. The medical literature translation method based on a large language model and a terminology library according to claim 1, characterized in that, In step S4, a large language model is set as the main model in the AI system, and the main model is used for translation first; the matching results of step S3 are provided to the main model in the form of prompt engineering.

7. The medical literature translation method based on a large language model and a terminology library according to claim 1, characterized in that, The structured prompt words constructed in step S42 include: a paragraph data structure description part that defines the data format of paragraph identification and original content; a merging rule instruction part that specifies which paragraphs need to be merged and translated and the storage location of the merged translation; and a list of paragraphs to be translated part that contains all structured paragraphs data that need to be translated.

8. The medical literature translation method based on a large language model and a terminology library according to claim 1, characterized in that, In step S54, when the matching result is "yes", the specific method of format restoration in different cases is: when the object is a domain, it is replaced with the translation; when the object is a superscript or subscript, the content remains unchanged, but the target language is adapted; at the same time, the matching position is split on the parent node of the translation container, and the original format object tree is inserted, and the remaining content is stored in the new translation container.

Images