Medical literature large-scale document self-adaptive block translation method and system

Abstract

The invention belongs to the technical field of document engineering, and particularly relates to a medical literature large-scale document self-adaptive block translation method and system. Reading the medical literature, and retaining the original paragraph structure of the document; according to the complexity of the medical literature document content and the document length, document self-adaptive partitioning is carried out; concurrently submitting the blocked document content to a large language model for batch translation; inserting translations corresponding to the contents of each block in combination with the original paragraph structure to obtain full-text translation retaining the original paragraph structure of the document, and performing line number matching by adopting a dichotomy aiming at the condition that the line number of a result returned by a large language model is not matched with the line number of the original document; according to the dichotomy, an optimal segmentation point is found in a result returned by the large language model, so that the line number of the segmented result is matched with the line number of the original text. The large file can be translated, and the original paragraph structure of the file is reserved.

Description

Technical Field

[0001] This invention belongs to the field of document engineering technology, specifically relating to an adaptive chunking translation method and system for large-scale medical literature documents. Background Technology

[0002] Many medical documents are large files, and current mainstream translation tools (such as Google Translate) cannot handle documents larger than 50MB. Existing API interfaces have a call limit of 15,000 tokens (such as the OpenAI Davinci model), making it impossible to handle large files. Traditional OCR + translation solutions result in a document formatting error rate of >40%; mixed text and image documents require manual splitting and reorganization, accounting for 65% of the total time, and it is difficult to maintain the original paragraph format. Summary of the Invention

[0003] This invention aims to address the technical problems existing in the prior art by providing a method and system for adaptive block translation of large-scale medical literature documents.

[0004] To achieve the above technical objectives, the present invention adopts the following technical solution: This invention provides a method for adaptive chunking translation of large-scale medical literature documents, comprising the following steps: Step S0: Read the medical literature and preserve the original paragraph structure of the document; Step S1: Perform adaptive document segmentation based on the complexity and length of the medical literature's content. Step S2: Submit the segmented document content concurrently to the large language model for batch translation to obtain the translation of each segment. Step S3: Combine the original paragraph structure with the corresponding translations of each block of content to obtain a full-text translation that retains the original paragraph structure of the document. For cases where the number of lines returned by the large language model does not match the number of lines in the original document, a binary search method is used to match the number of lines. The binary search method is to find an optimal split point in the result returned by the large language model so that the number of lines in the split result matches the number of lines in the original document.

[0005] Preferably, in step S3, the specific steps for row number matching using the binary search method are as follows: Step S301: Set the starting index low and the ending index high, where the starting index low is 0 and the ending index high is the number of rows returned by the large language model - 1; Step S302: Calculate the segmentation point mid = (low + high) / 2; Step S303: The results returned by the large language model are split from the starting index low to the split point mid to obtain a subset output_subset; Step S304: Calculate the number of lines in the subset `output_subset` (`output_lines`), and compare `output_lines` with the number of lines in the original text (`original_lines`). If output_lines is greater than original_lines, it means that the split point mid is too large. Update the end index high to mid - 1. If low <= high, jump to step S302; if low > high, select low or high as the final split point according to actual needs and end the process. If output_lines is less than original_lines, it means that the split point mid is too small. Update the starting index low to mid + 1. If low <= high, jump to step S302. If low > high, select low or high as the final split point according to the actual needs and end the process. If output_lines equals original_lines, then the optimal split point has been found; the process ends.

[0006] Preferably, if low > high at the end of the loop, the line number closer to the original text between low and high is selected as the split point.

[0007] Preferably, if low > high at the end of the loop, the optimal split point is selected based on the semantic rationality determined by low and high.

[0008] Preferably, in step S1, the document is divided into blocks according to the changes in the chapters, and the same chapter is divided into one block; at the same time, the document is divided into multiple blocks according to a pre-set length threshold by using any one of the number of characters, the number of words, or the number of sentences as the length index, and the content of the same paragraph is kept in the same block.

[0009] Preferably, step S0 includes the following steps: Step S001: Use the python-lxml library to parse the input document, extract text blocks and merge them into paragraphs; Step S002: Using the DOM tree structure, save the original topological relationships in the document for paragraphs, tables, images, and cross-references.

[0010] Preferably, step S001 specifically includes the following steps: Step S0011: Use Python's zipfile library to decompress the docx file, and then read the contents of word / document.xml, which contains the document's text and formatting information; Step S0012: Use the xml.etree.ElementTree library to parse word / document.xml, traverse the XML tree, and find the elements. <w:p>and <w:t>,in, <w:p>The element represents a paragraph. <w:t>The element contains specific text content; Step S0013, Extract <w:t>The text within the element, and based on <w:p>The hierarchical relationship of elements groups text into paragraphs; Step S0014: Merge the extracted text belonging to a paragraph into a paragraph list, with each paragraph corresponding to a paragraph list.

[0011] Preferably, step S002 specifically includes the following steps: Step S0021: Map the XML tree structure of the document to a DOM tree. Each paragraph, table, image, and cross-reference is represented as a node in the DOM tree. Each node includes node type, content, attributes, and child nodes. Step S0022: When parsing word / document.xml, a DOM tree is constructed based on the nesting relationship of the document structure. The entire document is taken as the root node, and the parent-child relationship is determined based on the nesting relationship of the document structure. Parallel relationships under the same parent node are sibling relationships. Step S0023: Output the merged paragraph list to the DOM tree according to the DOM tree structure.

[0012] Preferably, in step S3, after a block translation result is returned, a candidate is selected sequentially from the DOM tree. <w:t>Clear the content of the node and insert the translation.

[0013] This invention provides a large-scale adaptive block translation system for medical literature, including a processor capable of executing a computer program that, when executed, implements the aforementioned large-scale adaptive block translation method for medical literature.

[0014] Compared with the prior art, the beneficial effects of the present invention are: (1) This invention uses the DOM tree to preserve the original paragraph structure of the document. When the translation is obtained, it is inserted according to the original paragraph structure, so that the translation can be obtained while preserving the original paragraph structure of the document.

[0015] (2) In the case of a mismatch between the number of lines in the translation result returned by the large language model and the number of lines in the original text, the present invention uses a binary search method to match the number of lines, so as to preserve the paragraph format of the original document to the greatest extent while maintaining the accuracy of the translation result.

[0016] (3) The present invention divides large-scale documents into blocks and submits them concurrently to a large language model for translation, which enables the large language model to process large files. Attached Figure Description

[0017] Figure 1 This is a schematic diagram illustrating the workflow of a large-scale adaptive chunked translation method for medical literature, according to an embodiment of the present invention. Detailed Implementation

[0018] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0019] like Figure 1 As shown, this invention provides a method for adaptive chunking translation of large-scale medical literature documents, comprising the following steps: Step S0: Read the medical literature and preserve the original paragraph structure of the document; Step S1: Perform adaptive document segmentation based on the complexity and length of the medical literature's content. Step S2: The segmented document content is submitted concurrently to the large language model for batch translation to obtain the translation of each segment. The large language model translation can use existing large language model technology. It is only necessary to send the segmented text to the large language model, and the large language model can return the translated text. Using the large language model for translation is a relatively mature technology and is not protected by this invention. It will not be described in detail here.

[0020] Step S3 involves inserting the translations corresponding to each block's content, combining the original paragraph structure to obtain a full-text translation that preserves the original document's paragraph structure. For cases where the number of lines returned by the large language model does not match the number of lines in the original document, a binary search method is used for line number matching. This binary search method finds an optimal segmentation point in the large language model's returned results, ensuring that the number of lines in the segmented result matches the number of lines in the original text. Each block is a collection of multiple paragraphs, and the program records the paragraph numbers within each block, such as "1", "2", ... "99", to preserve the paragraph structure.

[0021] According to a specific embodiment of the present invention, in step S3, the specific steps of matching the number of rows using the binary search method are as follows: Step S301: Set the starting index low and the ending index high, where the starting index low is 0 and the ending index high is the number of rows returned by the large language model - 1; Step S302: Calculate the segmentation point mid = (low + high) / 2; Step S303: The results returned by the large language model are split from the starting index low to the split point mid to obtain a subset output_subset; Step S304: Calculate the number of lines in the subset `output_subset` (`output_lines`), and compare `output_lines` with the number of lines in the original text (`original_lines`). If output_lines is greater than original_lines, it means that the split point mid is too large. Update the end index high to mid - 1. If low <= high, jump to step S302; if low > high, select low or high as the final split point according to actual needs and end the process. If output_lines is less than original_lines, it means that the split point mid is too small. Update the starting index low to mid + 1. If low <= high, jump to step S302. If low > high, select low or high as the final split point according to the actual needs and end the process. If output_lines equals original_lines, then the optimal split point has been found; the process ends.

[0022] According to a specific embodiment of the present invention, if low > high at the end of the loop, the line number of the original text that is closer to low and high is selected as the split point.

[0023] According to a specific embodiment of the present invention, if low > high at the end of the loop, the optimal split point is selected based on the semantic rationality determined by low and high.

[0024] According to a specific embodiment of the present invention, in step S1, the document is divided into blocks according to the changes in the chapters, and the same chapter is divided into blocks; at the same time, the document is divided into multiple blocks according to a pre-set length threshold using any one of the number of characters, the number of words, or the number of sentences, wherein the content of the same paragraph is retained in the same block.

[0025] According to a specific embodiment of the present invention, step S0 includes the following steps: Step S001: Use the python-lxml library to parse the input document, extract text blocks and merge them into paragraphs; Step S002: Using the DOM tree structure, save the original topological relationships in the document for paragraphs, tables, images, and cross-references.

[0026] According to a specific embodiment of the present invention, step S001 specifically includes the following steps: Step S0011: Use Python's zipfile library to decompress the docx file, and then read the contents of word / document.xml, which contains the document's text and formatting information; Step S0012: Use the xml.etree.ElementTree library to parse word / document.xml, traverse the XML tree, and find the elements. <w:p>and <w:t>,in, <w:p>The element represents a paragraph. <w:t>The element contains specific text content; Step S0013, Extract <w:t>The text within the element, and based on <w:p>The hierarchical relationship of elements groups text into paragraphs; Step S0014: Merge the extracted text belonging to a paragraph into a paragraph list, with each paragraph corresponding to a paragraph list.

[0027] According to a specific embodiment of the present invention, step S002 specifically includes the following steps: Step S0021: Map the XML tree structure of the document to a DOM tree. Each paragraph, table, image, and cross-reference is represented as a node in the DOM tree. Each node includes node type, content, attributes, and child nodes. Step S0022: When parsing word / document.xml, a DOM tree is constructed based on the nesting relationship of the document structure. The entire document is taken as the root node, and the parent-child relationship is determined based on the nesting relationship of the document structure. Parallel relationships under the same parent node are sibling relationships. Step S0023: Output the merged paragraph list to the DOM tree according to the DOM tree structure.

[0028] According to a specific embodiment of the present invention, in step S3, after a block translation result is returned, a candidate is sequentially selected in the DOM tree. <w:t>Clear the content of the node and insert the translation.

[0029] This invention provides a large-scale adaptive block translation system for medical literature, including a processor capable of executing a computer program that, when executed, implements the aforementioned large-scale adaptive block translation method for medical literature.

[0030] Example 1 This invention provides a method for adaptive chunking translation of large-scale medical literature documents, comprising the following steps: Step S0: Read the medical literature and preserve the original paragraph structure of the document; Step S1: Perform adaptive document segmentation based on the complexity and length of the medical literature's content. Step S2: The segmented document content is submitted concurrently to the large language model for batch translation to obtain the translation of each segment. The large language model translation can use existing large language model technology. It is only necessary to send the segmented text to the large language model, and the large language model can return the translated text. Using the large language model for translation is a relatively mature technology and is not protected by this invention. It will not be described in detail here.

[0031] Step S3: Combine the original paragraph structure with the corresponding translations of each block of content to obtain a full-text translation that retains the original paragraph structure of the document. For cases where the number of lines returned by the large language model does not match the number of lines in the original document, a binary search method is used to match the number of lines. The binary search method is to find an optimal split point in the result returned by the large language model so that the number of lines in the split result matches the number of lines in the original document.

[0032] Example 2 This invention provides a method for adaptive chunking translation of large-scale medical literature documents, comprising the following steps: Step S0: Read the medical literature and preserve the original paragraph structure of the document; Step S0 includes the following steps: Step S001: Use the python-lxml library to parse the input document, extract text blocks and merge them into paragraphs; Step S002: Using the DOM tree structure, save the original topological relationships in the document for paragraphs, tables, images, and cross-references.

[0033] Furthermore, step S001 specifically includes the following steps: Step S0011: Use Python's zipfile library to decompress the docx file, and then read the contents of word / document.xml, which contains the document's text and formatting information; Step S0012: Use the xml.etree.ElementTree library to parse word / document.xml, traverse the XML tree, and find the elements. <w:p>and <w:t>,in, <w:p>The element represents a paragraph. <w:t>The element contains specific text content; Step S0013, Extract <w:t>The text within the element, and based on <w:p>The hierarchical relationship of elements combines the text into paragraphs; Step S0014, merging the extracted texts belonging to one paragraph into a paragraph list, each paragraph corresponding to a paragraph list.

[0034] Further, step S002 specifically comprises the following steps: Step S0021, mapping the XML tree structure of the document into a DOM tree, each paragraph, table, picture and cross-reference being represented as a node in the DOM tree, each node including node type, content, attribute and child node; Step S0022, during parsing of word / document.xml, constructing the DOM tree according to the nesting relationship of the document structure, taking the entire document as the root node, determining the parent-child relationship according to the nesting relationship of the document structure, and the parallel relationship under the same parent node being the sibling relationship; Step S0023, outputting the merged paragraph list to the DOM tree according to the structure of the DOM tree.

[0035] Step S1, performing document adaptive blocking according to the complexity of the document content and the document length of the medical literature; in this embodiment, the document is blocked according to the chapter change, and the same chapter is divided into a block; at the same time, any one of the number of characters, the number of words and the number of sentences is used as an index of the length of the document, and the document is divided into multiple blocks according to the pre-set length threshold, wherein the contents of the same paragraph are kept in the same block.

[0036] Step S2, submitting the document content after blocking to a large language model for batch translation to obtain the translation of each block content; the large language model translation can use existing large language model technology, only needs to send the text after blocking to the large language model, and the large language model can return the translated text, the translation using the large language model belongs to a relatively mature technology and does not belong to the content protected by the present application, which will not be described in detail here.

[0037] Step S3, inserting the translation corresponding to each block content in combination with the original paragraph structure to obtain the full-text translation retaining the original paragraph structure of the document, and adopting bisection method for line number matching in the case that the returned result line number of the large language model does not match the original document line number; the bisection method is to find a best split point in the returned result of the large language model, so that the line number of the split result matches the original document line number.

[0038] Further, in step S3, the specific steps of adopting bisection method for line number matching are as follows: Step S301, setting a start index low and an end index high, wherein the start index low is 0 and the end index high is the line number of the returned result of the large language model minus 1; Step S302: Calculate the segmentation point mid = (low + high) / 2; Step S303: The results returned by the large language model are split from the starting index low to the split point mid to obtain a subset output_subset; Step S304: Calculate the number of lines in the subset `output_subset` (`output_lines`), and compare `output_lines` with the number of lines in the original text (`original_lines`). If output_lines is greater than original_lines, it means that the split point mid is too large. Update the end index high to mid - 1. If low <= high, jump to step S302; if low > high, select low or high as the final split point according to actual needs and end the process. If output_lines is less than original_lines, it means that the split point mid is too small. Update the starting index low to mid + 1. If low <= high, jump to step S302. If low > high, select low or high as the final split point according to the actual needs and end the process. If output_lines equals original_lines, then the optimal split point has been found; the process ends.

[0039] Furthermore, if low > high at the end of the loop, the line number closer to the original text between low and high is chosen as the split point.

[0040] Furthermore, if low > high at the end of the loop, the optimal split point is selected based on the semantic rationality determined by low and high.

[0041] Furthermore, in step S3, after a block translation result is returned, a candidate is sequentially selected from the DOM tree. <w:t>The node is emptied, the translation is inserted, and the translation of all blocks is inserted after which the full translation is obtained, preserving the original structure of the document.

[0042] The above descriptions are only the preferred embodiments of the present application, not intended to limit the present application. Any modification, equivalent replacement and improvement made without departing from the spirit and principle of the present application shall fall within the scope of protection of the present application.< / w:t> < / w:p> < / w:t> < / w:t> < / w:p> < / w:t> < / w:p> < / w:t> < / w:p> < / w:t> < / w:t> < / w:p> < / w:t> < / w:p> < / w:t> < / w:p> < / w:t> < / w:t> < / w:p> < / w:t> < / w:p>

Claims

1. A method for adaptive chunking translation of large-scale medical literature documents, characterized in that, Includes the following steps: Step S0: Read the medical literature and preserve the original paragraph structure of the document; Step S1: Perform adaptive document segmentation based on the complexity and length of the medical literature's content. Step S2: Submit the segmented document content concurrently to the large language model for batch translation to obtain the translation of each segment. Step S3: Combine the original paragraph structure with the corresponding translations of each block of content to obtain a full-text translation that retains the original paragraph structure of the document. For cases where the number of lines returned by the large language model does not match the number of lines in the original document, a binary search method is used to match the number of lines. The binary search method is to find an optimal split point in the result returned by the large language model so that the number of lines in the split result matches the number of lines in the original document.

2. The adaptive chunking translation method for large-scale medical literature documents according to claim 1, characterized in that, In step S3, the specific steps for row number matching using the binary search method are as follows: Step S301: Set the starting index low and the ending index high, where the starting index low is 0 and the ending index high is the number of rows returned by the large language model - 1; Step S302: Calculate the segmentation point mid = (low + high) / 2; Step S303: The results returned by the large language model are split from the starting index low to the split point mid to obtain a subset output_subset; Step S304: Calculate the number of lines in the subset `output_subset` (`output_lines`), and compare `output_lines` with the number of lines in the original text (`original_lines`). If output_lines is greater than original_lines, it means that the split point mid is too large. Update the end index high to mid - 1. If low <= high, jump to step S302. If low > high, select low or high as the final split point according to the actual needs and end the process. If output_lines is less than original_lines, it means that the split point mid is too small. Update the starting index low to mid + 1. If low <= high, jump to step S302. If low > high, select low or high as the final split point according to the actual needs and end the process. If output_lines equals original_lines, then the optimal split point has been found; the process ends.

3. The adaptive chunking translation method for large-scale medical literature documents according to claim 2, characterized in that, If low > high at the end of the loop, choose the line number of the original text that is closer to low or high as the split point.

4. The adaptive chunking translation method for large-scale medical literature documents according to claim 2, characterized in that, If low > high at the end of the loop, select the best split point based on the semantic rationality determined by low and high.

5. The adaptive chunking translation method for large-scale medical literature documents according to claim 1, characterized in that, In step S1, the document is divided into blocks according to the changes in the chapters, and the same chapter is divided into one block; at the same time, the document length is divided into multiple blocks according to a pre-set length threshold, using any one of the number of characters, the number of words, or the number of sentences as the document length indicator, and the content of the same paragraph is kept in the same block.

6. The adaptive chunking translation method for large-scale medical literature documents according to claim 1, characterized in that, Step S0 includes the following steps: Step S001: Use the python-lxml library to parse the input document, extract text blocks and merge them into paragraphs; Step S002: Using the DOM tree structure, save the original topological relationships in the document for paragraphs, tables, images, and cross-references.

7. The adaptive chunking translation method for large-scale medical literature documents according to claim 6, characterized in that, Step S001 specifically includes the following steps: Step S0011: Use Python's zipfile library to decompress the docx file, and then read the contents of word / document.xml, which contains the document's text and formatting information; Step S0012: Use the xml.etree.ElementTree library to parse word / document.xml, traverse the XML tree, and find the elements. <w:p>and <w:t>,in, <w:p>The element represents a paragraph. <w:t> The element contains specific text content;< / w:t> < / w:p> < / w:t> < / w:p> Step S0013, Extract <w:t>The text in the element, and according to <w:p> The hierarchical relationship of elements groups text into paragraphs;< / w:p> < / w:t> Step S0014: Merge the extracted text belonging to a paragraph into a paragraph list, with each paragraph corresponding to a paragraph list.

8. The adaptive chunking translation method for large-scale medical literature documents according to claim 7, characterized in that, Step S002 specifically includes the following steps: Step S0021: Map the XML tree structure of the document to a DOM tree. Each paragraph, table, image, and cross-reference is represented as a node in the DOM tree. Each node includes node type, content, attributes, and child nodes. Step S0022: When parsing word / document.xml, a DOM tree is constructed based on the nesting relationship of the document structure. The entire document is taken as the root node, and the parent-child relationship is determined based on the nesting relationship of the document structure. Parallel relationships under the same parent node are sibling relationships. Step S0023: Output the merged paragraph list to the DOM tree according to the DOM tree structure.

9. The adaptive chunking translation method for large-scale medical literature documents according to claim 8, characterized in that, In step S3, after a block translation result is returned, a candidate is selected sequentially from the DOM tree. <w:t> Clear the content of the node and insert the translation.< / w:t> 10. A large-scale adaptive chunking translation system for medical literature, characterized in that, The system includes a processor capable of executing a computer program that, when executed, implements the adaptive chunking translation method for large-scale medical literature documents as described in any one of claims 1-9.

Images