Method and system for large-scale document adaptive chunking translation of medical literature

Abstract

The present application belongs to the technical field of document engineering, and particularly relates to a medical literature large-scale document self-adaptive block translation method and system. The medical literature is read, and the original paragraph structure of the document is retained; the document is self-adaptively blocked according to the complexity of the medical literature document content and the document length; the blocked document content is concurrently submitted to a large language model for batch translation; the translation corresponding to each block content is inserted into the original paragraph structure to obtain full-text translation retaining the original paragraph structure of the document; in the case that the number of returned results of the large language model does not match the number of original document lines, a dichotomy is used for line number matching; the dichotomy is to find a best segmentation point in the returned results of the large language model, so that the number of segmented result lines matches the number of original lines. The present application can translate large files while retaining the original paragraph structure of the document.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of document engineering, and particularly relates to a medical literature large-scale document adaptive block translation method and system. BACKGROUND

[0002] Most medical literatures are large files. The current mainstream translation tools (such as Google Translate) cannot process documents exceeding 50 MB. The existing API interface has a calling limit of 15,000 tokens (such as OpenAI Davinci model), and cannot process large files. The traditional OCR+translation scheme results in a document layout disorder rate of >40%; mixed text and image documents need to be manually split and reorganized, and the time consumption accounts for 65% of the total working hours, making it difficult to maintain the original paragraph format. SUMMARY

[0003] The application aims at the technical problems in the prior art and provides a medical literature large-scale document adaptive block translation method and system.

[0004] To achieve the above technical purposes, the application adopts the following technical solutions:

[0005] The application provides a medical literature large-scale document adaptive block translation method, which comprises the following steps:

[0006] Step S0, reading medical literature and retaining the original paragraph structure of the document;

[0007] Step S1, performing document adaptive block according to the complexity and length of the document content of the medical literature;

[0008] Step S2, submitting the blocked document content to a large language model for batch translation in parallel to obtain translated texts of the blocks;

[0009] Step S3, inserting the translated texts corresponding to the blocks in combination with the original paragraph structure to obtain full-text translation retaining the original paragraph structure of the document. In the case that the number of returned results of the large language model does not match the number of original documents, a dichotomy is used for line number matching. The dichotomy is to find a best split point in the returned results of the large language model, so that the number of lines after splitting matches the number of original lines.

[0010] Preferably, in step S3, the dichotomy for line number matching is specifically as follows:

[0011] Step S301, setting a start index low and an end index high, wherein the start index low is 0, and the end index high is the number of lines of the returned results of the large language model minus 1;

[0012] Step S302, calculate the split point mid = (low + high) / 2;

[0013] Step S303, split the large language model return result from the starting index low to the split point mid to obtain a subset output_subset;

[0014] Step S304, calculate the number of lines of the subset output_subset, output_lines, and compare output_lines with the number of lines of the original text original_lines;

[0015] If output_lines is greater than original_lines, it means that the split point mid is too large, and the end index high is updated to mid - 1. If low<= high, jump to step S302; if low> high, select low or high as the final split point according to actual needs, and end the process;

[0016] If output_lines is less than original_lines, it means that the split point mid is too small, and the starting index low is updated to mid + 1. If low<= high, jump to step S302; if low> high, select low or high as the final split point according to actual needs, and end the process;

[0017] If output_lines is equal to original_lines, the best split point is found; end the process.

[0018] Preferably, if low> high when the loop ends, select the one closer to the number of lines of the original text as the split point from low and high.

[0019] Preferably, if low> high when the loop ends, select the best split point according to the rationality of the semantics determined by low and high.

[0020] Preferably, in step S1, the document is divided into blocks according to the changes in chapters, and the same chapter is divided into one block; at the same time, any one of the number of characters, the number of words, and the number of sentences is used as a length indicator of the document, and the document is divided into multiple blocks according to a pre-set length threshold, wherein the contents of the same paragraph are kept in the same block.

[0021] Preferably, step S0 includes the following steps:

[0022] Step S001, parse the input document using the python-lxml library, extract the text blocks and merge them into paragraphs;

[0023] Step S002, using DOM tree structure, save the original topological relationship in the document of paragraph, table, picture and cross reference.

[0024] Preferably, step S001 specifically includes the following steps:

[0025] Step S0011, use the zipfile library of Python to decompress the docx file, and then read the content of word / document.xml, which contains the text and format information of the document;

[0026] Step S0012, use the xml.etree.ElementTree library to parse word / document.xml, traverse the XML tree, find the element <w:p>and <w:t>wherein, <w:p>Element represents a paragraph, <w:t>The element contains specific text content.

[0027] Step S0013, extracting <w:t>text in the element, and according to <w:p>The hierarchical relationship of the elements combines the texts into paragraphs;

[0028] In step S0014, the extracted texts belonging to one paragraph are merged into a paragraph list, and each paragraph corresponds to a paragraph list.

[0029] Preferably, step S002 specifically comprises the following steps:

[0030] In step S0021, the XML tree structure of the document is mapped into a DOM tree, and each paragraph, table, picture and cross-reference is represented as a node in the DOM tree, and each node comprises a node type, content, attribute and child node.

[0031] In step S0022, when parsing word / document.xml, the DOM tree is constructed according to the nested relationship of the document structure, the entire document is taken as a root node, the parent-child relationship is determined according to the nested relationship of the document structure, and the parallel relationship under the same parent node is the sibling relationship.

[0032] In step S0023, the merged paragraph list is output to the DOM tree according to the structure of the DOM tree.

[0033] Preferably, in step S3, after a block translation result is returned, a candidate <w:t>Node, empty the contents of the node, insert the translation.

[0034] The application provides a medical literature large-scale document adaptive block translation system, comprising a processor capable of executing a computer program, which can implement the above-mentioned medical literature large-scale document adaptive block translation method when executed.

[0035] Compared with the prior art, the application has the following beneficial effects:

[0036] (1) The application uses the DOM tree to retain the original paragraph structure of the document, and inserts the translation according to the original paragraph structure, so that the translation can retain the original paragraph structure of the document.

[0037] (2) The application uses the bisection method to match the number of lines in the translation result and the number of lines in the original text, so that the paragraph format of the original document can be retained to the maximum extent while maintaining the accuracy of the translation result.

[0038] (3) The application divides the large-scale document into blocks and submits them to the large language model for translation, which can realize the processing of large files by the large language model. BRIEF DESCRIPTION OF DRAWINGS

[0039] Figure 1 The application provides a medical literature large-scale document adaptive block translation method, which comprises the following steps: DETAILED DESCRIPTION

[0040] The technical solutions in the embodiments of the application will be clearly and completely described below with reference to the drawings in the embodiments of the application. Obviously, the described embodiments are only part of the embodiments of the application, rather than all the embodiments of the application. Based on the embodiments in the application, all other embodiments obtained by those skilled in the art without creative work fall within the scope of protection of the application.

[0041] As shown in Figure 1 The application provides a medical literature large-scale document adaptive block translation method, which comprises the following steps:

[0042] Step S0, reading the medical literature, retaining the original paragraph structure of the document;

[0043] Step S1, adaptively blocking the document according to the complexity and length of the document content of the medical literature;

[0044] Step S2, the segmented document content is submitted to a large language model for batch translation in parallel to obtain translated texts of the block contents; the large language model translation can adopt an existing large language model technology, only needs to send the segmented text to the large language model, and the large language model can return the translated text, the translation by using the large language model belongs to a relatively mature technology and does not belong to the content protected by the application, and therefore is not described in detail here.

[0045] Step S3, the translated texts corresponding to the block contents are inserted in combination with the original paragraph structure to obtain a full-text translation retaining the original paragraph structure of the document, and a dichotomy is adopted for line number matching in a case where the returned result of the large language model does not match the line number of the original document; the dichotomy is to find a best split point in the returned result of the large language model, so that the line number of the split result matches the line number of the original document. Each block is a collection of multiple paragraphs, and the paragraph numbers in each block are recorded in the program, such as "1", "2",..., and "99", so that the paragraph structure can be retained.

[0046] According to one specific embodiment of the application, in step S3, the specific steps of line number matching by using the dichotomy are as follows:

[0047] Step S301, a start index low and an end index high are set, where the start index low is 0, and the end index high is the line number of the returned result of the large language model minus 1;

[0048] Step S302, a split point mid = (low + high) / 2 is calculated;

[0049] Step S303, the returned result of the large language model is split from the start index low to the split point mid to obtain a subset output_subset;

[0050] Step S304, the line number output_lines of the subset output_subset is calculated, and the output_lines is compared with the line number original_lines of the original document:

[0051] If the output_lines is greater than the original_lines, it is indicated that the split point mid is too large, the end index high is updated to mid - 1, if the low<= high, the step S302 is jumped to; if the low> high, the low or the high is selected as the final split point according to actual requirements, and the process is ended;

[0052] If output_lines is less than original_lines, it means that the split point mid is too small, update the start index low to mid + 1, if low<= high, jump to step S302; if low> high, select low or high as the final split point according to actual needs, end the process;

[0053] If output_lines is equal to original_lines, the best split point is found; end the process.

[0054] According to one specific embodiment of the present application, if low> high at the end of the loop, select the one closer to the original line number in low and high as the split point.

[0055] According to one specific embodiment of the present application, if low> high at the end of the loop, select the best split point according to the reasonableness of the semantics determined by low and high.

[0056] According to one specific embodiment of the present application, in step S1, according to the document chapter change, the same chapter is divided into a block; at the same time, using any one of the number of characters, the number of words, the number of sentences as the length index of the document, the document is divided into multiple blocks according to the pre-set length threshold, wherein the content of the same paragraph is kept in the same block.

[0057] According to one specific embodiment of the present application, step S0 includes the following steps:

[0058] Step S001, using python-lxml library to parse the input document, extracting text blocks and merging them into paragraphs;

[0059] Step S002, using DOM tree structure, saving the original topological relationship of paragraphs, tables, pictures and cross-references in the document.

[0060] According to one specific embodiment of the present application, step S001 specifically includes the following steps:

[0061] Step S0011, using Python's zipfile library to decompress the docx file, and then reading the content of word / document.xml, which contains the text and format information of the document;

[0062] Step S0012, using xml.etree.ElementTree library to parse word / document.xml, traversing the XML tree, finding the element <w:p>and <w:t>wherein, <w:p>Element represents a paragraph, <w:t>The element contains specific text content.

[0063] Step S0013, extracting <w:t>text in the element, and according to <w:p>The hierarchical relationship of the elements combines the texts into paragraphs;

[0064] In step S0014, the extracted texts belonging to one paragraph are merged into a paragraph list, and each paragraph corresponds to a paragraph list.

[0065] According to one specific embodiment of the present application, step S002 specifically comprises the following steps:

[0066] In step S0021, the XML tree structure of the document is mapped into a DOM tree, and each paragraph, table, picture and cross-reference is represented as a node in the DOM tree, and each node comprises a node type, content, attribute and child node;

[0067] In step S0022, when parsing word / document.xml, the DOM tree is constructed according to the nested relationship of the document structure, the whole document is taken as a root node, the parent-child relationship is determined according to the nested relationship of the document structure, and the parallel relationship under the same parent node is the sibling relationship;

[0068] In step S0023, the merged paragraph list is output to the DOM tree according to the structure of the DOM tree.

[0069] According to one specific embodiment of the present application, in step S3, after a block translation result is returned, a candidate <w:t>Node, emptying the contents of the node, inserting the translation.

[0070] The application provides a medical literature large-scale document adaptive block translation system, comprising a processor capable of executing a computer program, which can implement the medical literature large-scale document adaptive block translation method described above when executed.

[0071] Embodiment 1

[0072] The application provides a medical literature large-scale document adaptive block translation method, comprising the following steps:

[0073] Step S0, reading medical literature, preserving the original paragraph structure of the document;

[0074] Step S1, document adaptive block according to the complexity of the document content and the document length of the medical literature;

[0075] Step S2, the document content after block is submitted to the large language model for batch translation in parallel, and the translation of each block content is obtained; the large language model translation can adopt the existing large language model technology, only needs to send the text after block to the large language model, and the large language model can return the translated text, the translation by the large language model belongs to a relatively mature technology, and does not belong to the content protected by the application, which will not be described in detail here.

[0076] Step S3, inserting the translation corresponding to each block content in combination with the original paragraph structure to obtain the full-text translation preserving the original paragraph structure of the document; in the case that the number of rows of the large language model return result does not match the number of rows of the original document, the dichotomy is used for row number matching; the dichotomy is to find a best split point in the large language model return result, so that the number of rows after split matches the number of rows of the original document.

[0077] Embodiment 2

[0078] The application provides a medical literature large-scale document adaptive block translation method, comprising the following steps:

[0079] Step S0, reading medical literature, preserving the original paragraph structure of the document;

[0080] The step S0 comprises the following steps:

[0081] Step S001, using the python-lxml library to parse the input document, extracting the text block and merging it into a paragraph;

[0082] Step S002, using the DOM tree structure, saving the original topological relationship in the document of the paragraph, table, picture and cross reference.

[0083] Further, the step S001 specifically comprises the following steps:

[0084] In step S0011, the zipfile library of Python is used to decompress the docx file, and then the content of word / document.xml is read, which contains the text and format information of the document.

[0085] In step S0012, the xml.etree.ElementTree library is used to parse word / document.xml, and the XML tree is traversed to find the element <w:p>and <w:t>wherein, <w:p>Element represents a paragraph, <w:t>The element contains specific text content.

[0086] Step S0013, extracting <w:t>text in the element, and according to <w:p>The hierarchical relationship of elements combines the text into paragraphs;

[0087] In step S0014, the extracted text belonging to one paragraph is merged into a paragraph list, and each paragraph corresponds to a paragraph list.

[0088] Further, step S002 specifically includes the following steps:

[0089] In step S0021, the XML tree structure of the document is mapped into a DOM tree, and each paragraph, table, picture and cross-reference is represented as a node in the DOM tree, and each node includes node type, content, attribute and child node.

[0090] In step S0022, when parsing word / document.xml, the DOM tree is constructed according to the nesting relationship of the document structure, the entire document is taken as the root node, the parent-child relationship is determined according to the nesting relationship of the document structure, and the parallel relationship under the same parent node is the sibling relationship.

[0091] In step S0023, the merged paragraph list is output to the DOM tree according to the structure of the DOM tree.

[0092] In step S1, the document is adaptively segmented according to the complexity of the document content and the length of the document. In this embodiment, the document is segmented according to the chapter change, and the same chapter is divided into a block. At the same time, any one of the number of characters, the number of words and the number of sentences is used as the length indicator of the document, and the document is divided into multiple blocks according to the pre-set length threshold, wherein the contents of the same paragraph are kept in the same block.

[0093] In step S2, the segmented document content is submitted to the large language model for batch translation to obtain the translation of each block content. The large language model translation can use existing large language model technology, which only needs to send the segmented text to the large language model, and the large language model can return the translated text. The translation using the large language model is a relatively mature technology and does not belong to the protection content of the present application, which will not be described in detail here.

[0094] In step S3, the translated text corresponding to each block content is inserted in combination with the original paragraph structure to obtain the full text translation preserving the original paragraph structure of the document. In the case that the number of rows of the large language model return result does not match the number of rows of the original document, a dichotomy method is used for row number matching. The dichotomy method is to find a best split point in the large language model return result, so that the number of rows after splitting matches the number of rows of the original document.

[0095] Further, in step S3, the dichotomy method for row number matching specifically includes the following steps:

[0096] Step S301, set a start index low and an end index high, wherein the start index low is 0, and the end index high is the number of lines of the result returned by the large language model minus 1;

[0097] Step S302, calculate a split point mid = (low + high) / 2;

[0098] Step S303, split the result returned by the large language model from the start index low to the split point mid to obtain a subset output subset;

[0099] Step S304, calculate the number of lines of the subset output subset, and compare output lines with the number of lines of the original text original lines;

[0100] If output lines is greater than original lines, it means that the split point mid is too large, and the end index high is updated to mid - 1. If low<= high, jump to step S302; if low> high, select low or high as the final split point according to actual needs, and end the process;

[0101] If output lines is less than original lines, it means that the split point mid is too small, and the start index low is updated to mid + 1. If low<= high, jump to step S302; if low> high, select low or high as the final split point according to actual needs, and end the process;

[0102] If output lines is equal to original lines, the best split point is found; end the process.

[0103] Further, if low> high at the end of the loop, select the one closer to the number of lines of the original text from low and high as the split point.

[0104] Further, if low> high at the end of the loop, select the best split point according to the rationality of the semantics determined by low and high.

[0105] Further, in step S3, after a block translation result is returned, a candidate <w:t>The node is emptied, the translation is inserted, and the translation of all blocks is inserted after which the full translation is obtained, preserving the original structure of the document.

[0106] The above descriptions are only the preferred embodiments of the present application, not intended to limit the present application. Any modification, equivalent replacement and improvement made without departing from the spirit and principle of the present application shall fall within the scope of protection of the present application.< / w:t> < / w:p> < / w:t> < / w:t> < / w:p> < / w:t> < / w:p> < / w:t> < / w:p> < / w:t> < / w:t> < / w:p> < / w:t> < / w:p> < / w:t> < / w:p> < / w:t> < / w:t> < / w:p> < / w:t> < / w:p>

Claims

1. A method for adaptive chunking translation of large-scale medical literature documents, characterized in that, Includes the following steps: Step S0: Read the medical literature and preserve the original paragraph structure of the document; Step S1: Perform adaptive document segmentation based on the complexity and length of the medical literature's content. Step S2: Submit the segmented document content concurrently to the large language model for batch translation to obtain the translation of each segment. Step S3: Combine the original paragraph structure with the corresponding translation of each block of content to obtain the full text translation that retains the original paragraph structure of the document. For cases where the number of lines returned by the large language model does not match the number of lines in the original document, a binary search method is used to match the number of lines. The binary search method is to find an optimal split point in the result returned by the large language model so that the number of lines in the split result matches the number of lines in the original document. In step S3, the specific steps for row number matching using the binary search method are as follows: Step S301: Set the starting index low and the ending index high, where the starting index low is 0 and the ending index high is the number of rows returned by the large language model - 1; Step S302: Calculate the segmentation point mid = (low + high) / 2; Step S303: The results returned by the large language model are split from the starting index low to the split point mid to obtain a subset output_subset; Step S304: Calculate the number of lines in the subset `output_subset` (`output_lines`), and compare `output_lines` with the number of lines in the original text (`original_lines`). If output_lines is greater than original_lines, it means that the split point mid is too large. Update the end index high to mid - 1. If low <= high, jump to step S302. If low > high, select low or high as the final split point according to the actual needs and end the process. If output_lines is less than original_lines, it means that the split point mid is too small. Update the starting index low to mid + 1. If low <= high, jump to step S302. If low > high, select low or high as the final split point according to the actual needs and end the process. If output_lines equals original_lines, then the optimal split point has been found; the process ends.

2. The adaptive chunking translation method for large-scale medical literature documents according to claim 1, characterized in that, If low > high at the end of the loop, choose the line number of the original text that is closer to low or high as the split point.

3. The adaptive chunking translation method for large-scale medical literature documents according to claim 1, characterized in that, If low > high at the end of the loop, select the best split point based on the semantic rationality determined by low and high.

4. The adaptive chunking translation method for large-scale medical literature documents according to claim 1, characterized in that, In step S1, the document is divided into blocks according to the changes in the chapters, and the same chapter is divided into one block; at the same time, the document length is divided into multiple blocks according to a pre-set length threshold, using any one of the number of characters, the number of words, or the number of sentences as the document length indicator, and the content of the same paragraph is kept in the same block.

5. The adaptive chunking translation method for large-scale medical literature documents according to claim 1, characterized in that, Step S0 includes the following steps: Step S001: Use the python-lxml library to parse the input document, extract text blocks and merge them into paragraphs; Step S002: Using the DOM tree structure, save the original topological relationships in the document for paragraphs, tables, images, and cross-references.

6. The adaptive chunking translation method for large-scale medical literature documents according to claim 5, characterized in that, Step S001 specifically includes the following steps: Step S0011: Use Python's zipfile library to decompress the docx file, and then read the contents of word / document.xml, which contains the document's text and formatting information; Step S0012: Use the xml.etree.ElementTree library to parse word / document.xml, traverse the XML tree, and find the elements. <w:p>and <w:t>,in, <w:p>The element represents a paragraph. <w:t> The element contains specific text content;< / w:t> < / w:p> < / w:t> < / w:p> Step S0013, Extract <w:t>The text in the element, and according to <w:p> The hierarchical relationship of elements groups text into paragraphs;< / w:p> < / w:t> Step S0014: Merge the extracted text belonging to a paragraph into a paragraph list, with each paragraph corresponding to a paragraph list.

7. The adaptive chunking translation method for large-scale medical literature documents according to claim 6, characterized in that, Step S002 specifically includes the following steps: Step S0021: Map the XML tree structure of the document to a DOM tree. Each paragraph, table, image, and cross-reference is represented as a node in the DOM tree. Each node includes node type, content, attributes, and child nodes. Step S0022: When parsing word / document.xml, a DOM tree is constructed based on the nesting relationship of the document structure. The entire document is taken as the root node, and the parent-child relationship is determined based on the nesting relationship of the document structure. Parallel relationships under the same parent node are sibling relationships. Step S0023: Output the merged paragraph list to the DOM tree according to the DOM tree structure.

8. The adaptive chunking translation method for large-scale medical literature documents according to claim 7, characterized in that, In step S3, after a block translation result is returned, a candidate is selected sequentially from the DOM tree. <w:t> Clear the content of the node and insert the translation.< / w:t> 9. A large-scale adaptive chunking translation system for medical literature, characterized in that, The system includes a processor capable of executing a computer program that, when executed, implements the adaptive chunking translation method for large-scale medical literature documents as described in any one of claims 1-8.

Images