Document analysis method and system based on open source component and storage medium thereof

Abstract

The invention discloses a document analysis method and system based on an open source component and a storage medium thereof, and relates to the technical field of document processing, and the method comprises the following steps: judging a file format of a received original file through a dual detection mechanism based on a file magic number and a file extension name, and selecting and executing a differentiated analysis process according to the file format, the method comprises the following steps: acquiring structured table data and non-table content data, fusing the two parts of data to form a final document, performing structured optimization on the document subjected to file analysis to generate optimized output content convenient for vectorization segmentation and semantic retrieval, and performing intelligent format detection and differential analysis technologies to obtain an optimized output content. According to the method, a complex table structure and merged cells of a document are precisely restored, a placeholder fusion mechanism and RAG special optimization are adopted, a mixed format integrating a standardized title and keeping structural semantics is output, and vector retrieval relevance and LLM knowledge understanding accuracy are remarkably improved.

Description

Technical Field

[0001] This invention relates to the field of document processing technology, specifically to a document parsing method, system, and storage medium based on open-source components. Background Technology

[0002] Markdown is a lightweight markup language that has been widely used in technical documentation, blog publishing, and note-taking due to its concise syntax and good compatibility. Its advantages include ease of reading and writing, the ability to store in plain text, and ease of version control and cross-platform editing. With the popularity of Markdown, the demand for converting other document formats (such as Word documents, PDFs, etc.) to Markdown is growing, which has spurred the development of related technologies.

[0003] Currently, the following technical challenges exist when converting Word documents (DOC / DOCX) to Markdown format:

[0004] 1. Inaccurate table parsing: Existing document parsing tools (such as Apache Tika) have serious deficiencies in handling complex table structures, especially in the recognition and processing of merged cells.

[0005] 2. Poor format compatibility: DOC and DOCX formats require different parsing strategies, and existing solutions often can only handle one format.

[0006] 3. Disordered content: During the parsing process, the relative positions of the document's original structure (headings, paragraphs, tables, images) are often disrupted.

[0007] 4. Low table output quality: Traditional methods output tables with non-standard formats and cannot correctly handle merged cells that span multiple rows and columns. Summary of the Invention

[0008] The purpose of this invention is to provide a document parsing method, system, and storage medium based on open-source components, which solves problems such as inaccurate table processing, poor format compatibility, and disordered content order in existing Word document parsing technologies. In particular, it provides a low-cost, high-quality document parsing solution that does not require GPU hardware support, specifically targeting the knowledge processing needs of current RAG (Retrieval Enhanced Generation) and LLM (Large Language Model) systems. This provides an economical and efficient technical solution for the knowledge base construction of AI systems and the digital transformation of documents, thereby solving the problems mentioned in the background.

[0009] To achieve the above objectives, the present invention provides the following technical solution: a document parsing method based on open-source components, comprising the following steps:

[0010] File format detection uses a dual detection mechanism based on the file magic number and file extension to determine the file format of the received raw file. The file format includes a first format and a second format.

[0011] The file parsing process selects and executes a differentiated parsing process according to the file format to obtain structured tabular data and non-tabular content data. This includes using a first table parser based on XML parsing to extract tabular data from the original file of the first format, and calling a first full-text parser to process the non-tabular content. At the same time, the parsed tabular data is inserted into the processed non-tabular content data through a placeholder replacement mechanism to form a fused file. For the original file of the second format, a second table parser based on content similarity and positional relationship heuristic inference is used to extract its tabular data, and the second full-text parser is called to process the non-tabular content respectively.

[0012] RAG optimization output performs structural optimization on the parsed document, including standardizing the heading hierarchy and preserving the HTML format of tables, to generate optimized output content that is easy for vectorized segmentation and semantic retrieval.

[0013] Preferably, the dual detection mechanism prioritizes the detection method based on the file magic number. If the file format cannot be identified by the file magic number, a backup detection mechanism based on the file extension is activated to determine the file format.

[0014] Preferably, the first format is OOXML format. The first table parser accurately determines the vertical merging status of cells by parsing the vMerge attribute value in the underlying XML element of the file. Specifically, if the cell below is a continuation of a cell above, it means that it belongs to the merging area of ​​the current starting cell, and the merging row count is incremented by one. If it is not a continuation of a cell, the merging is terminated.

[0015] Preferably, the second format is OLE2 format, and the second table parser infers the horizontal and vertical merging status of cells by analyzing the content similarity and null value relationship of adjacent cells;

[0016] For horizontal merging, iterate through the adjacent cells on the right. If the text in the current cell is not empty and the text in the cell on the right is empty or has similar content, then it is determined to be a horizontal merge.

[0017] For vertical merging, iterate through the adjacent cells below. If the text in the current cell is not empty and the text in the cell below is empty or has similar content, then it is determined to be a vertical merge.

[0018] Preferably, the table data extracted by the first table parser and the second table parser is converted into HTML table data by an HTML table builder. The HTML table builder is configured to dynamically generate HTML table cell tags containing row span and column span attributes based on cell information objects containing row span and column span attributes.

[0019] Preferably, the HTML table builder also includes uniformly applying standardized inline styles to the generated cell tags to ensure the standardization and consistency of table display.

[0020] Preferably, the placeholder replacement mechanism includes matching the original table content in the non-table content processed by the first full-text parser using regular expressions, uniformly replacing it with a predefined placeholder, sequentially traversing the customized table HTML list generated by the parser, and replacing the first occurrence of the placeholder comment in the full-text content with each table HTML in turn.

[0021] Preferably, the standardized heading hierarchy includes a standard format that uniformly standardizes headings to contain only one space between the heading symbol and the heading text.

[0022] To address the aforementioned technical problems, this invention also provides a document parsing system based on open-source components, comprising:

[0023] Memory, used to store computer programs;

[0024] A processor for executing the computer program, which, when executed by the processor, implements the steps of a document parsing method based on open-source components as described in any of the preceding claims.

[0025] To address the aforementioned technical problems, the present invention also provides a readable storage medium having a computer program stored thereon.

[0026] When the computer program is executed by a processor, it implements the steps of a document parsing method based on open-source components as described in any of the above.

[0027] In summary, the beneficial effects of this invention are:

[0028] High processing accuracy and strong information fidelity: Targeting the characteristics of the two mainstream DOCX and DOC formats, a dual-track table processing scheme based on XML attribute parsing and heuristic content inference is adopted respectively. It can restore complex table structures with extremely high accuracy, especially merged cells that are difficult to handle, ensuring the integrity and accuracy of information extracted from the document.

[0029] The system is robust and has excellent compatibility: the intelligent format detection adopts a dual-insurance strategy of "main detection (magic number) + backup mechanism (extension)" to effectively deal with abnormal situations such as file corruption or non-standard naming, and ensure the stability and reliability of the system when processing documents from various sources.

[0030] The processing workflow is highly intelligent and automated: the entire process achieves fully automated processing from format recognition, content parsing, content fusion to output optimization, without the need for manual intervention, which greatly improves processing efficiency and is suitable for large-scale document batch processing scenarios.

[0031] The output results significantly improve downstream AI applications:

[0032] Standardized titles optimize text semantic segmentation, improving the quality of embedded vectors and retrieval relevance.

[0033] HTML table format preserves the structured semantics of tables to the greatest extent, providing a clear and accurate data foundation for large language models (LLMs), significantly reducing model comprehension bias and the risk of "illusions," thereby greatly enhancing the knowledge utilization efficiency and the accuracy of the final question answer in the RAG system.

[0034] The architecture is flexible and highly scalable: each processing module (such as format detection, dedicated parser, and placeholder replacement) has a single responsibility and collaborates through standardized interfaces, making the system easy to maintain and upgrade. For example, supporting new document formats in the future only requires adding a new parsing branch without changing the overall architecture.

[0035] The present invention also provides a document parsing system based on open source components and its storage medium, which has the above-mentioned beneficial effects, and will not be elaborated further here. Attached Figure Description

[0036] To more clearly illustrate the technical solutions in the embodiments of the invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0037] Figure 1 This is a schematic diagram of the overall process framework of a document parsing method based on open-source components according to the present invention;

[0038] Figure 2 This is a schematic diagram illustrating the accurate restoration and parsing of cross-page tables and cells by merging in an embodiment of a document parsing method based on open-source components according to the present invention.

[0039] Figure 3This is a schematic diagram illustrating the accurate restoration of cross-page horizontal and vertical table merging in an embodiment of a document parsing method based on open-source components according to the present invention. Detailed Implementation

[0040] The present invention will now be described in further detail with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without creative effort are within the scope of protection of the present invention. These drawings are simplified schematic diagrams, which are only used to illustrate the basic structure of the present invention in a schematic manner, and therefore only show the components related to the present invention.

[0041] To facilitate understanding of the present invention, a more complete description of the invention will be given below with reference to the accompanying drawings, which illustrate several embodiments of the invention. However, the invention can be implemented in many different forms and is not limited to the embodiments described herein. Rather, these embodiments are provided so that the disclosure of the invention will be more thorough and complete.

[0042] All features disclosed in this specification, or all steps in all disclosed methods or processes, may be combined in any way, except for mutually exclusive features and / or steps.

[0043] Any feature disclosed in this specification (including any appended claims, abstract, and drawings) may be replaced by other equivalent or similar features for a similar purpose, unless specifically stated otherwise. That is, unless specifically stated otherwise, each feature is merely one example of a series of equivalent or similar features.

[0044] In this invention, unless otherwise explicitly specified and limited, the terms "installation," "connection," "linking," and "fixing," etc., should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral part; they can refer to a mechanical connection, a direct connection, or an indirect connection through an intermediate medium; they can refer to the internal communication of at least two elements or the interaction relationship of at least two elements, unless otherwise explicitly limited. Those skilled in the art can understand the specific meaning of the above terms in this invention according to the specific circumstances.

[0045] The following is combined with Figures 1-3 This invention provides a detailed description of one embodiment: a document parsing method based on open-source components, which is an intelligent parsing method for Word documents based on a dual-engine fusion architecture. It innovatively combines the general parsing capabilities of Apache Tika with the precise table processing capabilities of Apache POI, achieving unified high-quality parsing of DOC / DOCX formats. Specifically, it includes the following steps:

[0046] Step 1: Intelligent Format Detection and Distribution Mechanism

[0047] The original file input is subjected to a dual detection mechanism to determine the file format. This combines two detection methods to improve the accuracy and robustness of file format detection. The dual detection mechanism includes precise detection based on the file magic number and backup detection based on the file extension. The detection method based on the file magic number is used first. If the file format cannot be identified by the file magic number, the backup detection mechanism based on the file extension is used to determine the file format. The file formats include OOXML or OLE2.

[0048] For example, the code for a specific dual-detection mechanism is as follows:

[0049] FileMagic fileMagic = FileMagicUtils.checkMagic(wordFile);

[0050] if (fileMagic == FileMagic.UNKNOWN) {

[0051] / / Backup detection mechanism

[0052] String suffix = FileUtil.getSuffix(wordFile.getName());

[0053] if (StrUtil.equalsIgnoreCase(suffix, "docx")) {

[0054] fileMagic = FileMagic.OOXML;

[0055] } else if (StrUtil.equalsIgnoreCase(suffix, "doc")) {

[0056] fileMagic = FileMagic.OLE2;

[0057] }

[0058] }

[0059] The purpose of the code above is to determine, as accurately as possible, whether a file named wordFile is in a specific format, specifically the Microsoft Word format: .doc or .docx.

[0060] FileMagicUtils.checkMagic(wordFile): This line of code is the core. It calls a method of a utility class to determine the actual type of the file by reading the first few bytes of the file, i.e., the magic number.

[0061] The Magic Number is a specific sequence of bytes contained within a file, serving as a "file format identifier." For example, all PDF files begin with %PDF-. This method is highly reliable because it is based on the file content itself and is unaffected by filename manipulation.

[0062] FileMagic: This is an enumeration type that may contain values ​​such as OOXML (representing .docx, .xlsx, etc.), OLE2 (representing .doc, .xls, etc.), PDF, JPEG, UNKNOWN, etc.

[0063] fileMagic: This variable is used to receive detection results.

[0064] If the result is FileMagic.UNKNOWN, it means that the file format cannot be recognized by the magic number. This could be because:

[0065] The file is indeed in an unknown format;

[0066] The file is corrupted; the magic number head information is missing or incorrect.

[0067] FileMagicUtils' detection algorithm does not support this format.

[0068] When the core detection fails, the code uses a fallback solution—inferring the type by the filename suffix;

[0069] wordFile.getName(): Gets the full name of the file, such as "mydocument.docx".

[0070] FileUtil.getSuffix(...): A utility method for extracting the file extension from a filename, for example, extracting "docx" from "mydocuments.docx";

[0071] StrUtil.equalsIgnoreCase: Another utility method for comparing whether two strings are equal, ignoring case. For example, "DOCX" and "docx" are considered the same.

[0072] If the extension is docx, then set the file type to OOXML;

[0073] If the extension is .doc, then set the file type to OLE2;

[0074] If the file extension is neither docx nor doc, then fileMagic will retain the original UNKNOWN value.

[0075] First, the most reliable magic number detection method is used to directly read the file content for judgment, which fundamentally avoids misjudgment caused by users mistakenly changing the file extension, such as renaming a .jpg image to .docx.

[0076] High robustness: It takes into account various edge cases where magic number detection may fail. When the most reliable method fails, it does not give up directly, but provides a backup plan that, although less reliable, is very practical. This greatly enhances the program's ability to handle various abnormal situations.

[0077] Good user experience: For ordinary users, changing file extensions is a common operation. If a user accidentally changes a .docx file to .doc, the magic number check will find that it is actually still in OOXML format, which may correct the user's mistake and ensure that the file can be processed correctly. Conversely, if the magic number check fails, the fallback to extension check can also cover most normal usage scenarios.

[0078] Step 2: File parsing and processing

[0079] After determining the format of the input raw file, a differentiated parsing process is selected and executed according to the file format to obtain structured tabular data and non-tabular content data. This includes processing the first format file type set to OOXML and the second format file type set to OLE2.

[0080] For files identified as OOXML (first-order format), a table parser based on XML parsing is used to extract table data, while non-table content is processed using a full-text parser. For example, this applies to DOCX files.

[0081] First, WordTableParser is used for high-precision table parsing. The specialized parser delves into the structure of the decompressed DOCX file and accurately locates the XML file describing the table. Specialized table parsing can more accurately obtain complex information such as cell merging, border styles, row and column widths, and paragraph formatting within cells. General parsers may simplify or lose these details. By directly processing the underlying XML structure, it avoids errors or information loss that may be introduced by intermediate format conversion. The core goal of this step is to extract clean, structurally accurate table data that can be used for subsequent calculations or database storage.

[0082] Secondly, Tika is used to process the overall document structure. Apache Tika is a powerful content inspection and extraction toolkit that uses libraries such as Apache POI to parse DOCX files.

[0083] The Tika here is likely configured to use ToXMLContentHandler or a similar processor to convert the entire document into a structured XHTML / XML document.

[0084] Tika is responsible for handling all content outside of tables and converting it into a unified, structured markup language (such as HTML) to facilitate subsequent unified processing and rendering. WordTableParser specializes in complex tables, while Tika is responsible for the overall process. This division of labor ensures that the best parsing results can be obtained for both simple paragraphs and complex tables.

[0085] Ultimately, the entire document was converted into a hybrid: mostly HTML generated by Tika, with space reserved for tables.

[0086] Finally, intelligent replacement of table placeholders cleverly merges the results of the two parts into a final high-quality document. When Tika processes the overall document, when it encounters a table, it does not attempt to perfectly parse it, but simply generates a placeholder or a simplified version of the table. Then, it replaces the corresponding placeholder generated by Tika with the HTML code of the high-precision table parsed by WordTableParser in the first step. One specific placeholder replacement mechanism is as follows:

[0087] / / Smart table replacement algorithm private static String replaceTablesWithCustomHtmlList(String tikaContent, List <string>customTablesHtmlList) { / / 1. Remove the original table parsed by Tika and set a placeholder String finalContent = tikaContent.replaceAll("(?s)<table[^> ]*>.*?", "<!--TABLE_PLACEHOLDER --> / / 2. Replace each table sequentially for (int i = 0; i <customTablesHtmlList.size(); i++) { String cleanedTableHtml = cleanTableHtml(customTablesHtmlList.get(i)); if (finalContent.contains(" <!-- TABLE_PLACEHOLDER -->")) { finalContent = finalContent.replaceFirst("<!-- TABLE_PLACEHOLDER --> ", cleanedTableHtml);}} return finalContent;}

[0088] Remove all native tables from Tika and leave a unique tag for each table that is removed.

[0089] (?s)<table[^> ]*>.*?:

[0090] This is a regular expression used to match HTML elements.

[0091]

[0092] The tag and its entire content. `(?s)`: Allows the dot (.) to match all characters, including newline characters, ensuring a complete table spanning multiple lines is matched. `.*?`: Non-greedy mode, matches as few characters as possible until the first `.*?` is encountered. The closing tag ensures that multiple consecutive tables are correctly matched, rather than treating the entire document as one large table.

[0093] <!--TABLE_PLACEHOLDER--> :

[0094] This is the replaced placeholder. It's an HTML comment that's not visible during the final rendering, but can be precisely located by the program.

[0095] The tables parsed by Tika may have lost styles, simplified structures, or inaccurate handling of merged cells. We don't trust its table output, so we'll clear it all first.

[0096] Iterate through the list of high-quality tables generated by the dedicated parser and replace them in the placeholder positions in order.

[0097] customTablesHtmlList:

[0098] This list contains the HTML code of high-quality tables that were previously parsed by WordTableParser or DocTableParser and are arranged in the order they appear in the document.

[0099] cleanTableHtml(...):

[0100] This is a custom method used to perform final cleaning and optimization on the generated table HTML, such as removing unnecessary styles, standardizing attributes, etc., to ensure consistency in the output.

[0101] replaceFirst(...):

[0102] This is the most crucial operation. It finds the first occurrence of a placeholder and replaces it with high-quality table HTML from the current loop.

[0103] The order is crucial: it ensures that the placeholder for the first table in the document is replaced by the first high-quality table in the list, the second placeholder by the second table, and so on. This consistency in order is a prerequisite for the entire mechanism to function correctly.

[0104] By employing a "demolish first, rebuild later" strategy, all the inconsistent tables output by Tika were smoothed out and uniformly replaced with simple placeholders, providing a clean and unified foundation for subsequent processing. High-quality content generated by external professional tools was then precisely "injected" into the standardized foundation in a strict order.

[0105] The final output document combines the integrity of Tika's overall structure with the high-quality, high-fidelity tables provided by a dedicated parser, achieving a powerful synergy. This plug-and-play design allows for easy replacement or upgrading of the table parser in the future without affecting the overall document processing workflow.

[0106] It should be noted that in this embodiment, the table parsing process of DOCX requires the detection of merged cells. The principle is to directly parse the underlying XML structure attributes of the document, and accurately determine the vertical merging status of cells by parsing the vMerge attribute value in the underlying XML elements of the file. One type of merged cell detection code is as follows:

[0107] private static int calculateRowspan(XWPFTable table, intstartRowIndex, int cellIndex) { int rowspan = 1; for (int rowIndex =startRowIndex + 1; rowIndex < rows.size(); rowIndex++) { != null){ String vMergeValue = cell.getCTTc().getTcPr().getVMerge().getVal().toString(); if ("continue".equals(vMergeValue)) { rowspan++; continue;}}break;} return rowspan;}

[0108] It uses the Apache POI library to delve into the underlying layer, obtain the cell's (CTTc) attribute (TcPr), and directly read its vertical merge (VMerge) attribute.

[0109] The vMerge property has two possible values:

[0110] restart: This indicates that this is the starting cell for a vertically merged cell.

[0111] Continue: This indicates that the cell is a continuation of a previous restart cell; it is not a new cell itself.

[0112] If this attribute does not exist (null), it means that the cell has not participated in vertical merging.

[0113] During the detection process, if the vMerge property value of the cell below is "continue", it means that it belongs to the merged area of ​​the current starting cell.

[0114] rowspan++: Therefore, the number of rows to be merged is increased by one.

[0115] continue: Continues to inspect the cells in the same column of the next row.

[0116] The loop terminates once a cell's vMerge property is not "continue", because "restart" means a new merged region has begun.

[0117] DOCX is ZIP + XML: DOCX files use well-structured XML to describe all document elements. Merging cells is an explicit, documented attribute (w:vMerge), like a tag on the cell. As mentioned in the previous discussion, "using WordTableParser for high-precision table parsing," this code demonstrates this high-precision parsing. It directly reads the "standard answer," so the result is 100% accurate.

[0118] For second-format files identified as OLE2, a table parser based on heuristic inference of content similarity and positional relationships extracts its table data, while non-table content is processed using the corresponding full-text parser. For example, this applies to DOC files.

[0119] DOC files are the traditional OLE2 compound document format, a complex binary format. Parsing them is far more difficult than with DOCX files.

[0120] 1. Use DocTableParser to handle DOC format tables. Similar to the DOCX approach, use a parser specifically designed for binary DOC format. Based on the HWPF component of Apache POI, extract table data. The structure of binary table is more obscure and difficult to understand. The dedicated parser can better handle its complexity and ensure that as much table information as possible is extracted accurately.

[0121] 2. Use the enhanced DocMarkdownWithHtmlTableContentHandler to handle non-table content. This is a custom Tika ContentHandler that converts all content except tables to Markdown, while tables are converted to HTML.

[0122] Throughout the processing, the corresponding method of this ContentHandler will be triggered every time an element is encountered.

[0123] When encountering plain text / headings: it chooses to output as Markdown for simplicity and ease of vectorization.

[0124] When a table is encountered: it immediately interrupts the general process and calls the dedicated DocTableParser in real time.

[0125] DocTableParser analyzes the structure of the current table, intelligently infers and merges cells, and returns a high-quality HTML table string on the spot. ContentHandler receives this string and directly outputs it to the final result stream. The final output content stream is naturally a final form in which Markdown text and high-quality HTML tables are interspersed and spliced ​​together.

[0126] 3. The caching mechanism optimizes processing performance. Parsing, especially binary DOC format parsing, is a CPU-intensive and I / O-intensive operation. The caching mechanism stores the result of processing a file for the first time. When the same file is requested again, the system can directly return the cached result, avoiding repeated parsing processes. The response speed is extremely fast, significantly reducing the system's computation and I / O pressure. It is particularly suitable for high-concurrency or large file processing scenarios, allowing users to obtain the results of processed documents almost instantly.

[0127] It should be noted that in this embodiment, the table parsing process of the DOC requires the detection of merged cells. The principle is based on heuristic algorithms and content similarity to infer the horizontal and vertical merging status of cells by analyzing the content similarity and null value relationships of adjacent cells. One type of merged cell detection code is as follows:

[0128] / / Merge detection based on structural features private static MergeInfodetectMergeByStructure(Table table, int rowIndex, int cellIndex, String cellText, String[][] cellContents, int[] rowCellCounts, int maxCols) { / / Infer merged cells based on content similarity and positional relationships int colspan = 1; int rowspan = 1; / / Detect horizontal merges for (int nextCol = cellIndex + 1; nextCol < rowCellCounts[rowIndex]; nextCol++) { if (shouldMergeHorizontally(cellText, cellContents[rowIndex][nextCol])) { colspan++;} else { break;}} / / Detect vertical merges for(int nextRow = rowIndex + 1; nextRow < cellContents.length; nextRow++) { if(shouldMergeVertically(cellText, cellContents[nextRow][cellIndex])) { rowspan++;} else { break;}} return new MergeInfo(colspan, rowspan);}

[0129] String[][] cellContents: This is a two-dimensional array that has extracted the contents of the entire table and stored them in memory. cellContents[row][col] is the text content of the cell in row row and column col.

[0130] int[]rowCellCounts: The actual number of cells in each row, since DOC table rows may be irregular.

[0131] String cellText: The text of the starting cell currently being examined.

[0132] Colspan detection level pooling:

[0133] for (int nextCol = cellIndex + 1; ...): Iterates to the right, starting from the cell immediately adjacent to the current cell.

[0134] `shouldMergeHorizontally(cellText, cellContents[rowIndex][nextCol])`: This is a custom conditional function. Its logic is typically:

[0135] If the text (cellText) of the current cell is very long, and the content of the adjacent cell to the right is empty, then they have likely been merged.

[0136] Alternatively, if the text in the current cell is equal to the sum of the text in the cells to its right, it can also be inferred that the cells have been merged.

[0137] If it is determined that a merge is necessary, then colspan++.

[0138] Detect vertical merging (Rowspan):

[0139] for(int nextRow=rowIndex + 1; ...): Iterates downwards from the cell immediately below the current cell.

[0140] shouldMergeVertically(...): This is also a custom conditional function. The logic is similar:

[0141] If the cell below is empty, while the current cell above has content, they are likely to be vertically merged.

[0142] Alternatively, check if the content of the cell below is the same as or similar to that of the cell above.

[0143] If it is determined that rowspan should be merged, then rowspan++.

[0144] Step 3: RAG Optimization Output

[0145] The final document undergoes structural optimization, including standardizing the heading hierarchy and preserving the HTML format of tables, to generate optimized output content that facilitates vectorized segmentation and semantic retrieval. One specific operational logic is as follows:

[0146] / / Structured output optimized for the RAG system public class RAGOptimizedMarkdownBuilder{ public String buildStructuredContent(String content, List <string>tables){ StringBuilder result = new StringBuilder(); / / Preserve the heading hierarchy for easy vectorized segmentation result.append(normalizeHeadingStructure(content)); / / Tables are in HTML format to preserve structured information for (String table : tables) { result.append("\n\n").append(table).append("\n\n");} return result.toString();} / / Standardize the heading structure for optimized vector retrieval private String normalizeHeadingStructure(String content) { / / Ensure clear heading hierarchy for easy semantic segmentation by the RAG system return content.replaceAll("(#{1,6})\\s*","$1 ");}} The high-quality document content processed in the previous steps is further optimized to adapt to the working modes of RAG and LLM systems. Its core goal is to maximize knowledge retention and retrieval efficiency.

[0147] In terms of maintaining the title hierarchy structure and optimizing semantic segmentation and retrieval:

[0148] Because Markdown is converted from different sources, the heading format may be inconsistent, such as # heading, # heading, ## heading. This inconsistency can cause problems for the text segmentation process in the RAG system;

[0149] The `normalizeHeadingStructure` method uses regular expressions:

[0150] (#{1,6}): Matches 1-6 # characters, representing all levels of Markdown headings.

[0151] \\s*: Matches zero or more whitespace characters that may appear after the heading symbol.

[0152] "$1": Replace it with the standard format of "heading symbol + one space". Many advanced chunking strategies (such as splitting by heading) rely on a clear heading structure to divide the document into semantically coherent segments, and standardized headings ensure the accuracy and consistency of chunking.

[0153] A clear document structure helps the embedding model better understand the context and importance of each chunk, so that more relevant and complete knowledge fragments can be returned during retrieval, rather than fragmented information.

[0154] It is worth mentioning that in this embodiment, the parsed structured data, especially the merged cell information data, is converted into an HTML table with good visual effects presented to the user through an HTML table builder. The HTML table builder is configured to dynamically generate HTML table cell tags with row span and column span attributes according to a cell information object containing row - spanning and column - spanning attributes, and uniformly append standardized inline styles to ensure the standardization and consistency of table display. The principle of a specific running code is as follows:

[0155] / / Flexible HTML table building system public class HtmlTableBuilder { public String buildCell(CellInfo cellInfo) { StringBuilder html = new StringBuilder(); html.append(" <td"); / / Dynamically add merge attributes if (cellInfo.getColspan() > 1){ html.append(" colspan=\"").append(cellInfo.getColspan()).append("\"");} if (cellInfo.getRowspan() > 1) { html.append(" rowspan=\"").append(cellInfo.getRowspan()).append("\"");} / / Standardized style html.append(" style=\"border: 1px solid #ccc; padding: 8px;\">"); html.append(cellInfo.getText()); html.append("\n"); return html.toString();}} Only when the merge number is greater than 1 will the colspan or rowspan attribute be added. This is a very important optimization.

[0156] For the vast majority of ordinary cells without merging, the generated HTML is concise...

[0157] The merge attributes are added only when necessary, which makes the output code cleaner and more compliant with standards.

[0158] This method can handle any type of cell—whether it's a regular cell, a header cell merged across columns, or an item cell merged across rows. Its behavior is entirely driven by the input data, CellInfo, making it highly flexible and versatile.

[0159] Inline Styles: Here, CSS styles are added directly through the style property.

[0160] `border: 1px solid #ccc;`: Adds a light gray border to each cell. This is fundamental to table readability; regardless of whether the original document has borders, the final output will maintain a clear structure.

[0161] padding: 8px;: Adds inner padding between cell content and border, improving aesthetics and readability.

[0162] No matter how complex or inconsistent the styles of the original document are, the tables output to HTML will have a consistent, clean, and professional look. This provides a reliable user experience, reduces dependencies, and uses inline styles to reduce reliance on external CSS style sheets, making the generated HTML fragments easier to embed into any platform or system without style loss or conflicts.

[0163] Its output is entirely determined by the input CellInfo data object and can be paired with any parser, as long as that parser provides the correct data. This decouples it from the front-end parsing logic, making it highly flexible.

[0164] It elegantly handles the most complex merged cell problem in tables and simplifies it into simple attribute output, hiding the complexity of the underlying parsing.

[0165] It ensures that the final output HTML is not only structurally accurate, but also visually consistent and aesthetically pleasing, directly producing high-quality results that can be used for web page display.

[0166] In summary, the present invention has the following advantages:

[0167] Significantly improve the accuracy and completeness of information extraction: By adopting a format-differentiated parsing strategy (DOCX based on attribute detection, DOC based on content inference), the core challenge of merging cells in table processing has been overcome, ensuring that the table data extracted from complex documents has a complete structure and accurate content, fundamentally avoiding information loss or incorrect parsing.

[0168] The system's robustness and compatibility have been enhanced: the intelligent format detection mechanism, combined with dual-path detection (magic number + extension), can reliably handle documents from various sources, effectively cope with abnormal situations such as file corruption or incorrect extensions, and ensure the smoothness and stability of the processing flow.

[0169] The efficiency of downstream AI applications has been optimized: the final output is a content format (standardized title + HTML table) specifically optimized for RAG and LLM systems, which greatly improves the retrieval relevance and accuracy of knowledge fragments after vectorization, as well as the ability of large language models to understand and utilize table information, effectively reducing model "illusion" and laying a solid foundation for high-quality knowledge question answering and decision support.

[0170] It achieves a high degree of automation and intelligence in the processing flow: the entire process seamlessly integrates multiple parsing technologies, fusion mechanisms and optimization strategies, and can automatically complete the conversion from raw documents to high-quality knowledge carriers without human intervention, which significantly improves data processing efficiency and automation level.

[0171] The above describes in detail an embodiment of a document parsing method based on open-source components. Based on this, the present invention also discloses a document parsing system and storage medium based on open-source components corresponding to the above method.

[0172] A document parsing system based on open-source components, comprising:

[0173] Memory, used to store computer programs;

[0174] A processor is used to execute the computer program, which, when executed by the processor, is capable of implementing the relevant steps in the document parsing method based on open-source components disclosed in any of the foregoing embodiments.

[0175] The processor may include one or more processing cores, such as a core processor or a core processor. The processor can be implemented using at least one of the following hardware forms: Digital Signal Processing (DSP), Field-Programmable Gate Array (FPGA), or Programmable Logic Array (PLA). The processor may also include a main processor and coprocessors. The main processor, also known as the Central Processing Unit (CPU), is used to process data in the wake-up state; the coprocessor is a low-power processor used to process data in the standby state.

[0176] In some embodiments, the processor may integrate a Graphics Processing Unit (GPU) responsible for rendering and drawing the content to be displayed on the screen. In some embodiments, the processor may also include an Artificial Intelligence (AI) processor for handling computational operations related to machine learning.

[0177] The memory may include one or more readable storage media, which may be non-transitory. The memory may also include high-speed random access memory and non-volatile memory, such as one or more disk storage devices or flash memory devices. In this embodiment, the memory is used to store at least the following computer program, which, after being loaded and executed by the processor, is capable of implementing the relevant steps in the document parsing method based on open-source components disclosed in any of the foregoing embodiments. In addition, the resources stored in the memory may also include operating systems and data, and the storage method may be temporary or permanent storage. The operating system may be Windows. The data may include, but is not limited to, the data involved in the above methods.

[0178] Furthermore, the functional modules in the various embodiments of the present invention can be integrated into one processing module, or each module can exist physically separately, or two or more modules can be integrated into one module. The integrated modules can be implemented in hardware or as software functional modules. If the integrated module is implemented as a software functional module and sold or used as an independent product, it can be stored in a readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and executes all or part of the steps of the methods described in the various embodiments of the present invention.

[0179] To this end, embodiments of the present invention also provide a readable storage medium storing a computer program, which, when executed by a processor, implements steps such as those of a document parsing method based on open-source components.

[0180] The readable storage medium may include: USB flash drive, portable hard drive, read-only memory (ROM), random access memory (RAM), magnetic disk or optical disk, and other media that can store program code.

[0181] The computer program contained in the readable storage medium provided in this embodiment can implement the steps of a document parsing method based on open-source components as described above when executed by a processor, achieving the same effect.

[0182] The foregoing has provided a detailed description of a document parsing method, system, and storage medium based on open-source components provided by this invention. The various embodiments in the specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For the apparatus, devices, and readable storage media disclosed in the embodiments, since they correspond to the methods disclosed in the embodiments, the descriptions are relatively simple, and relevant parts can be referred to in the method section. It should be noted that those skilled in the art can make several improvements and modifications to this invention without departing from the principles of the invention, and these improvements and modifications also fall within the protection scope of the claims of this invention.

[0183] The above description is merely a specific embodiment of the invention, but the scope of protection of the invention is not limited thereto. Any variations or substitutions conceived without inventive effort should be included within the scope of protection of the invention. Therefore, the scope of protection of the invention should be determined by the scope defined in the claims.< / string> < / string>

Claims

1. A document parsing method based on open-source components, characterized in that: Includes the following steps: File format detection uses a dual detection mechanism based on the file magic number and file extension to determine the file format of the received raw file. The file format includes a first format and a second format. The file parsing process selects and executes differentiated parsing procedures based on the file format to obtain structured tabular data and non-tabular content data. This includes using a first table parser based on XML parsing to extract tabular data from the original file of the first format, and calling a first full-text parser to process the non-tabular content. At the same time, the parsed tabular data is inserted into the processed non-tabular content data through a placeholder replacement mechanism to form a fused file. For the original file of the second format, a second table parser based on content similarity and positional relationship heuristic inference is used to extract its tabular data. The non-tabular content is processed by calling a second full-text parser, and the fused file is directly output. RAG optimization output performs structural optimization on the parsed document, including standardizing the heading hierarchy and preserving the HTML format of tables, to generate optimized output content that is easy for vectorized segmentation and semantic retrieval.

2. The document parsing method based on open-source components according to claim 1, characterized in that: The dual detection mechanism prioritizes the detection method based on the file magic number. If the file format cannot be identified by the file magic number, a backup detection mechanism based on the file extension is activated to determine the file format.

3. The document parsing method based on open-source components according to claim 1, characterized in that: The first format is OOXML format. The first table parser accurately determines the vertical merging status of cells by parsing the vMerge attribute value in the underlying XML element of the file. Specifically, if the cell below is a continuation of a cell above, it means that it belongs to the merging area of ​​the current starting cell, and the merging row count is incremented by one. If it is not a continuation of a cell, the merging is terminated.

4. The document parsing method based on open-source components according to claim 1, characterized in that: The second format is OLE2 format. The second table parser infers the horizontal and vertical merging status of cells by analyzing the content similarity and null value relationship of adjacent cells. For horizontal merging, iterate through the adjacent cells on the right. If the text in the current cell is not empty and the text in the cell on the right is empty or has similar content, then it is determined to be a horizontal merge. For vertical merging, iterate through the adjacent cells below. If the text in the current cell is not empty and the text in the cell below is empty or has similar content, then it is determined to be a vertical merge.

5. The document parsing method based on open-source components according to claim 4, characterized in that: The table data extracted by the first table parser and the second table parser is converted into HTML table data by an HTML table builder. The HTML table builder is configured to dynamically generate HTML table cell tags containing row span and column span attributes based on cell information objects containing row span and column span attributes.

6. The document parsing method based on open-source components according to claim 5, characterized in that: The HTML table builder also includes uniformly applying standardized inline styles to the generated cell tags to ensure the standardization and consistency of table display.

7. The document parsing method based on open-source components according to claim 6, characterized in that: The placeholder replacement mechanism includes matching the original table content in the non-table content processed by the first full-text parser using regular expressions and uniformly replacing it with a predefined placeholder. The customized table HTML list generated by the parser is traversed sequentially, and each table HTML is used to replace the first placeholder comment that appears in the full-text content.

8. The document parsing method based on open-source components according to claim 1, characterized in that: The standardized heading hierarchy includes a standard format that unifies headings into a format where the heading symbol and the heading text contain one and only one space.

9. A document parsing system based on open-source components, characterized in that: include Memory, used to store computer programs; A processor for executing the computer program, wherein the computer program, when executed by the processor, implements the steps of the document parsing method based on open-source components as described in any one of claims 1-8.

10. A readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by the processor, it implements the steps of the document parsing method based on open-source components as described in any one of claims 1-8.

Images