Document analysis and enhancement generation method and related device

Abstract

The invention belongs to a document generation method, and provides a document analysis and enhancement generation method and a related device aiming at the technical problem that factual errors or information missing easily occur in generated contents when table contents are designed by an existing document retrieval enhancement generation technology. The method comprises the following steps: analyzing a document image through a layout detection model, identifying to obtain different functional regions of the document image, then constructing a document structure tree for representing a spatial and hierarchical relationship between the functional regions, respectively performing fine-grained semantic analysis on each functional region by using a visual language model, inputting semantic analysis into a large language model, and constructing a document structure tree for representing the spatial and hierarchical relationship between the functional regions; the corresponding natural language description is obtained, then the natural language description is coded into dense vectors, and all the dense vectors are constructed into a vector index database; according to the method, fine-grained layout detection, multi-modal structure understanding and structured language representation are introduced into a retrieval enhancement generation process, and an end-to-end analytic framework is formed.

Description

A document parsing and enhancement generation method and related apparatus Technical Field

[0001] This application pertains to a document generation method, specifically a document parsing and enhancement generation method and related apparatus. Background Technology

[0002] With the advent of the big data era, extracting information quickly and accurately from massive amounts of documents has become a key challenge. Retrieval-enhanced generative techniques significantly improve the professionalism and accuracy of responses from large language models by first retrieving relevant information from a knowledge base and then generating answers based on the retrieved context. However, existing technologies are primarily geared towards plain text and perform poorly on documents containing large amounts of complex, structured content.

[0003] Traditional methods typically rely on optical character recognition (OCR) technology to convert document images into plain text. This approach has significant drawbacks: First, the conversion process disrupts the inherent two-dimensional spatial structure and logical relationships of tables, such as merging cells and the hierarchical relationships of row and column headings, leading to the loss of semantic information. Second, the generated plain text sequence is difficult for retrieval models to understand effectively because retrieval models are usually based on bag-of-words models or shallow semantic matching, failing to grasp the complex relationships between numbers and table headers. When users query comparisons, summaries, or calculations involving table content, the correct context is often not retrieved, resulting in factual errors or missing information in the subsequently generated content. Summary of the Invention

[0004] This application addresses the technical problem that existing document retrieval enhancement generation technologies often result in factual errors or missing information in the generated table content. It provides a document parsing and enhancement generation method and related apparatus.

[0005] To achieve the above objectives, this application adopts the following technical solution: Firstly, this application proposes a document parsing and enhancement generation method, comprising: converting user-inputted natural language for querying into dense vectors; querying a vector index database using the dense vectors to obtain the most relevant multiple natural language descriptions and corresponding original document regions; combining the queried natural language descriptions with the user-inputted natural language for querying to obtain the final natural language answer; the method for constructing the vector index database comprises: parsing a document image using a layout detection model to identify different functional regions of the document image, and then constructing a document structure tree representing the spatial and hierarchical relationships between the functional regions; performing fine-grained semantic parsing on each functional region using a visual language model; inputting the semantic parsing into a large language model to obtain the corresponding natural language description; encoding the natural language descriptions into dense vectors, and then constructing a vector index database from all the dense vectors.

[0006] Furthermore, the different functional areas of the document image include text blocks, tables, graphics, and titles.

[0007] Furthermore, the method for constructing a document structure tree representing the spatial and hierarchical relationships between functional regions includes: obtaining the bounding box coordinates and category information of each functional region; normalizing the bounding box coordinates and grouping them according to spatial relationships; and constructing a document structure tree representing the spatial and hierarchical relationships between functional regions based on the normalization and spatial relationship grouping results.

[0008] Furthermore, the method for performing fine-grained semantic parsing of each functional region using a visual language model includes: extracting independent image blocks of each functional region from the document image based on the bounding box coordinates of each functional region; designing differentiated prompt words for different categories of functional regions to determine the parsing task of the visual language model; and outputting structured semantic parsing based on the independent image blocks of each functional region and the corresponding differentiated prompt words.

[0009] Furthermore, the method for encoding natural language descriptions into dense vectors includes: encoding natural language descriptions into dense vectors using a text embedding model.

[0010] Furthermore, the method for obtaining the final natural language answer by combining the natural language description obtained from the query with the natural language input by the user for the query includes: using the natural language description obtained from the query as context, and inputting it together with the natural language input by the user for the query into a generative large language model to obtain the final natural language answer.

[0011] Secondly, this application proposes a document parsing and enhancement generation system, comprising: a data module for converting user-inputted natural language for querying into dense vectors; a query module for querying a vector index database using the dense vectors to obtain the most relevant natural language descriptions and corresponding original document regions; and a generation module for combining the natural language descriptions obtained from the query with the user-inputted natural language to obtain the final natural language answer. The method for constructing the vector index database includes: parsing a document image using a layout detection model to identify different functional regions of the document image, and then constructing a document structure tree representing the spatial and hierarchical relationships between the functional regions; performing fine-grained semantic parsing on each functional region using a visual language model; inputting the semantic parsing into a large language model to obtain the corresponding natural language description; encoding the natural language descriptions into dense vectors, and then constructing all the dense vectors into a vector index database.

[0012] Thirdly, this application proposes an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of the above-described document parsing and enhancement generation method.

[0013] Fourthly, this application proposes a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps of the above-described document parsing and enhancement generation method.

[0014] Fifthly, this application proposes a computer program product, which includes computer instructions. A processor of a computer device reads the computer instructions and executes the computer instructions to implement the steps of the above-described document parsing and enhancement generation method.

[0015] Compared with existing technologies, this application has the following advantages: This application proposes a document parsing and enhancement generation method. When constructing a vector index database, a layout detection model is used to parse the document image, identify different functional regions of the document image, and then construct a document structure tree representing the spatial and hierarchical relationships between functional regions. Fine-grained semantic parsing is performed on each functional region using a visual language model. The semantic parsing is input into a large language model to obtain the corresponding natural language description, which is then encoded into dense vectors. All dense vectors are used to construct a vector index database. This application introduces fine-grained layout detection, multimodal structure understanding, and structured language representation into the retrieval enhancement generation process, forming an end-to-end parsing framework. It solves the problem of structural information loss when traditional methods process complex tabular documents. By transforming the visual layout into machine-readable structured data, and then into natural language descriptions that are friendly to large language models, this application bridges the semantic space of structured data and natural language, making retrieval more accurate and the generated content more faithful to the original text. It is particularly suitable for scenarios such as financial analysis and academic research that require in-depth queries of tabular data.

[0016] This application also proposes a document parsing and enhancement generation system, an electronic device, a computer-readable storage medium, and a computer program product, which possess all the advantages of the aforementioned document parsing and enhancement generation methods. Attached Figure Description

[0017] To more clearly illustrate the technical solutions of the embodiments of this application, the accompanying drawings used in the embodiments will be briefly introduced below. It should be understood that the following drawings only show some embodiments of this application and should not be regarded as a limitation of the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0018] Figure 1 is a flowchart illustrating one of the document parsing and enhancement generation methods of this application.

[0019] Figure 2 is a schematic diagram of the second process of document parsing and enhancement generation method of this application.

[0020] Figure 3 is a schematic diagram of the document parsing and enhancement generation system of this application. Detailed Implementation

[0021] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. The components of the embodiments of this application described and shown in the accompanying drawings can generally be arranged and designed in various different configurations.

[0022] Therefore, the following detailed description of the embodiments of this application provided in the accompanying drawings is not intended to limit the scope of the claimed application, but merely to illustrate selected embodiments of the application. All other embodiments obtained by those skilled in the art based on the embodiments of this application without inventive effort are within the scope of protection of this application.

[0023] It should be noted that similar labels and letters in the following figures indicate similar items. Therefore, once an item is defined in one figure, it does not need to be further defined and explained in subsequent figures.

[0024] In the description of the embodiments of this application, it should be noted that if terms such as "upper," "lower," "horizontal," or "inner" indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings, or the orientation or positional relationship commonly used when the product of the invention is in use, they are only for the convenience of describing this application and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation, and therefore should not be construed as a limitation on this application. In addition, terms such as "first" and "second" are only used to distinguish descriptions and should not be construed as indicating or implying relative importance.

[0025] Furthermore, the use of the term "horizontal" does not imply that the component must be absolutely horizontal, but rather that it can be slightly tilted. For example, "horizontal" simply means that its direction is more horizontal than "vertical," and does not mean that the structure must be completely horizontal, but can be slightly tilted.

[0026] In the description of the embodiments of this application, it should also be noted that, unless otherwise explicitly specified and limited, the terms "set," "install," "connect," and "link" should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral connection; they can refer to a mechanical connection or an electrical connection; they can refer to a direct connection or an indirect connection through an intermediate medium; and they can refer to the internal connection of two components. Those skilled in the art can understand the specific meaning of the above terms in this application according to the specific circumstances.

[0027] In the interdisciplinary fields of natural language processing and computer vision, with the full arrival of the big data era, the efficient utilization of massive amounts of document information has become a crucial support for the development of various industries. This is especially true in core areas such as financial analysis, academic research, and business decision-making, where the efficiency and accuracy of document information extraction directly impact work effectiveness. Currently, with the development of large-scale language model technology, retrieval-enhanced generation technology has gradually become the mainstream solution for solving the problem of extracting massive amounts of document information. This technology first retrieves and queries relevant information from a knowledge base, and then generates answers based on the retrieved context, effectively improving the professionalism and accuracy of large-scale language model responses. It has been widely applied in pure text document information extraction scenarios.

[0028] In applications such as financial analysis and academic research that rely on complex tabular documents, existing information extraction methods face numerous unresolved issues. On one hand, core data in these scenarios is often stored in tabular form, containing complex two-dimensional spatial structures such as merged cells and multi-level row and column headers. These structures are crucial for maintaining logical connections between the data. However, current information extraction methods struggle to accurately preserve this structural information, leading to the disruption of semantic relationships within the data. On the other hand, user queries in these scenarios often involve data comparison, summarization, or calculation. These needs require the system to accurately understand the complex relationships between numbers and headers, and between numbers themselves. Current information extraction methods cannot effectively support these in-depth query needs, often resulting in incomplete information extraction and misaligned data correspondences, impacting subsequent decision-making or research.

[0029] To address the information extraction challenges of complex tabular documents, existing technologies generally employ a "Optical Character Recognition (OCR) + Plain Text Retrieval Enhancement Generation" approach. Specifically, firstly, OCR technology converts the document image containing the complex table into plain text format. The core purpose of this step is to transform image-based documents that cannot be directly retrieved into processable text data. Subsequently, the converted plain text is input into a retrieval enhancement generation system. The system uses a retrieval model to retrieve content relevant to the user's query from a plain text document library. The retrieved content is then used as context input to a large language model to generate the final answer. Furthermore, to optimize the performance of the retrieval model, existing technologies also attempt to use traditional text representation methods such as the Bag of Words (BOW) model and TF-IDF, or algorithms based on shallow semantic matching, to improve the efficiency and relevance of plain text retrieval, thereby enhancing the information extraction effect from complex tabular documents.

[0030] While existing technologies attempt to address the information extraction problem from complex tabular documents through optical character recognition (OCR) and enhanced generation of plain text retrieval, they still suffer from fundamental flaws that fail to meet the in-depth needs of scenarios such as financial analysis and academic research. First, OCR technology, in converting document images into plain text, completely destroys the inherent two-dimensional spatial structure and logical relationships of tables. For complex structures such as merged cells and multi-level row and column headings, it often only converts them into disordered text sequences, resulting in the loss of crucial semantic information. Second, plain text-based retrieval models have limitations in semantic understanding. Neither traditional bag-of-words models nor shallow semantic matching algorithms can effectively capture the complex relationships hidden in tabular data, such as the attribution relationship between numbers and corresponding table headers, or the comparison or summarization logic between different rows and columns. This makes it difficult for retrieval models to accurately locate the correct contextual information when users submit queries involving data comparison, summarization, or calculation. Finally, deviations in the retrieval process directly lead to factual errors or missing key information in the answers generated by subsequent large-scale language models, such as confusing different categories of data, inaccurate calculation results, or even generating content that does not match the original data. In summary, existing technologies cannot effectively process documents containing complex structured content, especially in terms of preserving the structure of tabular data, understanding semantics, and retrieving accurate information. They are insufficient to meet the needs of in-depth querying of tabular data in scenarios such as financial analysis and academic research. There is an urgent need for a technical solution that can accurately preserve the complex structure of documents and deeply understand semantic relationships to break through the existing bottlenecks.

[0031] Based on the above, this application proposes a document parsing and enhancement generation method and related apparatus. The following describes this application in further detail with reference to embodiments and accompanying drawings.

[0032] Figure 1 shows a flowchart of the document parsing and enhancement generation method of this application, which may include: S101, converting the natural language input by the user for querying into a dense vector.

[0033] To achieve semantic matching between user queries and text in the index, the user query can be converted into a dense vector with the same dimension and semantic space as the index vector, ensuring that the similarity between the two can be directly calculated. This guarantees that the user query and the index text are in the same semantic space, providing a foundation for the accuracy of subsequent similarity retrieval and avoiding semantic matching biases caused by model differences.

[0034] S102, by querying the vector index database using the dense vector, the most relevant natural language descriptions and corresponding original document regions are obtained.

[0035] This application uses a distance metric in a high-dimensional vector space to calculate the similarity between the user's query vector and all vectors in the index, selecting the K vectors with the highest similarity. The corresponding natural language descriptions and original document regions then constitute the most relevant contextual information for the query. This allows for rapid location of the most relevant document content, solving the problem of traditional retrieval methods struggling to accurately match complex tabular data associations, and providing high-quality context for subsequent enhancements.

[0036] S103, combine the natural language description obtained from the query with the natural language input by the user for the query to obtain the final natural language answer.

[0037] Large language models possess the ability to generate coherent text based on context. By combining user queries with relevant natural language descriptions retrieved, the model can generate accurate and faithful answers to the original text based on contextual information, avoiding unfounded information fabrication. It enables a leap from retrieving document fragments to outputting accurate answers. The answers directly address user needs and possess high fidelity due to being generated from original document information, thus solving the problem of factual errors that traditional generative models are prone to.

[0038] The method for constructing the vector index database includes: (1) parsing the document image through a layout detection model, identifying different functional regions of the document image, and then constructing a document structure tree that represents the spatial and hierarchical relationship between the functional regions.

[0039] It should be noted that different content modules in a document image have a fixed spatial distribution and hierarchical relationship. A pre-trained model can identify the boundaries and categories of these modules, and then a tree structure can be constructed based on their spatial relationships to achieve a structured representation of the document layout. This approach achieves the decomposition of a document image from a whole image to structured regions, fully preserving the spatial location and hierarchical relationship of each region. This provides accurate region delimitation for subsequent fine-grained analysis and avoids region confusion caused by direct conversion in traditional optical character recognition.

[0040] (2) Fine-grained semantic parsing is performed on each functional area using a visual language model.

[0041] It's important to note that a single visual model can only recognize the appearance of a region, and a single language model cannot process image information. A visual-language model, by fusing visual and linguistic features, can simultaneously understand the visual layout and semantic content of functional areas, achieving cross-modal conversion from image to structure and semantics. This overcomes the limitations of traditional OCR, which only extracts text characters, accurately capturing the structural information of complex layouts and converting visualized regional content into machine-understandable structured data, providing core data support for subsequent natural language description generation.

[0042] (3) Input the semantic parsing into the large language model to obtain the corresponding natural language description.

[0043] While structured data obtained through semantic parsing is machine-readable, it cannot be directly used for semantic retrieval. Large language models, with their powerful semantic integration and natural language generation capabilities, can convert structured data into text that conforms to human reading habits while preserving the logical connections of the original data. This establishes a semantic bridge between structured data and natural language; the generated text retains the structure and data accuracy of the original document while possessing excellent semantic readability, making it directly usable for subsequent text embedding and retrieval.

[0044] (4) Encode the natural language description into dense vectors, and then construct a vector index database from all the dense vectors.

[0045] The semantics of natural language text are difficult to accurately represent using traditional keyword matching. Text embedding models can convert text into high-dimensional dense vectors, and the distance between vectors in high-dimensional space corresponds to the semantic similarity of the text. Vector index databases optimize vector retrieval efficiency through a special index structure, enabling rapid matching of large-scale vectors. This achieves the quantification of semantic representation of natural language text, solving the problem that traditional keyword retrieval cannot capture deep semantic relationships. Vector index databases can guarantee millisecond-level response times for subsequent user queries, providing support for efficient retrieval of large-scale documents.

[0046] Figure 2 shows a schematic diagram of the second type of document parsing and enhancement generation method of this application, which may include: S201, layout detection and structure parsing.

[0047] A pre-trained layout detection model is used to parse the input document image and identify different functional regions within it. In this embodiment, different functional regions may include text blocks, tables, graphics, and titles, which are used for targeted parsing in subsequent multimodal semantic understanding.

[0048] It should be noted that the pre-trained layout detection model can adopt a YOLOv8 / LayoutLMv3 architecture. In practical applications, when training the layout detection model, publicly available datasets can be used directly, such as the PubLayNet dataset. The PubLayNet dataset contains millions of document layout annotations, covering typical functional areas such as text blocks, tables, graphics, and titles. After pre-training, the layout detection model has high-precision region classification and bounding box regression capabilities, and can effectively identify small regions, overlapping regions, and the outline boundaries of complex tables in document images. The layout detection model performs pixel-level feature extraction and region segmentation on standardized document images. According to a preset category system, it classifies and identifies independent visual units within the document image into four core functional areas: text blocks, tables, graphics, and titles. Among them, the title area refers to the hierarchical title of the document; the table area refers to the two-dimensional data area containing row and column structures and cells, which is the core area that is the focus of processing in this embodiment; the text block area is a continuous paragraph text, annotation text, explanatory text, and other plain text areas; and the graphics area is a visual graphic area such as line charts, bar charts, pie charts, and flowcharts in the document.

[0049] Next, obtain the bounding box coordinates and category information for each functional area. Then, by normalizing the bounding box coordinates and grouping them according to spatial relationships, construct a document structure tree that represents the spatial and hierarchical relationships between functional areas.

[0050] It should be noted that the bounding box coordinates use pixel coordinates to represent the bounding box coordinates of the region's circumscribed rectangle, used to accurately characterize the physical location of each functional region within the document image. Normalization is performed on the bounding box coordinates of all functional regions to eliminate coordinate deviations caused by differences in resolution, size, and layout among different document images, thus standardizing the functional region location information. Specifically, global size parameters of the document image can be extracted, and then the bounding box coordinates of individual regions can be normalized and calculated, followed by constraints on the normalization results. Based on the normalized bounding box coordinates, combined with the natural reading logic and visual spatial rules of document layout, all functional regions are spatially grouped, ultimately constructing a document structure tree representing the spatial positional relationships and hierarchical relationships between regions.

[0051] S202, Regional-level multimodal semantic understanding.

[0052] For each functional area identified by the layout detection model, especially the table area, a visual language model can be used for fine-grained semantic parsing.

[0053] The Vision-Language Model (VLM) receives image patches of functional regions and their category information, and outputs a structured data representation that describes the internal structure of the region. This structured data representation can capture complex layouts such as merged cells and stacked headers.

[0054] In practical applications, based on the original pixel bounding box coordinates of each functional area obtained in step S201, independent image blocks for each area are accurately extracted from the standardized document image. Simultaneously, to adapt to the input requirements of the visual language model, the extracted image blocks undergo targeted preprocessing, scaling them to a standard size supported by the model. For example, contrast is enhanced in table areas to highlight cell borders and the dividing lines of merged cells, and blank areas at the edges are cropped to improve processing efficiency. Each preprocessed region image block is accompanied by its corresponding category label, such as "Table," "Heading 1," or "Body Text Block."

[0055] Subsequently, differentiated prompts were designed for different types of areas to clarify the model's parsing tasks. For example, for table areas, the prompts would require the model to parse the number of rows and columns, the header level, the position and range of merged cells, the content of each cell, and the corresponding header relationship; for title areas, the model would be required to identify the title level, whether it is a cascading title, and the hierarchical relationship between the main and subtitles.

[0056] After receiving input, the visual language model works collaboratively with its built-in visual feature branch to complete cross-modal semantic parsing. The visual branch extracts visual features from the region image, such as table borders, cell distribution, connected components of merged cells, and font size differences in headings. The language branch, based on prompts, maps these visual features to corresponding semantic concepts; for example, it identifies "horizontally connected cell regions" as "merged cells" and "text with larger font sizes located above the table" as "first-level headers." For complex layouts such as merged cells and cascading headings, which are difficult to handle with traditional methods, the model employs a dedicated parsing strategy: for merged cells, it identifies their starting and ending row and column coordinates, while simultaneously associating the semantic relationship between the text within the cell and the corresponding heading; for cascading headings, it constructs a tree-like hierarchical structure of main and subheadings based on font size, weight, and positional relationships.

[0057] Ultimately, the visual language model outputs structured data in JSON format. This data includes both category information for the regions and a complete record of the structural features and semantic content within each region. Taking a table region as an example, the output structured data clearly specifies the total number of rows, columns, header levels, details of merged cells, as well as the row and column position, content, and header level of each cell. For other regions such as titles, text blocks, and graphics, corresponding structured data is also output to ensure that subsequent steps can directly read and process it.

[0058] This application achieves accurate conversion from visual layout to structured semantics, completely solving the problem of traditional OCR technology losing complex structural information, and providing solid data support for the subsequent generation of natural language descriptions that are faithful to the original text.

[0059] S203, Generate a structured text description.

[0060] The structured data representations of all regions obtained in step S202 are aggregated and input into a large language model. The large language model converts the structured JSON data into a fluent and coherent natural language description based on predefined rules or prompts. This description retains the accuracy of the original data while conforming more closely to the grammatical and semantic conventions of natural language.

[0061] After receiving input, the large language model first parses the core information in the structured data. For example, for table areas, it extracts the header hierarchy, the content range corresponding to merged cells, and the numerical correspondence between rows and columns. For title areas, it clarifies the title hierarchy and the content modules to which it belongs. For text blocks and graphic areas, it extracts the core viewpoints and key data. Then, the large language model integrates this fragmented structured information into a coherent natural language description according to predefined rules and prompt word requirements.

[0062] S204, Construct a retrieval index library.

[0063] The natural language descriptions generated in step S203 are encoded into dense vectors in a high-dimensional vector space using a dedicated text embedding model. Then, the dense vectors generated from all documents are used to construct a fast-searchable vector index database.

[0064] In practical applications, the Qwen3-Embedding-8B model can be used for text embedding. This model has the ability to encode deep semantics for long texts and structured descriptions, and can accurately capture the hidden logical connections in the text. The text embedding model can convert each natural language description into a 768-dimensional high-dimensional dense vector. This vector can represent the core semantic features of the text in a high-dimensional space. Vectors corresponding to semantically similar descriptions are closer in space, and vice versa.

[0065] When building a fast-searchable vector index database, the FAISS vector database can be used. This database supports efficient storage and similarity retrieval of large-scale vectors. During index construction, vector data is associated with corresponding region source tags and original document fragment information, forming a ternary mapping relationship between high-dimensional dense vectors, semantic description text, and original document regions.

[0066] In practical applications, to ensure retrieval efficiency, vector indexes can be optimized. For example, hierarchical clustering indexes or product quantization compression algorithms can be used to reduce the space overhead of vector storage without significantly losing semantic accuracy, thereby achieving millisecond-level retrieval response for hundreds of millions of vectors.

[0067] This application leverages the semantic encoding capabilities of text embedding models to transform natural language descriptions into machine-computable vector features, thus overcoming the semantic limitations of traditional keyword retrieval.

[0068] S205, Retrieval and Enhanced Generation.

[0069] The system receives natural language queries from users and converts them into query vectors using the same embedding model. It then performs a similarity search in a vector index database to find the Top-K (pre-defined) most relevant natural language descriptions and their corresponding original document regions. Finally, the retrieved natural language descriptions, used as context, are input along with the user query into a generative large language model to generate the final natural language answer.

[0070] It should be noted that each search result contains three key layers of information: a natural language description text that highly matches the query, a high-dimensional dense vector corresponding to that text, and information about the original document from which the text originated. This source information allows for precise location of the original text corresponding to the answer, providing a basis for the reliability of subsequently generated content.

[0071] Generative large language models first parse the core needs of a user's query, then extract relevant key data and logical connections from the context, ultimately generating fluent and accurate natural language answers. This ensures that the generated answers are faithful to the original text and directly respond to the user's query needs, solving the problem that traditional retrieval systems can only return document fragments and cannot directly output conclusions.

[0072] The following is an example from this application: A user wants to query "What percentage of the company's revenue was invested in R&D over the past two years?". First, layout detection and structure parsing are performed. The document image is input into a layout detection model pre-trained on the PubLayNet dataset. The model identifies a heading, three paragraph text blocks, and a large table in the document. The layout detection module outputs the bounding box for each region, with the bounding box of the table region being precisely extracted.

[0073] Next, regional multimodal semantic understanding is performed. The captured table area image is fed into Qwen2.5-VL-32B-Instruct. The VLM prompts are designed to require it to parse the table structure. After analyzing the image, VLM outputs a structured JSON object. This JSON clearly defines that the table has 5 columns, with column headers for "Project", "2023", "2022", "2021", and "Remarks". It correctly identifies "Operating Revenue" and "R&D Investment" as row headers and accurately extracts the corresponding year values. It also notices that a merged cell in the "Operating Revenue" row spans the "Remarks" column. This step is crucial, accurately capturing the logical structure that OCR cannot provide.

[0074] Next, a structured text description is generated. The JSON representation above, along with the OCR results of the text regions, is input into the Qwen3-14B model. The task of the LLM is to transform this structured data into a coherent summary. It might generate a description like this: "This table shows the company's key financial data from 2021 to 2023. Among them, operating revenue has continued to grow over the three years, at X million yuan, Y million yuan, and Z million yuan respectively; R&D investment has also steadily increased, at A million yuan, B million yuan, and C million yuan respectively. It is worth noting that the growth rate of R&D investment in 2023 is relatively high." Subsequently, a retrieval index is built. This generated natural language description is converted into a 768-dimensional dense vector by the Qwen3-Embedding-8B model and stored in the FAISS vector database along with its source document information. The database contains many similar vectors extracted and transformed from different documents.

[0075] Finally, retrieval and enhanced generation are performed. When a user queries "R&D investment ratio," the query itself is embedded as a vector. The system performs a similarity search in the FAISS database. The generated description vector, containing both the key entities "operating revenue" and "R&D investment" and their values, has the highest similarity to the query vector and is therefore successfully retrieved as context. Ultimately, the LLM combines this context with the user query to perform simple mathematical calculations and generate an accurate answer: "According to the report, the R&D investment ratio of operating revenue in 2023 was C / Z, approximately X%; in 2022 it was B / Y, approximately Y%."

[0076] Figure 3 illustrates a schematic diagram of the document parsing and enhancement generation system of this application. It may include: a data module for converting user-inputted natural language for querying into dense vectors; a query module for querying a vector index database using the dense vectors to obtain the most relevant natural language descriptions and corresponding original document regions; and a generation module for combining the natural language descriptions obtained from the query with the user-inputted natural language to obtain the final natural language answer. The method for constructing the vector index database includes: parsing the document image using a layout detection model to identify different functional regions of the document image, and then constructing a document structure tree representing the spatial and hierarchical relationships between the functional regions; performing fine-grained semantic parsing on each functional region using a visual language model; inputting the semantic parsing into a large language model to obtain the corresponding natural language description; encoding the natural language descriptions into dense vectors, and then constructing all the dense vectors into a vector index database.

[0077] It should be noted that, in the several embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the system embodiments described above are merely illustrative; for instance, the division of each block is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple blocks may be combined or integrated into another device, or some features may be ignored or not executed. The modules described as separate components may or may not be physically separated. The components shown as modules may be one or more physical units, that is, they may be located in one place or distributed in multiple different places. Some or all of the modules can be selected to achieve the purpose of the solution in this embodiment according to actual needs.

[0078] Furthermore, in the various embodiments of the present invention, the modules can be integrated into one processing unit, or each module can exist physically separately, or two or more modules can be integrated into one unit. The integrated unit described above can be implemented in hardware or as a software functional unit.

[0079] This application also provides an electronic device, which may include one or more processors, memory and communication interfaces.

[0080] The memory, communication interface, and processor are coupled together. For example, the memory, communication interface, and processor can be coupled together via a bus.

[0081] The communication interface is used for data transmission with other devices. The memory stores computer program code. This computer program code includes computer instructions, which, when executed by the processor, cause the electronic device to perform the steps of the document parsing and enhancement generation method described above.

[0082] The processor can be a processor or controller, such as a Central Processing Unit (CPU), a general-purpose processor, a Digital Signal Processor (DSP), an Application-Specific Integrated Circuit (ASIC), a Field-Programmable Gate Array (FPGA), or other programmable logic devices, transistor logic devices, hardware components, or any combination thereof. It can implement or execute the various exemplary logic blocks, modules, and circuits described in conjunction with this disclosure. The processor can also be a combination that implements computing functions, such as a combination of one or more microprocessors, a combination of a DSP and a microprocessor, etc. The processor can be used to support an electronic device in performing the method steps provided in the above embodiments.

[0083] The bus can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. These buses can be categorized as address buses, data buses, control buses, etc.

[0084] This application provides a computer-readable storage medium storing a computer program. When the computer program is executed by a processor, it implements the steps of the above-described document parsing and enhancement generation method.

[0085] The computer-readable storage media involved in this application include random access memory (RAM), memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, hard disks, removable disks, CD-ROMs, or any other form of storage media known in the art.

[0086] The above are merely preferred embodiments of this application and are not intended to limit this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the protection scope of this application.

Claims

1. A document parsing and enhancement generation method, characterized in that, include: The user-input natural language for querying is converted into a dense vector; the dense vector is then used to query a vector index database to obtain the most relevant natural language descriptions and their corresponding original document regions. By combining the natural language description obtained from the query with the natural language input by the user for the query, the final natural language answer is obtained; The method for constructing the vector index database includes: parsing a document image using a layout detection model to identify different functional regions of the document image, and then constructing a document structure tree that represents the spatial and hierarchical relationships between the functional regions; performing fine-grained semantic parsing on each functional region using a visual language model; inputting the semantic parsing into a large language model to obtain the corresponding natural language description; encoding the natural language description into dense vectors, and then constructing all the dense vectors into a vector index database.

2. The document parsing and enhancement generation method according to claim 1, characterized in that, The different functional areas of the document image include text blocks, tables, graphics, and titles.

3. The document parsing and enhancement generation method according to claim 1, characterized in that, The method for constructing a document structure tree representing the spatial and hierarchical relationships between functional regions includes: obtaining the bounding box coordinates and category information of each functional region; normalizing the bounding box coordinates and grouping them according to spatial relationships; and constructing a document structure tree representing the spatial and hierarchical relationships between functional regions based on the normalization and spatial relationship grouping results.

4. The document parsing and enhancement generation method according to claim 3, characterized in that, The method for fine-grained semantic parsing of each functional region using a visual language model includes: extracting independent image blocks of each functional region from the document image based on the bounding box coordinates of each functional region; designing differentiated prompt words for different categories of functional regions to determine the parsing task of the visual language model; and outputting structured semantic parsing based on the independent image blocks of each functional region and the corresponding differentiated prompt words.

5. The document parsing and enhancement generation method according to claim 1, characterized in that, The method for encoding natural language descriptions into dense vectors includes: using a text embedding model to encode natural language descriptions into dense vectors.

6. The document parsing and enhancement generation method according to claim 1, characterized in that, The method for obtaining a final natural language answer by combining the natural language description obtained from the query with the natural language input by the user for the query includes: using the natural language description obtained from the query as context, and inputting it together with the natural language input by the user for the query into a generative large language model to obtain the final natural language answer.

7. A document parsing and enhancement generation system, characterized in that, include: The data module is used to convert the natural language input by the user for querying into dense vectors; the query module is used to query the vector index database using the dense vectors to obtain the most relevant natural language descriptions and corresponding original document regions. The generation module is used to combine the natural language description obtained from the query with the natural language input by the user to obtain the final natural language answer; The method for constructing the vector index database includes: parsing a document image using a layout detection model to identify different functional regions of the document image, and then constructing a document structure tree that represents the spatial and hierarchical relationships between the functional regions; performing fine-grained semantic parsing on each functional region using a visual language model; inputting the semantic parsing into a large language model to obtain the corresponding natural language description; encoding the natural language description into dense vectors, and then constructing all the dense vectors into a vector index database.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the steps of the document parsing and enhancement generation method as described in any one of claims 1-6.

9. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the document parsing and enhancement generation method as described in any one of claims 1-6.

10. A computer program product, characterized in that, The computer program product includes computer instructions, the processor of the computer device reads the computer instructions, and the processor of the computer device executes the computer instructions to implement the steps of the document parsing and enhancement generation method as described in any one of claims 1 to 6.

Images