A RAG-oriented document parsing method, system and computer device

Abstract

This invention relates to the field of artificial intelligence, providing a document parsing method, system, and computer device for RAG (Reference Object Type). The RAG-oriented document parsing method includes: uniformly normalizing acquired documents of different formats to obtain Markdown format for all documents; extracting plain text content from the Markdown format documents, segmenting the plain text content according to Markdown semantic structure to obtain several text fragments, and vectorizing all text fragments; extracting non-text content from the Markdown format documents, associating the extracted non-text content with the text fragment vectors, and storing them in a relational database; and retrieving text fragment vectors and associated non-text content from the relational database based on a user-input query to generate context fragments. This improves retrieval accuracy and the completeness of the generated answer.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology, and in particular to a document parsing method, system, and computer device for RAG. Background Technology

[0002] The statements in this section are merely background information related to the present invention and do not necessarily constitute prior art.

[0003] In recent years, Large Language Models (LLMs) have been widely used due to their powerful generative capabilities. However, the application of LLMs in specific knowledge domains often relies on Retrieval-Augmented Generation (RAG) techniques. RAGs enhance the output of LLMs by retrieving relevant information from external databases, effectively addressing issues such as insufficient timeliness of model knowledge and the illusion of knowledge. One of the core challenges of this technique is how to efficiently and accurately transform massive amounts of unstructured documents into retrieveable vector representations.

[0004] The document processing workflow of existing RAG systems usually has the following defects: (1) Different document types require independent parsing libraries, such as using PyMuPDF to parse PDF, python-docx to parse DOCX, and python-pptx to parse PPTX. This fragmented processing method leads to a complex workflow and makes it difficult to guarantee the consistency of parsing results for documents of different formats. (2) Existing technologies mostly use fixed-size segmentation (e.g., segmentation every 512 characters) or segmentation based on punctuation marks. This method ignores the original semantic structure of the document (such as headings, paragraphs, lists, etc.) and may segment a complete semantic unit into different blocks, resulting in incomplete semantics. (3) Most methods directly discard images, code blocks, tables, etc. when converting document content into plain text, arguing that this information cannot be used for vectorization. However, this content is often the key to understanding the document context, and its absence will seriously affect the quality of the answers generated by the RAG system.

[0005] LangChain offers a variety of document loaders and text splitters to handle various document types. However, its processing flow is typically linear: first load the document as plain text, then split it. This approach fails to effectively preserve and utilize the original document structure and non-textual information during processing, resulting in the split text chunks losing their context.

[0006] Some commercial RAG platforms have adopted more intelligent segmentation strategies, such as segmentation based on headings or paragraphs. However, they still generally suffer from the following shortcomings: (1) Documents of different formats still need to be processed separately, lacking a unified intermediate representation layer. (2) Even if images and code can be recognized, they are often simply replaced with placeholders instead of being stored independently and indexed and associated with relevant text. This results in the system being unable to directly return images or code when searching, affecting the completeness of the answer. Summary of the Invention

[0007] To address the technical problems mentioned above, this invention provides a document parsing method, system, and computer device for RAG (Reference Artificial Intelligence). First, this invention normalizes all documents into Markdown (MD) format, simplifying subsequent processing. Second, it employs an MD semantic structured segmentation method, preserving the complete document structure while storing and associating different types of content (text, images, code, etc.) separately. This allows for dynamic reorganization during retrieval, ensuring no information loss or redundancy, significantly improving the retrieval accuracy and completeness of generated answers in the RAG system.

[0008] To achieve the above objectives, the present invention adopts the following technical solution:

[0009] The first aspect of this invention provides a document parsing method for RAG.

[0010] A document parsing method for RAG includes:

[0011] The documents of different formats are uniformly normalized to obtain the Markdown format of all documents;

[0012] Based on Markdown format documents, the plain text content is extracted from the Markdown format documents. The plain text content is then segmented according to the Markdown semantic structure to obtain several text fragments. All text fragments are vectorized. The non-text content is extracted from the Markdown format documents. The extracted non-text content is associated with the text fragment vectors and stored in a relational database.

[0013] When a user performs a search, based on the user's query request, the system retrieves text fragment vectors and associated non-text content from the relational database, merges and reassembles them to generate a context fragment.

[0014] Furthermore, the method involves uniformly normalizing the acquired documents of different formats to obtain the Markdown format for all documents; the method includes:

[0015] Based on DOCX / DOC documents, a library is used to parse and recognize paragraphs, headings, lists, and tables, and convert them into Markdown format.

[0016] Based on PDF documents, layout analysis is performed to identify text blocks, images, and tables. The structure of the PDF document is inferred based on the style information of text blocks, images, and tables on the page, and then converted into Markdown format.

[0017] Based on PPTX / PPT documents, extract text and non-text content, and convert it into Markdown format according to the order and hierarchy of text and non-text content in the slides;

[0018] Based on XLSX / XLS documents, convert them to Markdown format table syntax;

[0019] Extract the plain text content from a TXT document and convert it to Markdown format.

[0020] Furthermore, after obtaining the text fragment vectors, they are stored in a vector database; after extracting the non-text content from the Markdown format document, the extracted non-text content is stored in a relational database.

[0021] Furthermore, the non-text content includes: images, code, and tables.

[0022] Furthermore, the method for segmenting plain text content according to Markdown semantic structure includes: segmenting plain text content by headings and paragraphs.

[0023] Furthermore, after segmenting the plain text content according to the Markdown semantic structure, a total loss function is used to optimize the semantic coherence within text segments and the semantic distinguishability between different text segments.

[0024] Furthermore, the total loss function is expressed by the following formula:

[0025]

[0026]

[0027]

[0028] in, Represents the total loss function. This indicates a loss of content consistency. This indicates the loss of coherence in text segments. , Represents the coefficient. This indicates the number of sentences in the text segment. Sentence With sentences Semantic similarity between them This represents the weight of the i-th sentence. This indicates the total number of sentences included in the calculation. Indicates the scope of a comparison across text segments. This indicates the sentence at the end of the current text segment. Sentences located at the beginning of other text segments Perform semantic similarity. , Each represents a different text fragment. This indicates the number of sentences in a text segment.

[0029] Furthermore, when a user performs a search, based on the user's input query request, text fragment vectors and associated non-text content in the relational database are retrieved, merged, and recombined to generate a context fragment. The method includes: vectorizing the user's input query request and performing a similarity search in the vector database to obtain the most relevant text fragment vectors; for each retrieved text fragment vector, checking its association information in the relational database; if the text fragment vector is associated with non-text content, retrieving the associated non-text content from the relational database; merging and recombining the content corresponding to the retrieved text fragment vector with the associated non-text content to generate a context fragment; and generating and displaying the answer using a large language model.

[0030] A second aspect of the present invention provides a document parsing system for RAG.

[0031] A document parsing system for RAG (Reference Art Documents), comprising:

[0032] The normalization module is configured to: uniformly normalize the documents of different formats obtained, and obtain the Markdown format of all documents;

[0033] The segmentation module is configured to: extract plain text content from Markdown format documents, segment the plain text content according to the Markdown semantic structure to obtain several text fragments, vectorize all text fragments; extract non-text content from Markdown format documents, associate the extracted non-text content with the text fragment vectors, and store them in a relational database.

[0034] The retrieval generation module is configured to: when a user performs a retrieval, retrieve text fragment vectors and non-text content associated with the text fragment vectors from the relational database according to the user's query request, merge and reorganize them to generate context fragments.

[0035] A third aspect of the present invention provides a computer device comprising:

[0036] A processor, adapted to execute computer programs;

[0037] A computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps of the RAG-oriented document parsing method described in the first aspect above.

[0038] Compared with the prior art, the beneficial effects of the present invention are:

[0039] To address the shortcomings of existing RAG systems, which require independent parsing libraries for different document types and suffer from fragmented processing leading to complex workflows and poor consistency, this invention unifies all documents into Markdown format, achieving defrauding of document processing. On one hand, it eliminates the need for separate parsing libraries for different document formats such as PDF, DOCX, and PPTX; document preprocessing can be completed simply through a unified Markdown conversion logic, significantly simplifying the document processing workflow of the RAG system and reducing development and maintenance costs. On the other hand, the unified Markdown format provides standardized input for subsequent processing, avoiding content discrepancies caused by differences in parsing rules between different parsing libraries (such as inconsistent font styles and paragraph boundary recognition), significantly improving the consistency of parsing results for documents of different formats, and laying the foundation for subsequent semantic segmentation and retrieval quality.

[0040] To address the issues of incomplete semantics caused by fixed-size or punctuation-based segmentation in existing technologies that neglect the original semantic structure of the document, and the limitations of LangChain's linear processing in preserving the original structure, this invention employs the MD semantic structured segmentation method, achieving precise preservation of the original semantic structure of the document. Markdown format itself possesses a natural ability to identify semantic structures such as headings (# to ######), paragraphs, lists (ordered / unordered lists), and quotation blocks. Based on this, structured segmentation can directly use semantic units as the segmentation unit (e.g., a complete heading + corresponding paragraph, or an independent list item group), rather than mechanically dividing by character count or punctuation marks. This segmentation method fundamentally avoids the problem of splitting complete semantic units into different chunks, ensuring that each text block carries complete semantic information while preserving the document's hierarchical structure (e.g., the hierarchical relationship between headings and subheadings, and the association between paragraphs and lists). During retrieval, the system can match user needs based on complete semantic units, significantly reducing retrieval bias caused by semantic fragmentation and improving the accuracy and logic of the answers generated by the RAG system. Attached Figure Description

[0041] The accompanying drawings, which form part of this invention, are used to provide a further understanding of the invention. The illustrative embodiments of the invention and their descriptions are used to explain the invention and do not constitute an improper limitation of the invention.

[0042] Figure 1 This is a flowchart illustrating a document parsing method for RAG as shown in an embodiment of the present invention;

[0043] Figure 2 This is a structural diagram of a RAG-oriented document parsing system shown in an embodiment of the present invention;

[0044] Figure 3 This is a structural diagram of a computer device shown in an embodiment of the present invention. Detailed Implementation

[0045] The present invention will be further described below with reference to the accompanying drawings and embodiments.

[0046] It should be noted that the following detailed description is illustrative and intended to provide further explanation of the invention. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains.

[0047] It should be noted that the terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the scope of exemplary embodiments according to the invention. As used herein, the singular form is intended to include the plural form as well, unless the context clearly indicates otherwise. Furthermore, it should be understood that when the terms "comprising" and / or "including" are used in this specification, they indicate the presence of features, steps, operations, devices, components, and / or combinations thereof.

[0048] To facilitate understanding of the technical solutions of this invention, some technical terms involved in this invention will be introduced below.

[0049] Retrieval-Enhanced Generation (RAG) is a model that combines retrieval and generation techniques. It generates answers or content by referencing information from external knowledge bases, offering strong interpretability and customizability. It is suitable for various natural language processing tasks, including question-answering systems, document generation, and intelligent assistants. The advantages of RAG models lie in their versatility, ability to achieve real-time knowledge updates, and the provision of more efficient and accurate information services through end-to-end evaluation methods.

[0050] As introduced in the background section, existing RAG systems face challenges when processing documents with multiple formats and complex structures, mainly manifested in the following ways: (1) Inconsistent formats: Different document formats (such as PDF, DOCX, PPTX, etc.) require different parsers, increasing system complexity and maintenance costs, and making it difficult to achieve unified subsequent processing. (2) Structural redundancy: Non-textual information contained in documents (such as images, code blocks, tables, etc.) is difficult to process effectively in the traditional text chunking process, resulting in the generated embedding vector containing a large amount of irrelevant information, affecting retrieval accuracy. (3) Information loss: Traditional text segmentation methods usually simply remove non-textual content such as images and code, resulting in these key information not being effectively utilized in the retrieval and generation stages, affecting the completeness and accuracy of the final answer.

[0051] To address at least one of the aforementioned problems, this invention provides a document parsing method, system, and computer device for RAG (Related Aspects of Documents). Through document normalization and structured segmentation, it significantly improves the retrieval efficiency and generation quality of the RAG system. The invention is described in detail below through several embodiments.

[0052] Figure 1 This is a flowchart illustrating a RAG-oriented document parsing method according to an embodiment of the present invention; see reference. Figure 1 The method includes,

[0053] The documents of different formats are uniformly normalized to obtain the Markdown format of all documents;

[0054] Based on Markdown format documents, the plain text content is extracted from the Markdown format documents. The plain text content is then segmented according to the Markdown semantic structure to obtain several text fragments. All text fragments are vectorized. The non-text content is extracted from the Markdown format documents. The extracted non-text content is associated with the text fragment vectors and stored in a relational database.

[0055] When a user performs a search, based on the user's query request, the system retrieves text fragment vectors and associated non-text content from the relational database, merges and reassembles them to generate a context fragment.

[0056] This invention first normalizes all documents into Markdown (MD) format to simplify subsequent processing; secondly, it adopts the MD semantic structured segmentation method, which stores and associates different types of content (text, images, code, etc.) separately while preserving the complete structure of the documents. This allows for dynamic reorganization during retrieval, ensuring that information is not lost or redundant, and significantly improving the retrieval accuracy and completeness of the generated answers in the RAG system.

[0057] The document parsing method for RAG described in this embodiment is described in detail below. The method can be divided into two stages: document normalization and MD semantic structure segmentation.

[0058] The document normalization stage aims to convert documents of various formats (such as doc / docx, pdf, pptx / ppt, xls / xlsx, txt, etc.) into Markdown (MD) format. MD format is characterized by its simplicity, structure, and good readability, making it an ideal intermediate representation. This embodiment uses document normalization to unify the complexity of all documents into MD format, providing a unified and standardized input for subsequent structured processing.

[0059] More specifically, different parsers are used for documents of different formats, while adhering to a unified MD conversion rule:

[0060] (1) For DOCX / DOC documents, use libraries such as python-docx to parse them, identify elements such as paragraphs, headings, lists, and tables, and convert them into the corresponding MD syntax. For example, convert "Heading 1" to # Heading 1, and "Unordered List" to - item.

[0061] (2) For PDF documents, layout analysis is performed using libraries such as PyMuPDF. Text blocks, images, tables, etc., are identified, and their structure is inferred based on their relative positions on the page (e.g., X / Y coordinates) and style information such as font size and bolding, and then converted to Material Design (MD). For example, bold text is often considered a title. Image content can be extracted and saved as a file, referenced in the MD in the form of .

[0062] (3) For PPTX / PPT documents, use libraries such as python-pptx to extract the content of the slides (text boxes, pictures, charts, etc.) and convert them into MD according to their order and hierarchy in the slides.

[0063] (4) For XLSX / XLS documents, use libraries such as openpyxl to convert the table content into MD table syntax (|column 1|column 2|).

[0064] (5) For TXT documents, directly convert the plain text content to MD, or format it according to simple rules such as line breaks and indentation.

[0065] To address the shortcomings of commercial RAG platforms, which require separate processing of different document formats and lack a unified intermediate representation layer, this invention employs a unified Markdown normalization design to construct a unified intermediate representation layer for cross-format documents. Whether it's tables in PDFs, images in DOCX documents, or lists and text combinations in PPTX documents, all are ultimately converted into a standardized Markdown structure, and the logic for associating non-textual and textual information remains consistent across different document formats. This allows the RAG system to retrieve cross-format documents based on a unified indexing system. For example, when a user queries "the technical architecture of a project," the system can simultaneously match relevant content from images in PDF technical documents, text descriptions in DOCX requirement documents, and list summaries in PPTX reports, and reassemble them into a complete answer through structured association, without needing to switch between retrieval modules for different document formats. This unified cross-format retrieval capability not only improves retrieval efficiency but also avoids the omission of relevant information due to format isolation, further enhancing the comprehensive retrieval capabilities of the RAG system.

[0066] In some embodiments, the MD semantic structured segmentation stage aims to leverage the structured characteristics of the MD format to more accurately segment and store documents.

[0067] More specifically, the normalized MD document is viewed as a sequence containing different modules, each module corresponding to a semantic unit in the MD, such as: headings (#, ##, ###, etc.), paragraphs, code blocks (python...), images (![...]), tables (|...|), lists (-, *, etc.).

[0068] Simultaneously, this embodiment creates a multi-schema database or utilizes multiple tables from an existing database to process the content of each module into different sections as needed and stores this information. For Section 1 (text), plain text content is extracted from the MD document, removing non-text content such as images and code. This text content will be used for subsequent vectorization and segmentation. For Section 2 (non-text), images, code, tables, and other content in the MD are stored separately and associated with their position and context in the original document (e.g., the paragraph or heading to which they belong).

[0069] To address the shortcomings of existing technologies that discard non-textual information such as images, code blocks, and tables, or simply replace them with placeholders, this embodiment fully activates the value of non-textual information by storing and associating different types of content separately. When a document is normalized to Markdown format, images, code blocks, tables, and other content can be identified using Markdown's proprietary syntax (e.g., code block | table | format |) and indexed and associated with related text (e.g., images are bound to adjacent explanatory text, and code blocks are bound to their corresponding explanatory paragraphs with index IDs). Simultaneously, the system establishes separate storage modules for non-textual information (e.g., image repository, code snippet repository), rather than directly discarding or only retaining placeholders. During the retrieval phase, if the user's request is related to non-textual information (e.g., "Please explain the meaning of a certain architecture diagram in the document" or "Execute the code example in the document"), the system can accurately locate the corresponding non-textual content through index association and return it along with the relevant text blocks, rather than missing answers due to the inability to identify the relevance of non-textual information or only returning placeholders. This design not only fills the gap in the utilization of non-textual information in existing RAG systems, but also makes non-textual information a key support for understanding the context of documents, significantly improving the completeness and richness of the answers generated by the RAG system. It is especially suitable for scenarios containing a large amount of non-textual information, such as technical documents and scientific research reports.

[0070] In this embodiment, Section 1 (plain text) is segmented. The segmentation strategy can be based on a Material Design (MD) structure, for example, segmenting by headings or paragraphs, rather than a fixed number of characters, to ensure semantic integrity. The segmented text fragments are vectorized and stored in a vector database. The content of Section 2 (non-text) and its association information with the text fragments are stored in a relational database (such as PostgreSQL) or a document-oriented database (such as MongoDB). This database records the images, code, and other information associated with each text fragment (chunk).

[0071] In this embodiment, to ensure semantic coherence within the segmented text fragments and to prevent significant semantic jumps between different text fragments, a total loss function is designed for optimization. The total loss function is as follows: Expressed using the following formula:

[0072]

[0073] in, This indicates a loss of content consistency. This indicates the loss of coherence in text segments. , The coefficient can be set based on experimental data.

[0074] More specifically, loss of content consistency This is to ensure that the sentences within each text segment are very similar, resulting in content consistency loss. Expressed using the following formula:

[0075]

[0076] in, This indicates the number of sentences in the text segment. Sentence With sentences Semantic similarity between them This represents the weight of the i-th sentence.

[0077] More specifically, loss of text fragment coherence This is to ensure that the semantic changes between different text segments are obvious and that the coherence of the text segments is not lost. Expressed using the following formula:

[0078]

[0079] in, This indicates the total number of sentences included in the calculation. Indicates the scope of a comparison across text segments. This indicates the sentence at the end of the current text segment. Sentences located at the beginning of other text segments Perform semantic similarity. , Each represents a different text fragment. This indicates the number of sentences in a text segment.

[0080] This embodiment improves the quality and accuracy of text segmentation by designing a total loss function.

[0081] In some embodiments, if a user performs a query, it can be implemented according to the following process:

[0082] Step 1: The user enters a query request.

[0083] In this process, the user's query request is transformed into a standardized requirement that the system can process. First, it is compatible with various input formats such as short sentences, long texts, and colloquial language, and automatically cleans up redundant symbols and corrects spelling errors. Then, it uses an NLP model to identify the user's intent, such as fact queries, method tutorials, etc. For fuzzy queries (such as "learn programming"), it completes them into "languages ​​and paths for learning programming from scratch" to ensure that the requirements are clear.

[0084] Step 2: Vectorize the user query and perform a similarity search in the vector database to obtain the most relevant text fragments.

[0085] In this process, in order to overcome the limitations of traditional keyword matching, the query requests and knowledge base text input by users are first transformed into 768-dimensional or 1024-dimensional vectors using models such as Sentence-BERT. The vectors are then stored in dedicated databases such as Milvus, and the retrieval speed is optimized with the help of indexes such as HNSW. Next, the cosine similarity between the query vector and the text vector is calculated. The closer the value is to 1, the more similar they are. Results below the threshold of 0.7 are filtered out, and the top 5-10 most relevant text fragments are selected after sorting by similarity.

[0086] Step 3: For each retrieved text fragment, the system first checks its association information in the relational database.

[0087] In this process, the vector library only returns text fragments, which need to be linked to a relational database using unique text IDs. The "text-non-text" mapping table in the relational database is queried to confirm the existence of images, code, tables, etc.; simultaneously, the validity of the linked content is verified, such as checking if the image URL is available and if the code syntax is correct, to prevent invalid information from entering subsequent stages.

[0088] Step 4: If the text fragment is associated with images, code, tables, or other content, the system will retrieve this content from the relational database.

[0089] In this process, first merge duplicate content in text fragments, sort them according to user needs (such as "installation-use-problem solving"), and add transitional statements; then embed relevant content into corresponding text positions, such as embedding code in Python format after tutorial steps, and displaying comparison text in tables; finally, optimize the format with titles and lists, highlight key information, and ensure that the output is concise, coherent, and directly meets user needs.

[0090] Step 5: First, integrate the retrieved text fragments, remove duplicates, and sort them logically, using transitional sentences to connect the context. Then, accurately embed related non-text content such as images, code, and tables into the corresponding text locations, such as embedding code after tutorial steps or matching comparison content in tables. Finally, optimize the format, highlight key information, and form a complete and context-rich fragment. This fragment serves as an enhanced information input LLM, providing it with comprehensive materials to help generate more accurate and complete answers.

[0091] The above combination Figure 1 The document parsing method for RAG provided in the embodiments of the present invention has been described in detail. Next, the document parsing system for RAG provided in the embodiments of the present invention will be described in conjunction with the accompanying drawings.

[0092] Figure 2 This is a schematic diagram of the structure of a RAG-oriented document parsing system according to an embodiment of the present invention, with reference to... Figure 2 The system described in this invention includes:

[0093] The normalization module is configured to: uniformly normalize the documents of different formats obtained, and obtain the Markdown format of all documents;

[0094] The segmentation module is configured to: extract plain text content from Markdown format documents, segment the plain text content according to the Markdown semantic structure to obtain several text fragments, vectorize all text fragments; extract non-text content from Markdown format documents, associate the extracted non-text content with the text fragment vectors, and store them in a relational database.

[0095] The retrieval generation module is configured to: when a user performs a retrieval, retrieve text fragment vectors and non-text content associated with the text fragment vectors from the relational database according to the user's query request, merge and reorganize them to generate context fragments.

[0096] In some embodiments, the method of uniformly normalizing the acquired documents of different formats to obtain Markdown format for all documents includes:

[0097] Based on DOCX / DOC documents, a library is used to parse and recognize paragraphs, headings, lists, and tables, and convert them into Markdown format.

[0098] Based on PDF documents, layout analysis is performed to identify text blocks, images, and tables. The structure of the PDF document is inferred based on the style information of text blocks, images, and tables on the page, and then converted into Markdown format.

[0099] Based on PPTX / PPT documents, extract text and non-text content, and convert it into Markdown format according to the order and hierarchy of text and non-text content in the slides;

[0100] Based on XLSX / XLS documents, convert them to Markdown format table syntax;

[0101] Extract the plain text content from a TXT document and convert it to Markdown format.

[0102] In some embodiments, after obtaining the text fragment vector, it is stored in a vector database; after extracting the non-text content from the Markdown format document, the extracted non-text content is stored in a relational database.

[0103] In some embodiments, the non-text content includes images, code, and tables.

[0104] In some embodiments, the method of segmenting plain text content according to Markdown semantic structure includes: segmenting plain text content by headings and paragraphs.

[0105] In some embodiments, after segmenting the plain text content according to the Markdown semantic structure, a total loss function is used to optimize the semantic coherence within the text fragments and the semantic distinguishability between different text fragments.

[0106] Specifically, the total loss function is expressed by the following formula:

[0107]

[0108]

[0109]

[0110] in, Represents the total loss function. This indicates a loss of content consistency. This indicates the loss of coherence in text segments. , Represents the coefficient. This indicates the number of sentences in the text segment. Sentence With sentences Semantic similarity between them This represents the weight of the i-th sentence. This indicates the total number of sentences included in the calculation. Indicates the scope of a comparison across text segments. This indicates the sentence at the end of the current text segment. Sentences located at the beginning of other text segments Perform semantic similarity. , Each represents a different text fragment. This indicates the number of sentences in a text segment.

[0111] In some embodiments, when a user performs a search, based on the user's input query request, text fragment vectors and associated non-text content in a relational database are retrieved, merged, and recombined to generate a context fragment. The method includes: vectorizing the user's input query request and performing a similarity search in a vector database to obtain the most relevant text fragment vectors; for each retrieved text fragment vector, checking its association information in the relational database; if the text fragment vector is associated with non-text content, retrieving the associated non-text content from the relational database; merging and recombining the content corresponding to the retrieved text fragment vector with the associated non-text content to generate a context fragment; and generating and displaying the answer using a large language model.

[0112] According to embodiments of the present invention, a document parsing system for RAG can correspond to performing the methods described in the embodiments of the present invention, and the above and other operations and / or functions of each module of the document parsing system for RAG are respectively for implementing Figure 1 For the sake of brevity, the corresponding processes of each method in the code will not be elaborated here.

[0113] See Figure 3The diagram shows the structure of a computer device, which includes a processor, a communication interface, and a computer-readable storage medium. The processor, communication interface, and computer-readable storage medium are connected via a bus or other means. The communication interface is used to receive and send data. The computer-readable storage medium can be stored in the computer device's memory. The computer-readable storage medium stores computer programs, including program instructions, and the processor executes the program instructions stored in the computer-readable storage medium. The processor (or CPU, Central Processing Unit) is the computing and control core of the computer device, adapted to implement one or more instructions, specifically adapted to load and execute one or more instructions to implement the corresponding steps in the RAG-oriented document parsing method embodiment.

[0114] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of hardware embodiments, software embodiments, or embodiments combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage and optical storage) containing computer-usable program code.

[0115] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0116] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0117] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0118] Those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware. The program can be stored in a computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. The storage medium can be a magnetic disk, optical disk, read-only memory (ROM), or random access memory (RAM), etc.

[0119] The above description is merely a preferred embodiment of the present invention and is not intended to limit the invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A document parsing method for RAG, characterized in that, include: The documents of different formats are uniformly normalized to obtain the Markdown format of all documents; The method involves uniformly normalizing the acquired documents of different formats to obtain the Markdown format for all documents; the method includes: Based on DOCX / DOC documents, a library is used to parse and recognize paragraphs, headings, lists, and tables, and convert them into Markdown format. Based on PDF documents, layout analysis is performed to identify text blocks, images, and tables. The structure of the PDF document is inferred based on the style information of text blocks, images, and tables on the page, and then converted into Markdown format. Based on PPTX / PPT documents, extract text and non-text content, and convert it to Markdown format according to the order and hierarchy of text and non-text content in the slides; the non-text content includes: images, code, and tables. Based on XLSX / XLS documents, convert them to Markdown format table syntax; Extract plain text content from a TXT document and convert it to Markdown format; Based on Markdown format documents, the plain text content is extracted from the Markdown format documents. The plain text content is then segmented according to the Markdown semantic structure to obtain several text fragments. All text fragments are vectorized. The non-text content is extracted from the Markdown format documents. The extracted non-text content is associated with the text fragment vectors and stored in a relational database. The method for segmenting plain text content according to Markdown semantic structure includes: segmenting plain text content by headings and paragraphs; For non-text elements, images, code, and table content in the MD are stored separately, and their positions and contexts in the original document are associated with them. When a user performs a search, based on the user's query request, text fragment vectors and associated non-text content are retrieved from a relational database, merged and recombined to generate a context fragment. The method includes: vectorizing the user's query request and performing a similarity search in a vector database to obtain the most relevant text fragment vectors; for each retrieved text fragment vector, checking its association information in the relational database; if the text fragment vector is associated with non-text content, retrieving the associated non-text content from the relational database; merging and recombining the content corresponding to the retrieved text fragment vector with the associated non-text content to generate a context fragment; and finally generating and displaying the answer using a large language model. After obtaining the text fragment vectors, they are stored in a vector database; after extracting the non-text content from the Markdown document, the extracted non-text content is stored in a relational database. After segmenting the plain text content according to the Markdown semantic structure, the total loss function is used to optimize the semantic coherence within the text segment and the semantic distinguishability between different text segments. The total loss function is expressed by the following formula: in, Represents the total loss function. This indicates a loss of content consistency. This indicates the loss of coherence in text segments. , Represents the coefficient. This indicates the number of sentences in the text segment. Sentence With sentences Semantic similarity between them This represents the weight of the i-th sentence. This indicates the total number of sentences included in the calculation. Indicates the scope of a comparison across text segments. This indicates the sentence at the end of the current text segment. Sentences located at the beginning of other text segments Perform semantic similarity. Both j represent different text segments. This indicates the number of sentences in a text segment.

2. A document parsing system for RAG, characterized in that, include: The normalization module is configured to: uniformly normalize the documents of different formats obtained, and obtain the Markdown format of all documents; The method involves uniformly normalizing the acquired documents of different formats to obtain the Markdown format for all documents; the method includes: Based on DOCX / DOC documents, a library is used to parse and recognize paragraphs, headings, lists, and tables, and convert them into Markdown format. Based on PDF documents, layout analysis is performed to identify text blocks, images, and tables. The structure of the PDF document is inferred based on the style information of text blocks, images, and tables on the page, and then converted into Markdown format. Based on PPTX / PPT documents, extract text and non-text content, and convert it to Markdown format according to the order and hierarchy of text and non-text content in the slides; the non-text content includes: images, code, and tables. Based on XLSX / XLS documents, convert them to Markdown format table syntax; Extract plain text content from a TXT document and convert it to Markdown format; The segmentation module is configured to: extract plain text content from Markdown format documents, segment the plain text content according to the Markdown semantic structure to obtain several text fragments, vectorize all text fragments; extract non-text content from Markdown format documents, associate the extracted non-text content with the text fragment vectors, and store them in a relational database. The method for segmenting plain text content according to Markdown semantic structure includes: segmenting plain text content by headings and paragraphs; For non-text elements, images, code, and table content in the MD are stored separately, and their positions and contexts in the original document are associated with them. The retrieval generation module is configured to: when a user performs a retrieval, retrieve text fragment vectors and associated non-text content from a relational database based on the user's query request, merge and reassemble them to generate a context fragment; the method includes: vectorizing the user's query request and performing a similarity search in the vector database to obtain the most relevant text fragment vectors; for each retrieved text fragment vector, checking its association information in the relational database; if the text fragment vector is associated with non-text content, retrieving the associated non-text content from the relational database; merging and reassembling the content corresponding to the retrieved text fragment vector with the associated non-text content to generate a context fragment; and generating and displaying the answer using a large language model. After obtaining the text fragment vectors, they are stored in a vector database; after extracting the non-text content from the Markdown document, the extracted non-text content is stored in a relational database. After segmenting the plain text content according to the Markdown semantic structure, the total loss function is used to optimize the semantic coherence within the text segment and the semantic distinguishability between different text segments. The total loss function is expressed by the following formula: in, Represents the total loss function. This indicates a loss of content consistency. This indicates the loss of coherence in text segments. , Represents the coefficient. This indicates the number of sentences in the text segment. Sentence With sentences Semantic similarity between them This represents the weight of the i-th sentence. This indicates the total number of sentences included in the calculation. Indicates the scope of a comparison across text segments. This indicates the sentence at the end of the current text segment. Sentences located at the beginning of other text segments Perform semantic similarity. Both j represent different text segments. This indicates the number of sentences in a text segment.

3. A computer device, characterized in that, A processor, adapted to execute computer programs; A computer-readable storage medium storing a computer program, which, when executed by the processor, implements the steps of the RAG-oriented document parsing method as described in claim 1.

Images