RAG retrieval method and device based on document block attribute tag and computer readable storage medium

Abstract

The invention belongs to the technical field of data processing, and particularly relates to an RAG retrieval method and device based on document block attribute tags and a computer readable storage medium. The method comprises the following steps: checking that content main bodies of an input document are consistent, and renaming a document name into a format containing attribute information; dividing the document into smaller and manageable document blocks according to content logic or text structure blocks; marking attributes of the document blocks: automatically filling attribute information of each document block according to file name information of the renamed document, namely labeling; storing the document blocks and the corresponding attribute information into a vector database; the content required by the user is coded into vectors, similar fragments are retrieved in a vector database, and finally retrieved document blocks are obtained; and combination and answer generation. The problems that the relevance between the retrieved document and the question is not high or the relevance between the retrieved document or information is poor, and the generated answer may be inaccurate, even wrong answer and the like are solved.

Description

Technical Field

[0001] The present invention belongs to the technical field of data processing, and more specifically, relates to a RAG retrieval method, device and computer-readable storage medium based on document block attribute tags. Background Art

[0002] In the age of information explosion, especially in product document management and customer service scenarios, document content is complex, covering multiple product lines, different versions, dates, and other attributes. Traditional retrieval methods struggle to quickly locate the specific information users need, impacting the quality and efficiency of customer service. In recent years, retrieval methods based on document block attribute tags (RAG, Retrieval-Augmented Generation) have gradually gained attention. The RAG method is a natural language processing model that combines retrieval and generation techniques, primarily used to answer complex questions or generate fact-based text. The RAG model excels at tasks that require extensive background knowledge or factual information, compensating for the poor performance of language generation models in answering specialized knowledge or unfamiliar domains.

[0003] Chinese invention patent CN119003639A discloses a material extraction and generation method based on a large language model and multiple storage technologies. It uses an object database, a vector database, and a knowledge graph to store the collected original material. When a user enters text, relevant information is obtained from the object database, the vector database, and the knowledge graph respectively to achieve full-text accurate matching of the material, semantic similarity retrieval, and deep relationship mining. The text similarity and semantic similarity are calculated for the obtained information respectively. The obtained information is input into the large language model to generate corresponding materials, and the generated materials are scored using a scoring model. The multi-objective optimization value of each material is calculated by comprehensively considering the text similarity, semantic similarity, and score. The generated materials are sorted according to the multi-objective optimization value, and the highest ranking is taken as the final generated material.

[0004] Because the RAG model relies on accurately retrieving knowledge or relevant textual information and a large language model, the content retrieved will influence the large language model's responses. The accuracy of the retrieval phase directly impacts the performance of the generation phase. If the retrieved documents are not highly relevant to the question, or if the retrieved documents or information are poorly relevant, the generated answers may be inaccurate or even incorrect. Therefore, the retrieval of appropriate knowledge and relevant textual information is key to the effectiveness of the RAG model. Summary of the Invention

[0005] The present invention aims to overcome at least one of the deficiencies of the aforementioned prior art by providing a RAG retrieval method based on document block attribute tags. This method aims to significantly improve the efficiency and accuracy of document retrieval by combining and optimizing steps such as document organization, segmentation, attribute tag generation, storage, retrieval, and deduplication. The method is particularly suitable for processing large amounts of document data and can effectively manage and retrieve document information, particularly in product document management and customer service scenarios. By combining vector retrieval with attribute retrieval, the method of the present invention enables both approximate and precise retrieval, significantly improving retrieval quality and efficiency and ensuring that users obtain both comprehensive and accurate information.

[0006] The detailed technical solutions of the present invention are as follows: A RAG retrieval method based on document block attribute tags, the method comprising: S1. Check that the input document content is consistent, i.e., each document contains only a single product information, and rename the document name to include attribute information, including but not limited to model name, version number, date, and product line name; S2. Document chunking: Divide the document into smaller, manageable chunks based on content logic or text structure. S3. Marking document block attributes / labels: Automatically fill in the attribute information of each document block, i.e., label it, based on the file name information of the renamed document to enhance the retrieval and classification of the document block; S4. Storing the document block and the corresponding attribute information together in a vector database to facilitate subsequent retrieval operations and ensure rapid location and extraction of document information; S5. Encode the content of the user's request into a vector, and search the vector database for similar fragments to obtain the final retrieved document block; S6. Combination and answer generation: Based on the prompt word template combination, the final retrieved document block and the user demand question are combined, input into the large language model together, and the answer is generated and presented to the user.

[0007] Furthermore, the S2 specifically includes: S21. Analyze the document structure. Perform structural analysis on each document and identify structural elements. Structural elements mark different parts of the document content and serve as an important basis for block division. Structural elements include titles, subtitles, lists, and tables. S22, title-guided segmentation, using the identified title as the starting point for segmentation, and the content under each title is considered as an independent segment; S23. Content coherence is determined by setting a coherence threshold. If a title has too many paragraphs and texts exceeding the coherence threshold, the title is segmented into paragraphs, with each segmented paragraph being a block. Otherwise, the content under each title is a block. Finally, title classification information is added to each segmented block. S24, determining the block boundaries, determining the start and end boundaries of each block based on the principle that the content within the block at the boundary is complete and relevant; S25. Preliminary sorting of blocks: Perform preliminary sorting and inspection on the divided blocks to ensure that the content of each block is clear and complete.

[0008] Furthermore, the S4 specifically includes: S41, identifying document blocks and attribute information, ensuring that the document block attribute / labeling step has been completed; if not, repeat S1 to S3; S42. Selecting a vector database to be used according to the characteristics of the document and the retrieval requirements; S43. Select an Embedding Model: If the document corresponds to industry content, select an embedding model that is specific or fine-tuned for a specific field / industry; if the document does not involve professional knowledge, select a general embedding model. S44. Use the selected Embedding model to convert the document block content into a vector representation, and store it together with the attribute / label information of the document block in the used vector database and retrieve it for use together.

[0009] Furthermore, the S5 specifically includes: S51: Preset a prompt word template and input it into a large language model for analysis together with the user's needs to extract document block attribute information in the user's needs; preferably, the preset prompt word template is a combination of attribute information; S52, encoding the content of the user's request into a vector, searching for similar fragments in the vector database, using the vector similarity threshold as a relevance criterion, and searching for similar document blocks with similarity higher than the similarity threshold; S53, attribute information filtering: The document blocks retrieved in S52 are filtered according to the attribute information extracted in S51. Only document blocks that fully match the attributes specified in the user query are retained, ensuring the accuracy and relevance of the retrieval results.

[0010] S54. Deduplication of document blocks: Group similar document blocks based on attribute information to obtain the final retrieved document blocks, avoid redundant information, and improve retrieval efficiency and the answer effect of the large language model.

[0011] Furthermore, the S54 specifically includes: According to the document block attribute information in the user demand extracted by the large language model, all document blocks related to the user demand information are grouped together; Then, using the preset similarity threshold and embedding model, the content similarity of document blocks in the same group is compared; if the content similarity of two document blocks exceeds the threshold, one document block is retained and the other is removed; if the content similarity of two document blocks does not exceed the threshold, both document blocks are retained; Finally, the final retrieved document block is obtained.

[0012] In another aspect of the present invention, a device for a RAG retrieval method based on document block attribute tags is provided, the device comprising: at least one processor; and a memory storing instructions, which, when executed by the at least one processor, enable the at least one processor to execute the RAG retrieval method based on document block attribute tags as described above.

[0013] In another aspect of the present invention, a computer-readable storage medium is provided, which stores executable instructions. When the instructions are executed, the machine executes the RAG retrieval method based on document block attribute tags as described above.

[0014] Compared with the prior art, the present invention has the following beneficial effects: (1) The present invention provides a RAG retrieval method, device and computer-readable storage medium based on document block attribute tags. It does not require a knowledge graph or the extraction of entities and other content. It is convenient and concise to process and apply. It only needs to extract the attribute information content of the document block according to user needs. It also uses vector retrieval and attribute retrieval, which complement each other and can achieve both precise retrieval and approximate retrieval.

[0015] (2) The present invention provides a RAG retrieval method, device and computer-readable storage medium based on document block attribute labels, which uses a vector similarity threshold as a retrieval relevance criterion instead of a TOPK retrieval method, thereby solving the problem of incomplete topk retrieval. At the same time, similar document block removal enables the large language model to have a better answer effect. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1 This is a flowchart of a RAG retrieval method based on document block attribute tags described in the present invention.

[0017] Figure 2 This is a schematic diagram of extracting document block attribute information in user requirements in embodiment 1 of the present invention.

[0018] Figure 3This is a schematic diagram of querying similar fragments based on extracted attribute information in embodiment 1 of the present invention.

[0019] Figure 4 This is a schematic diagram of large model combination and answer generation after document block deduplication in Example 1 of the present invention. DETAILED DESCRIPTION

[0020] The present invention will be further described below with reference to the accompanying drawings and embodiments.

[0021] It should be noted that the following detailed descriptions are exemplary and intended to provide further explanation of the present invention. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as commonly understood by those skilled in the art to which the present invention belongs.

[0022] It should be noted that the terms used herein are only for describing specific embodiments and are not intended to limit the exemplary embodiments according to the present invention. As used herein, unless the context clearly indicates otherwise, the singular form is intended to include the plural form. In addition, it should be understood that when the terms "comprise" and / or "include" are used in this specification, they indicate the presence of features, steps, operations, devices, components and / or combinations thereof.

[0023] In the absence of conflict, the embodiments of the present invention and the features thereof may be combined with each other.

[0024] Example 1 Ginseng Figure 1 This embodiment provides a RAG retrieval method based on document block attribute tags, the method comprising: S1. Document sorting: Check the consistency of the input document content and rename the document; Ensure that the main content of each document is consistent. That is, each document only contains information about a single product. Avoid mixing information from different products. For example, take Product A as an example. Rename the document to include attribute information, including but not limited to model name, version number, date, and product line name, to ensure standardization and identifiable document information. For example, "Model A Manual_A_Manual_20240101_1.1.docx." This step aims to standardize document naming for easier management and retrieval.

[0025] S2. Document segmentation: Divide the document into smaller, manageable document blocks according to content logic or text structure to facilitate subsequent attribute tagging and retrieval.

[0026] Preferably, the document segmentation specifically includes: S21. Analyze document structure: Perform structural analysis on each document to identify structural elements such as titles, subtitles, lists, and tables. Structural elements mark different parts of the document content and are an important basis for segmentation.

[0027] S22. Title-guided chunking: Using the identified titles as the starting point for chunking, the content under each title is considered an independent chunk. For example, if a document has a title called "Product Features," all content under the title "Product Features" will be divided into one chunk.

[0028] S23. Content coherence judgment: In some cases, the content under a heading may be too long and need to be further divided.

[0029] Content coherence judgment: set a coherence threshold. If there are too many paragraphs and texts under a title, which is greater than the coherence threshold, the content will be segmented according to paragraphs, and each segmented paragraph will be a block; otherwise, the content under each title will be a block.

[0030] Add title and classification information to each block after segmentation. For example, if a block is under "Design Plan - Software Design Plan", then the "Design Plan - Software Design Plan" information needs to be added to the block title attributes.

[0031] At this point, the long block is further broken down into smaller blocks based on content coherence and thematic consistency.

[0032] S24. Block boundary determination: Block boundaries are determined based on the principle that the content within the block at the boundary is complete and relevant, and the start and end boundaries of each block are determined to avoid information fragmentation.

[0033] S25. Preliminary sorting of blocks: Conduct preliminary sorting and inspection of the divided blocks to ensure that the content of each block is clear and complete without omissions or duplications.

[0034] Through the above steps, the present invention divides the original document into a series of document blocks with clear structure and relevant content, laying the foundation for subsequent steps such as attribute tagging, storage and retrieval.

[0035] S3. Mark document block attributes / labels: Automatically fill in the attribute information of each document block, i.e., label it, based on the file name information of the renamed document, such as model name, version number, date, product line name, to enhance the retrieval and classification of the document block.

[0036] Specifically, after the document is segmented, each document block is populated with attribute information based on the document file name information, which is the act of "labeling" the document block. This process involves identifying and extracting key attributes in the file name, such as model name, version number, date, product line name, etc., and associating these attributes with the corresponding document block as tags or metadata; For example, for the file name "Model A's Manual_A_Manual_20240101_1.1.docx," "Model A's Manual," "A," "Manual," "20240101," and "1.1" are used as attribute information for the original file name, model name, document content type, date, and version number, respectively. This structured information is then written to the vector database along with the document block for retrieval and management. This step leverages the structured information in the file name to automatically add detailed attribute tags to the document block, improving retrieval efficiency and accuracy.

[0037] S4. Document storage: The document block and the corresponding attribute information are stored in the vector database to facilitate subsequent retrieval operations and ensure the rapid location and extraction of document information.

[0038] Specifically, document storage generally requires preparing document blocks and attribute information, selecting a database, selecting an embedding model, vectorizing data, and storing it in the database. For example: S41. Identify document blocks and attribute information, ensuring that the attribute / labeling step for document blocks has been completed and that each document block has been filled with corresponding attribute information, such as model name, version number, date, product line name, etc. Missing information is completed or specified characters are used to mark and fill in the missing information. For example, undetermined version numbers can be set to 0.0, etc., for subsequent retrieval processing; S42. Select database: Determine the vector database to be used, such as Chroma, and select the appropriate database type based on the characteristics of the document and retrieval requirements.

[0039] S43. Select an embedding model based on the domain, complexity, and semantic requirements of the document content: If the document corresponds to industry content, choose an embedding model that is specific or fine-tuned for a specialized domain / industry; if the document does not involve professional knowledge, choose a general embedding model; S44. Data vectorization and storage: Use the selected Embedding model to convert the document block content into a vector representation, and store it together with the attribute / label information of the document block in the vector database used and retrieve it together.

[0040] S5. Encode the user's requested content into a vector and search the vector database for similar fragments to obtain the final retrieved document block. This specifically includes: S51. Extracting attribute information: A preset prompt word template is input into the Large Language Model (LLM) along with the user's requirements for analysis, extracting document block attribute information in the user's requirements, such as date, document content category, model, etc. The Large Language Model (LLM) can also be replaced by a VLM model or a VLA model. The prompt word template varies depending on the search task and application. For example, if the search task and application is to retrieve product information, the prompt word template for the task of extracting product models can be set as: User's question: "What product models are included in the following questions?" Note (prompt template): If the product abbreviation is three digits, please add XXX in front; if there are multiple model numbers, list each model number and enclose them in square brackets; if there is no model number in the question, the output model number is NA.

[0041] For example, question 1: Introduce YYY. In this question, the product model should be 'XXXYYY'.

[0042] For example, question 2: What is the difference between YYY1 and YYY2? In this question, the product model should be ['XXXYYY1', 'XXXYYY2'].

[0043] The user's question is combined with the preset prompt word template and fed into the large language model. The large language model then extracts the document block attributes corresponding to the user's request. In the example above, the combination of the prompt word template and the user's request allows the large language model to analyze the model information contained in the user's request.

[0044] During the retrieval phase, the Large Language Model (LLM) is first used to analyze document attribute requirements implicitly or explicitly mentioned in the user's query, such as specific dates, document types, product models, etc. This process involves in-depth analysis of the natural language input by the user to identify key information related to the document attributes.

[0045] This information is then structured and converted into a specific format, such as JSON, containing key-value pairs such as {"model":"A","version number":"1.0"}. This provides precise query conditions for subsequent attribute information filtering and document block retrieval. This step is critical to ensuring that search results closely match user needs. By accurately extracting and structuring attribute information, search efficiency and accuracy can be significantly improved.

[0046] The large language model mentioned in this article does not restrict its name, architecture, or size. It focuses on the process of using the large language model as a tool. It can be the same large language model or different large language models. Typically, to conserve large language model resources, a large language model with larger parameters is used, such as a large language model with 72 bytes of parameters.

[0047] S52. Query similar segments: Encode the content according to the user's requirements into vectors, retrieve similar segments in the vector database, and design a vector similarity threshold as a relevance standard to query similar segments, i.e., similar document blocks.

[0048] The document blocks are converted into vectors and stored. The retrieval process encodes the user's question into a vector and performs a similarity calculation with the vector stored in S44 to find similar segments, or similar blocks. The specific similarity calculation method is not described here, and this patent does not cover this part of the algorithm.

[0049] This vector is then input into a constructed vector database for retrieval, searching for document blocks whose similarity to the user requirement vector exceeds a preset threshold. Similarity is typically determined based on the cosine similarity between vectors to determine the relevance of the document block to the user requirement. Preferably, the vector database is Chroma.

[0050] S53, attribute information filtering: The search results of S52 are filtered according to the attribute information extracted in S51. Only document blocks that fully match the attributes specified in the user query will be retained, ensuring the accuracy and relevance of the search results.

[0051] Specifically, the results of filtering S52 include: First, in step S52, similar document blocks are searched using the vector. This retrieves document blocks that are related to the user's question or have similar semantics. A document block is the text in the original document. Typically, multiple document blocks are similar to the user's question or need, and multiple document blocks can be retrieved.

[0052] Then, the document blocks are screened and filtered based on their attribute information, because S52 only obtains similar texts, but it cannot be accurate. For example, "apple" and "an apple" are similar in terms of textual semantics, but Wolong Mountain and Wohu Mountain are two different entities, with only one character difference between them. The algorithm will think that the two are similar in textual semantics. This problem can be solved by filtering attribute information. For example, the user asks about the camera parameters of model AB, but model AC and model AB only differ by one letter. In the process of S52 retrieval, model AC and model AB will be retrieved. The role and purpose of S53 is that the user accesses the camera parameters of model AB, and the content retrieved by S52 is filtered out through the "model" attribute of the document block to obtain the document block of model AB that the user wants to ask about. The extraction and storage of the "model" attribute of the document block have been explained in steps S3 and S4.

[0053] After obtaining document blocks with similarity to the user demand vector higher than the threshold, these document blocks are further screened using the attribute values ​​extracted by the large language model; This step ensures that only document blocks that exactly match the attributes specified in the user query are retained, thereby improving the accuracy of the search results. For example, if the user query specifies a specific "model" and "version number", only document blocks that have the same "model" and "version number" attributes will be selected.

[0054] S54, Document Block Deduplication: Group similar document blocks based on attribute information to avoid redundant information, improve retrieval efficiency, and enhance the performance of the large language model. After attribute filtering, the remaining document blocks are deduplicated to avoid providing duplicate or highly similar information to the large language model, which could interfere with its understanding and response.

[0055] Document block deduplication: S53 can accurately obtain the document blocks that users want. For example, if a user wants information on model A, S53 can eliminate information on model B. However, model A's information is likely to contain redundant information. For example, there are three documents for model A: a design plan document, a user manual, and a promotional brochure. All three documents contain camera parameter information for model A. For example, the design plan mentions camera parameters in the camera selection and design section, the user manual mentions camera parameters in the specifications section, and the promotional brochure may also contain camera parameter information. Therefore, all three parameters are redundant. The more redundant information there is, the more redundant information is fed into the large language model, the less effective information there is, and the worse the output of the large language model.

[0056] This process groups the document blocks based on their attribute values, for example: According to the document block attribute information in the user demand extracted by the large language model, all document blocks related to the user demand information are grouped together; Then, using a preset similarity threshold and embedding model, the content similarity of document blocks within the same group is compared. If the content similarity of two document blocks exceeds the threshold, indicating that the information they contain is highly similar, one document block will be retained and the other will be removed to reduce redundant information.

[0057] Finally, the final retrieved document block is obtained.

[0058] This process usually uses metrics such as cosine similarity to determine the similarity between document blocks, ensuring that the information ultimately provided to users is comprehensive and non-redundant, thereby improving the quality and efficiency of retrieval results.

[0059] S6. Combination and answer generation: Based on the prompt word template combination, the final retrieved document block and the user demand question are combined, input into the large language model together, and the answer is generated and presented to the user.

[0060] The final retrieved document block is combined with the user's question through a prompt word template after vector / semantic similarity query, attribute screening, and removal of duplicate / redundant information, and is sent to the large language model.

[0061] Example of prompt word template: Answer the questions based on the known information. Be clear and logical. Do not add fabricated information to your answers. Please answer in Chinese.

[0062] Question: The user's original question.

[0063] Known information: Document 1: Content (retrieved document chunk 1); Document 2: Content (retrieved document chunk 2); Specifically, after deduplication of document blocks, the user's query content is combined with the content of the document blocks retained after screening. This process integrates the user's conversation content with the content of multiple document blocks to form a coherent and comprehensive content. This integrated information is then input into the large language model, which uses its powerful text generation capabilities to generate a clear, accurate, and easy-to-understand answer. Based on the input information, the large language model comprehensively considers the context and language of the document block to generate a natural and fluent text to answer the user's question. Ultimately, this answer is presented to the user, completing the entire retrieval and answer generation process based on the document block attribute labels.

[0064] Specifically, this embodiment takes the topics of Fengxianshan Reservoir and Gangshan Reservoir as an example to extract document block attribute information in user needs, such as Figure 2 As shown, Figure 2 The obscured text is confidential and cannot be displayed. It includes: User Question: Which one has a larger watershed area, Fengxianshan Reservoir or Gangshan Reservoir? Input to the big model, model extraction: Which reservoir names are included in the following questions? If there are multiple reservoir names, list each reservoir name and enclose it in square brackets; if there is no reservoir name in the question or the reservoir name cannot be determined, the output reservoir name is NA.

[0065] For example, question 1: Introduce Luohan Reservoir. The name of the reservoir in question 1 should be ['Luohan Reservoir'].

[0066] For example, question 2: Where are Fengxianshan Reservoir and Gangshan Reservoir respectively? In question 2, the reservoir name should be ['Fengxianshan Reservoir', Gangshan Reservoir'.

[0067] The above user question: Which one has a larger basin area, Fengxianshan Reservoir or Gangshan Reservoir? Extraction results: {'Reservoir Name': ['Fengxianshan Reservoir', 'Gangshan Reservoir']}.

[0068] Furthermore, in this embodiment, similar fragments are searched based on the extracted attribute information. Figure 3 As shown, taking Fengxianshan Reservoir as an example: According to the extracted attribute information, including the file title Water Conservancy-Fengxianshan Reservoir-Fengxianshan Reservoir*Plan-*Pre-Plan-v*-.docx:'Model Name':'Fengxianshan Reservoir','File Title':'Fengxianshan Reservoir*Plan','File Type':'*Pre-Plan','Date':*,'Version Number':'*','Industry Name':'Water Conservancy'.

[0069] Similar query segments include: |project|characteristic indicator|characteristic indicator|unit|current status|remarks|remarks|, which corresponds to: |reservoir|controlled watershed area|controlled watershed area|*|*||empty|empty|.

[0070] Furthermore, in this embodiment, after the content is filtered, an answer is generated for the large model, such as Figure 4 As shown: user: Which one has a larger basin area, Fengxianshan Reservoir or Gangshan Reservoir? Document Contents: Fengxianshan Reservoir; 1.3 Reservoir Overview: 1.3.1 Overview of the reservoir; Fengxianshan Reservoir, including its controlled basin area, reservoir type, construction start time, completion time, reservoir project grade, main buildings, design flood control standards, verification standards, verification flood level, total storage capacity, design flood control level, flood control storage capacity, beneficial water level, beneficial storage capacity, dead water level, dead storage capacity and multi-year average runoff.

[0071] 1.3.2 Engineering level; Fengxianshan Reservoir Project Level.

[0072] 1.3.3 Flood control standards; Flood control standards for reservoir projects.

[0073] Gangshan Reservoir: 2. Overview 2.1 Overview of the reservoir project 2.1.1 Basin Overview Gangshan Reservoir, including its location, reservoir type, reservoir-controlled basin area, main river channel length and mainstream slope.

[0074] The method described in this embodiment extracts the problem: Which one has a larger basin area, Fengxianshan Reservoir or Gangshan Reservoir? The method described in this embodiment generates the answer: The drainage area of ​​Gangshan Reservoir is larger than that of Fengxianshan Reservoir. Specifically, the drainage area of ​​Gangshan Reservoir is *, while the drainage area of ​​Fengxianshan Reservoir is *.

[0075] Among them, * indicates that the content is inconvenient to display.

[0076] Example 2 This embodiment provides a device for implementing a RAG retrieval method based on document block attribute tags, the device comprising: at least one processor; and a memory storing instructions, which, when executed by the at least one processor, enable the at least one processor to execute the RAG retrieval method based on document block attribute tags as described above.

[0077] In this embodiment, electronic devices include but are not limited to: personal computers, server computers, workstations, desktop computers, laptop computers, notebook computers, mobile computing devices, smart phones, tablet computers, cellular phones, personal digital assistants (PDAs), handheld devices, messaging devices, wearable computing devices, consumer electronic devices, etc.

[0078] Example 3 This embodiment further provides a computer-readable storage medium storing executable instructions, which, when executed, enable the machine to execute the RAG retrieval method based on document block attribute tags as described above.

[0079] Specifically, a system or device equipped with a readable storage medium can be provided, on which software program codes that implement the functions of any of the above-mentioned embodiments are stored, and a computer or processor of the system or device can read and execute instructions stored in the readable storage medium.

[0080] In this case, the program code itself read from the computer-readable medium can realize the function of any one of the above embodiments, and thus the computer-readable code and the computer-readable storage medium storing the computer-readable code constitute part of this specification.

[0081] Examples of readable storage media include floppy disks, hard disks, magneto-optical disks, optical disks (e.g., CD-ROMs, CD-Rs, CD-RWs, DVD-ROMs, DVD-RAMs, DVD-RWs, DVD-RWs), magnetic tapes, non-volatile memory cards, and ROMs. Alternatively, the program code may be downloaded from a server computer or a cloud via a communication network.

[0082] Obviously, the above embodiments of the present invention are merely examples for the purpose of clearly illustrating the technical solutions of the present invention, and are not intended to limit the specific implementation methods of the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the claims of the present invention shall be included within the scope of protection of the claims of the present invention.

Claims

1. A RAG retrieval method based on document block attribute tags, characterized in that: The method comprises: S1. Check that the input document content is consistent, i.e., each document contains only a single product information, and rename the document name to include attribute information, including but not limited to model name, version number, date, and product line name; S2. Divide the document into smaller, manageable chunks based on content logic or text structure; S3. Marking document block attributes / labels: Automatically fill in the attribute information of each document block, i.e., label it, based on the file name information of the renamed document to enhance the retrieval and classification of the document block; S4. Storing the document block and the corresponding attribute information together in a vector database to facilitate subsequent retrieval operations and ensure rapid location and extraction of document information; S5. Encode the content of the user's request into a vector, and search the vector database for similar fragments to obtain the final retrieved document block; S6. Based on the prompt word template combination, the final retrieved document block and the user's demand question are combined and input into the large language model together to generate an answer, and the answer is presented to the user.

2. The RAG retrieval method based on document block attribute tags according to claim 1, characterized in that: The S2 specifically includes: S21. Analyze the document structure. Perform structural analysis on each document and identify structural elements. Structural elements mark different parts of the document content and serve as an important basis for block division. Structural elements include titles, subtitles, lists, and tables. S22, title-guided segmentation, using the identified title as the starting point for segmentation, and the content under each title is considered as an independent segment; S23. Content coherence is determined by setting a coherence threshold. If a title has too many paragraphs and texts exceeding the coherence threshold, the title is segmented into paragraphs, with each segmented paragraph being a block. Otherwise, the content under each title is a block. Finally, title classification information is added to each segmented block. S24, determining the block boundaries, determining the start and end boundaries of each block based on the principle that the content within the block at the boundary is complete and relevant; S25: Preliminary sorting of blocks: preliminary sorting and checking of the divided blocks.

3. The RAG retrieval method based on document block attribute tags according to claim 2, characterized in that: The S4 specifically includes: S41, identifying document blocks and attribute information, ensuring that the document block attribute / labeling step has been completed; if not, repeat S1 to S3; S42. Selecting a vector database to be used according to the characteristics of the document and the retrieval requirements; S43. Select an Embedding Model: If the document corresponds to industry content, select an embedding model that is specific or fine-tuned for a specific field / industry; if the document does not involve professional knowledge, select a general embedding model. S44. Use the selected Embedding model to convert the document block content into a vector representation, and store it together with the attribute / label information of the document block in the used vector database and retrieve it for use together.

4. The RAG retrieval method based on document block attribute tags according to claim 3, characterized in that: The S5 specifically includes: S51: Preset a prompt word template and input it into a large language model for analysis together with the user's needs to extract document block attribute information in the user's needs; the preset prompt word template is a combination of attribute information; S52, encoding the content of the user's request into a vector, searching for similar fragments in the vector database, using the vector similarity threshold as a relevance criterion, and searching for similar document blocks with similarity higher than the similarity threshold; S53, attribute information filtering, filtering the document blocks retrieved in S52 according to the attribute information extracted in S51, and only document blocks that fully match the attributes specified in the user query will be retained; S54, document block deduplication: similar document blocks are grouped and removed according to attribute information to obtain the final retrieved document blocks.

5. The RAG retrieval method based on document block attribute tags according to claim 4, characterized in that: The S54 specifically includes: According to the document block attribute information in the user demand extracted by the large language model, all document blocks related to the user demand information are grouped together; Then, using the preset similarity threshold and embedding model, the content similarity of document blocks in the same group is compared; if the content similarity of two document blocks exceeds the threshold, one document block is retained and the other is removed; if the content similarity of two document blocks does not exceed the threshold, both document blocks are retained; Finally, the final retrieved document block is obtained.

6. A device for a RAG retrieval method based on document block attribute tags, characterized in that: The device comprises: processor; and a memory having stored thereon a computer program executable on said processor; Wherein, when the computer program is executed by the processor, the steps of the RAG retrieval method based on document block attribute tags as described in any one of claims 1 to 5 are implemented.

7. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 5 are implemented.

Images