Search result processing methods, apparatus, equipment and computer-readable storage media

Abstract

This application discloses a method, apparatus, device, and computer-readable storage medium for processing search results. The method includes: S1, assigning a unique citation number to each document block in the search results; S2, generating basic citation data, which includes basic citation context and a basic citation list, wherein the content of each document block is formatted and its corresponding citation number is appended to generate the basic citation context, and a basic citation list is constructed based on the mapping relationship between each document block and each citation number, as well as the metadata and content of each document block; S3, using a large language model to analyze the content of each document block to extract existing knowledge points; S4, using the large language model to determine whether there are any missing knowledge points in the set of existing knowledge points. This application can make the answer traceable, improve the credibility of the answer, and determine whether there are any missing knowledge points in the set of existing knowledge points.

Description

Technical Field

[0001] This application relates to the field of retrieval technology, and in particular to a method, apparatus, device, and computer-readable storage medium for processing retrieval results. Background Technology

[0002] RAG (Retrieval-Augmented Generation) is an artificial intelligence system architecture that combines information retrieval and language generation, aiming to improve the accuracy, timeliness, and interpretability of large language models (LLMs) when answering questions.

[0003] A typical RAG system comprises two core components: a retrieval unit and a generator. The retrieval unit retrieves the most relevant information to the user's query from the knowledge base, employing methods such as dense vector retrieval (e.g., using Sentence-BERT, DPR), sparse retrieval (e.g., BM25), and hybrid retrieval. The retrieval results are usually several contextual fragments. The generator takes the user's question and the relevant context obtained from the retrieval as input and uses a large language model to generate a natural language response.

[0004] However, AI-generated answers currently lack traceability, affecting their credibility. Furthermore, AI currently does not perform knowledge integrity checks and cannot identify any knowledge gaps. Summary of the Invention

[0005] This application provides a method, apparatus, computer device, computer-readable storage medium, and computer program product for processing search results, which can solve at least one of the technical problems mentioned in the background art.

[0006] In view of this, firstly, embodiments of this application provide a method for processing search results in a RAG system, used to process search results obtained from searching a document chunking retrieval database based on user questions, the method comprising: S1, assign a unique reference number to each document block in the search results; S2, Generate basic citation data, which includes basic citation context and basic citation list. The basic citation context is generated by formatting the content of each document block and attaching its corresponding citation number. The basic citation list is constructed based on the mapping relationship between each document block and each citation number, as well as the metadata and content of the document block. S3, use a large language model to analyze the content of each document block to extract existing knowledge points and form a set of existing knowledge points; S4, the large language model determines whether there are any missing knowledge points in the set of existing knowledge points.

[0007] Optionally, the large language model determines whether there are any missing knowledge points in the set of existing knowledge points, including: S41, using the large language model to generate a set of expected knowledge points based on the user's question; S42, based on the set of existing knowledge points and the set of expected knowledge points, determine whether there are any missing knowledge points. If there are missing knowledge points, generate a missing knowledge point prompt.

[0008] Optionally, after step S4, the method further includes: S51, determine the corresponding document type based on the user question and the basic reference materials; S52, based on the determined document type, select the standard structure template corresponding to the document type from the preset standard structure templates, wherein each standard structure template is preset according to the document type; S53, Generate a corresponding standard document structure outline based on the selected standard structure template and the user question. The standard document structure outline includes each chapter structure unit and the knowledge points under each chapter structure unit. S54, Match the existing knowledge points with all knowledge points under each chapter structure unit; S55, Based on the matching results, calculate the coverage rate of knowledge points in each chapter structure unit; S56, determine whether the coverage rate has reached a set threshold. If it has not reached the set threshold, proceed to step S57. If it has reached the set threshold, proceed to step S58. S57, using the uncovered knowledge points of the chapter structure unit, search the document block retrieval database to obtain new search results, supplement the basic reference materials based on the new search results, extract knowledge points based on the supplemented content, and then return to step S54; S58, Return the final cited materials, which include the final cited context, the final cited reference list, and the knowledge point coverage report.

[0009] Optionally, the step of using the uncovered knowledge points of the chapter structure unit to obtain new search results based on the document block retrieval database includes: For each knowledge point that is not covered in the structural units of each chapter, a targeted query is constructed; For each of the targeted queries, a search is performed in the document block retrieval database to obtain new search results.

[0010] Optionally, supplementing the basic reference materials based on the new search results includes: For all the document blocks that have been obtained so far, reassign the reference numbers; The latest citation data is generated, which includes the latest citation context and the latest citation list. Specifically, the content of each document block is formatted and its corresponding reallocated citation number is appended to generate the latest citation context. The latest citation list is constructed based on the mapping relationship between all document blocks and the reallocated citation numbers, as well as the metadata and content of all document blocks.

[0011] Optionally, in step S51, if the corresponding document type cannot be determined based on the user question and the basic reference materials, a preset general template is used as the selected standard structure template to generate the standard document structure outline.

[0012] Optionally, the method further includes: Assign a Uniform Resource Identifier to each document block retrieved and create a corresponding citation page; The large language model analyzes the user question and the final output references to generate a search answer including several sentences. Each sentence in the search answer that comes from a document segment in the final output references is appended with the citation number of that document segment. When the citation number appended to the sentence in the search answer is clicked, it will redirect to the corresponding reference page through the corresponding Uniform Resource Identifier.

[0013] Secondly, embodiments of this application also provide a retrieval result processing apparatus for a RAG system, used to process retrieval results obtained from searching a document block retrieval database based on user questions, the apparatus comprising: The allocation module is used to assign a unique reference number to each document block in the search results; A generation module is used to generate basic citation data, which includes basic citation context and basic citation list. The basic citation context is generated by formatting the content of each document block and attaching its corresponding citation number. The basic citation list is constructed based on the mapping relationship between each document block and each citation number, as well as the metadata and content of the document block. The extraction module is used to analyze the content of each document block using a large language model to extract existing knowledge points and form a set of existing knowledge points. The judgment module is used by the large language model to determine whether there are any missing knowledge points in the set of existing knowledge points.

[0014] Thirdly, embodiments of this application also provide a computer device, including a memory and a processor; The memory is connected to the processor, the memory is used to store computer programs, and the processor is used to invoke the computer programs so that the computer device executes the retrieval result processing method described in any one of the first aspects.

[0015] Fourthly, embodiments of this application also provide a computer-readable storage medium storing a computer program adapted to be loaded by a processor and to execute the retrieval result processing method described in any one of the first aspects.

[0016] Fifthly, embodiments of this application also provide a computer program product, including a computer program that, when executed by a processor, implements the steps of the retrieval result processing method described in any one of the first aspects.

[0017] The aforementioned retrieval result processing method, apparatus, computer equipment, computer-readable storage medium, and computer program product assign a unique citation number to each document block in the retrieval results. Subsequently, each block of the basic citation context in the generated basic citation materials is appended with a citation number. The basic citation reference list of the basic citation materials sets the citation number for each document block. Therefore, when generating answers based on the basic citation materials, citation numbers can be appended to relevant sentences in the answer, making the answer traceable and improving its credibility. Furthermore, embodiments of this application can extract a set of existing knowledge points and use a large language model to determine whether there are any missing knowledge points in the set of existing knowledge points. Attached Figure Description

[0018] Figure 1 This is a flowchart illustrating the retrieval result processing method according to an embodiment of this application.

[0019] Figure 2 This is a flowchart illustrating the process of determining whether there are missing knowledge points in the existing set of knowledge points according to an embodiment of this application.

[0020] Figure 3 This is a partial flowchart illustrating the search result processing method of this application when executing strict mode.

[0021] Figure 4 This is a schematic diagram of the retrieval result processing device according to an embodiment of this application.

[0022] Figure 5 This is a schematic diagram of a portion of the modules of the retrieval result processing device in strict mode according to an embodiment of this application.

[0023] Figure 6This is a schematic diagram of a computer device according to an embodiment of this application. Detailed Implementation

[0024] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of this application.

[0025] Please see Figures 1 to 3 This application discloses a method for processing search results in a RAG system. This method processes search results obtained from searching a document chunk retrieval database based on user queries. For example, the document chunk retrieval database is a medical database built from various books, papers, guidelines, and manuals in the medical field. Document chunks are formed by dividing several documents (such as books). The document chunk retrieval database can include several document chunks and several document chunk vectors, allowing for simultaneous keyword and vector searches during retrieval. However, this is not a limitation. The database records metadata for each document chunk, which may include the filename of the file to which the document chunk belongs, the chapter in which the document chunk is located, and the starting page number of the document chunk within its file.

[0026] The method includes: S1 assigns a unique reference number to each document block in the search results.

[0027] For example, if the search results include 9 document blocks, you can assign citation numbers 1, 2, 3, 4, 5, 6, 7, 8, and 9 to the 9 document blocks respectively.

[0028] S2, Generate basic citation data, which includes basic citation context and basic citation list. In this process, the content of each document block is formatted and its corresponding citation number is appended to generate the basic citation context. The basic citation list is constructed based on the mapping relationship between each document block and each citation number, as well as the metadata and content of each document block.

[0029] It is understood that formatting involves converting the content of document blocks into a corresponding format, which is known to those skilled in the art and will not be elaborated upon here. The formatted block content, appended with corresponding citation numbers, forms the basic citation context in this application. The citation numbers can be added before or after the corresponding block content. For each document block, the basic citation list can record its citation number, related metadata, block content, etc.

[0030] S3 uses a large language model to analyze the content of each document block to extract existing knowledge points and form a set of existing knowledge points.

[0031] Specifically, large language models can be used to analyze the basic reference context to extract existing knowledge points.

[0032] S4, the large language model determines whether there are any missing knowledge points in the existing set of knowledge points.

[0033] Since the search results only take the top-ranked document blocks, it is easy to miss some knowledge points.

[0034] Please combine Figure 2 In some embodiments, the large language model determines whether there are missing knowledge points in the existing set of knowledge points, including: S41 utilizes a large language model to generate a set of expected knowledge points based on user questions.

[0035] It should be noted that the user question refers to the question entered by the user during the search. The large language model can search and analyze within its covered knowledge scope based on the user question, extracting the corresponding knowledge points. All the extracted knowledge points constitute the set of expected knowledge points, that is, the set of knowledge points that should be covered.

[0036] S42, based on the existing set of knowledge points and the expected set of knowledge points, determine whether there are any missing knowledge points. If there are missing knowledge points, generate a missing knowledge point prompt.

[0037] Specifically, to determine whether there are missing knowledge points based on the existing set of knowledge points and the expected set of knowledge points, we can directly remove the knowledge points in the expected set that correspond to the existing knowledge points. The remaining knowledge points in the expected set are the missing knowledge points in the existing set of knowledge points.

[0038] When there are missing knowledge points, the generated missing information can be used as a reference when generating the final search answer.

[0039] Please combine Figure 3 In some embodiments, after step S42, the method further includes: S51, determine the corresponding document type based on user questions and basic reference materials.

[0040] Specifically, AI analysis can be performed directly on user questions and sampled chapter names and content summaries (such as the first 3 cited chapter names and the first 150 words of the content) from the basic citation list to determine the corresponding document type.

[0041] For example, in the medical field, document types can include: medical textbooks, academic papers, technical tutorials, clinical guidelines, examination materials, and operation manuals.

[0042] In addition to outputting the document type, AI analysis can also output the content organization method, target audience, and knowledge structure characteristics of each document type.

[0043] For example, in the medical field, content organization methods can include: theoretical framework, experimental research, operational procedures, evidence-based recommendations, and knowledge point summaries. The target audience can include: students, researchers, clinicians, test-takers, and technical personnel. The characteristics of the knowledge structure can include: systematicity, practicality, operability, and authority.

[0044] S52, based on the determined document type, select the standard structure template corresponding to the document type from the preset standard structure templates, wherein each standard structure template is preset according to the document type.

[0045] For example, if the document type is determined to be a medical textbook, the corresponding standard structure template includes chapter structure units such as: basic theory, anatomy and physiology, etiology and pathology, clinical manifestations, diagnostic methods, treatment plans, prevention and prognosis.

[0046] If the corresponding document type cannot be determined based on the user's question and the basic reference materials, the preset general template will be used as the selected standard structure template to generate the subsequent standard document structure outline.

[0047] Specifically, if the user's question does not contain obvious words such as "textbook / guide / paper" and the analysis fails, then the general template (overview and background, core content, detailed analysis, practical application, summary and outlook, etc.) should be used directly.

[0048] S53 generates a corresponding standard document structure outline (using AI) based on the selected standard structure template and user questions. The standard document structure outline includes the structural units of each chapter and the knowledge points under each structural unit.

[0049] The output standard document structure outline can be in the form of: each element: {Chapter: Structural unit name, Knowledge point: [Key point 1, Key point 2, ...]}, which is a two-level tree structure of "Chapter → Knowledge point". The knowledge points can be generated by AI based on user questions and information from each chapter.

[0050] To facilitate understanding of steps S51 to S53, the following examples are provided: **User Question:** "What are the principles of diagnosis and treatment for hypertension?" **Document Type Analysis** Input: User question + several cited chapters and summaries (such as "essential hypertension", "diagnostic criteria", "antihypertensive drugs", etc.).

[0051] Output example: Medical textbooks | Theoretical framework | Clinicians | Systematic In other words: document type = medical textbook, organization method = theoretical system, target audience = clinicians, characteristics = systematic.

[0052] **Structural Guidance** Document type = Medical textbook → Use medical textbook template: basic theory, anatomy and physiology, etiology and pathology, clinical manifestations, diagnostic methods, treatment plans, prevention and prognosis, etc.

[0053] **Example of generating a standard document structure outline** Basic Theory | Definitions, Classification, Epidemiology Etiology and Pathology | Etiology, Pathogenesis, Risk Factors Clinical manifestations | symptoms, signs, complications Diagnostic methods | Blood pressure measurement, diagnostic criteria, differential diagnosis, laboratory tests Treatment options | Non-pharmacological treatment, antihypertensive drugs, combination therapy, surgical indications Prevention and prognosis | Lifestyle, follow-up, prognostic factors S54 matches the existing knowledge points with all the knowledge points under each chapter's structural unit.

[0054] For example, a chapter structure unit includes five knowledge points, and there are three existing knowledge points corresponding to that chapter structure unit, that is, three are matched.

[0055] S55. Based on the matching results, calculate the coverage rate of knowledge points in each chapter's structural unit.

[0056] Continuing with the matching example above, since the matching result is three matches, the corresponding coverage is 3 / 5 = 60%.

[0057] S56. Determine whether the coverage rate has reached the set threshold. If it has not reached the set threshold, proceed to step S57. If it has reached the set threshold, proceed to step S58.

[0058] If the threshold is set at 70%, then the 60% coverage in the previous example does not reach the set threshold.

[0059] S57, using the uncovered knowledge points in the chapter structure unit, search the document block retrieval database to obtain new search results, supplement the basic reference materials based on the new search results, extract knowledge points based on the supplemented content, and then return to step S54.

[0060] In some embodiments, new search results are obtained by utilizing the uncovered knowledge points of chapter structure units and searching based on a document chunk retrieval database, including: For each knowledge point that is not covered in the structural units of each chapter, a targeted query is constructed; For each targeted query, a search is performed in the document block retrieval database to obtain new search results.

[0061] The targeted query constructed above can be created by using AI to extract keywords from corresponding knowledge points, or it can be constructed as a vector query. The search can be performed in the document chunking retrieval database using the same method as the initial search. After obtaining new search results, steps S54 to S56 can be repeated until the coverage reaches a set threshold.

[0062] In some embodiments, supplementing the basic references based on new search results includes: For all the document blocks that have been obtained so far, reassign the reference numbers; The latest citation data is generated, which includes the latest citation context and the latest list of cited references. Specifically, the content of each document block is formatted and its corresponding reallocated citation number is appended to generate the latest citation context. The latest list of cited references is constructed based on the mapping relationship between all document blocks and the reallocated citation numbers, as well as the metadata and content of all document blocks.

[0063] The latest cited materials generated can be regarded as supplementary materials to the basic cited materials.

[0064] S58, Returns the final cited sources, which include the final citation context, the final list of cited references, and a report on knowledge coverage.

[0065] After the final cited data is output, the large language model can use the final cited data to generate answers for user searches, and based on the aforementioned final cited data, it can ensure the coverage of knowledge points in the answers.

[0066] It should be noted that steps S51 to S58 are not mandatory and can be executed only when the AI ​​determines that strict mode can be applied. The decision to apply strict mode can be made by judging whether the user's question contains specific keywords and whether the search area belongs to a high-risk area. Strict mode is applied only when both the high-risk area and keyword criteria are met.

[0067] Among them, specific keywords can be: Complete vocabulary: 'what is', 'what are', 'including', 'category', 'type' Systematic vocabulary: 'mechanism', 'principle', 'process', 'development', 'evolution' Comprehensive vocabulary: 'overview', 'summary', 'comprehensive', 'systematic', 'detailed' For high-risk areas, a list of high-risk areas can be set in advance, and AI can determine which areas belong to high-risk areas based on user questions.

[0068] In some embodiments, the method further includes: Each retrieved document chunk is assigned a Uniform Resource Identifier (URI) and a corresponding citation page is created. The citation page can contain metadata and content for the corresponding document chunk.

[0069] The large language model analyzes the user's question and the final output references to generate a search answer consisting of several sentences. Each sentence in the search answer that comes from a document segment in the final output references is appended with the corresponding document segment's citation number. When a sentence in the search answer is clicked, it is configured to redirect to the corresponding reference page via the corresponding Uniform Resource Identifier.

[0070] In this embodiment, a unique citation number is assigned to each document block in the search results. Subsequently, each block of the basic citation context in the generated basic citation materials is appended with a citation number. The citation number for each document block is set in the basic citation bibliography list of the basic citation materials. Therefore, when generating answers based on the basic citation materials, citation numbers can be appended to relevant sentences in the answer, making the answer traceable and improving its credibility. Furthermore, this embodiment can extract a set of existing knowledge points and use a large language model to determine whether there are any missing knowledge points in the set.

[0071] It should be understood that although the steps in the flowcharts of the embodiments described above are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the embodiments described above may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages of other steps.

[0072] Based on the same inventive concept, this application also provides a retrieval result processing apparatus for implementing the retrieval result processing method described above. The solution provided by this apparatus is similar to the implementation scheme described in the above method; therefore, the specific limitations / descriptions in the verification apparatus embodiments provided below can be found in the limitations / descriptions of the retrieval result processing method above, and will not be repeated here.

[0073] Please see Figure 4 The retrieval result processing apparatus of the RAG system in this application embodiment is used to process the retrieval results obtained by searching a document block retrieval database based on user questions. The apparatus includes: The allocation module 201 is used to assign a unique reference number to each document block in the search results; The generation module 202 is used to generate basic citation data, which includes basic citation context and basic citation list. The basic citation context is generated by formatting the content of each document block and attaching its corresponding citation number. The basic citation list is constructed based on the mapping relationship between each document block and each citation number, as well as the metadata and content of the document block. Extraction module 203 is used to analyze the content of each document block using a large language model to extract existing knowledge points and form a set of existing knowledge points; The judgment module 204 is used by the large language model to determine whether there are any missing knowledge points in the existing set of knowledge points.

[0074] In some embodiments, the large language model determines whether there are any missing knowledge points in the existing set of knowledge points, including: Generate a set of expected knowledge points based on user questions using a large language model; Based on the existing set of knowledge points and the expected set of knowledge points, determine whether there are any missing knowledge points. If there are missing knowledge points, generate a missing knowledge point prompt.

[0075] Please see Figure 5 In some embodiments, the apparatus further includes: Module 301 is defined to determine the corresponding document type based on user questions and basic reference materials. The selection module 302 is used to select a standard structure template corresponding to a document type from a set of preset standard structure templates based on a determined document type, wherein each standard structure template is preset according to the document type; The generation module 303 is used to generate a corresponding standard document structure outline based on the selected standard structure template and user questions. The standard document structure outline includes the structural units of each chapter and the knowledge points under each structural unit. Matching module 304 is used to match existing knowledge points with all knowledge points under each chapter's structural unit; The calculation module 305 is used to calculate the coverage rate of knowledge points in each chapter's structural unit based on the matching results. The judgment module 306 is used to determine whether the coverage rate has reached the set threshold. If it has not reached the set threshold, it enters the supplementation and extraction module 307. If it has reached the set threshold, it enters the return module 308. The supplementation and extraction module 307 is used to obtain new search results by using the uncovered knowledge points of the chapter structure unit and the document block retrieval database, and to supplement the basic reference materials based on the new search results and extract knowledge points based on the supplemented content, and then return to the matching module 304. Return to module 308, which returns the final cited information, including the final citation context, the final list of cited references, and a report on knowledge point coverage.

[0076] After the final cited data is output, the large language model can use the final cited data to generate answers for user searches, and based on the aforementioned final cited data, it can ensure the coverage of knowledge points in the answers.

[0077] In this embodiment, a unique citation number is assigned to each document block in the search results. Subsequently, each block of the basic citation context in the generated basic citation materials is appended with a citation number. The citation number for each document block is set in the basic citation bibliography list of the basic citation materials. Therefore, when generating answers based on the basic citation materials, citation numbers can be appended to relevant sentences in the answer, making the answer traceable and improving its credibility. Furthermore, this embodiment can extract a set of existing knowledge points and use a large language model to determine whether there are any missing knowledge points in the set.

[0078] In this application embodiment, the term "module" refers to a computer program or part of a computer program that has a predetermined function and works with other related parts to achieve a predetermined goal, and can be implemented wholly or partially using software, hardware (such as processing circuitry or memory), or a combination thereof. Similarly, a processor (or multiple processors or memory) can be used to implement one or more modules or units. Furthermore, each module or unit can be part of an overall module or unit that includes the functionality of that module or unit.

[0079] Figure 6 This is a schematic diagram of the structure of a computer device provided in an embodiment of this application. Figure 6As shown, the computer device may include a processor 601 and a memory 602. The memory 602 is connected to the processor 601 and is used to store computer programs. The processor 601 calls the computer programs to cause the computer device to execute the retrieval result processing method described in the above embodiments. Furthermore, the computer device may also include at least one communication bus 603. The communication bus 603 is used to implement communication between components. The memory 602 may be a high-speed RAM or a non-volatile memory, such as at least one disk storage device.

[0080] This application also provides a computer-readable storage medium storing a computer program adapted to be loaded by a processor and to execute the retrieval result processing method described in the above embodiments.

[0081] This application also provides a computer program product or computer program that includes computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the electronic device to perform the retrieval result processing method as described in the above embodiments.

[0082] It should be understood that, in the embodiments of this application, the processor may be a central processing unit (CPU), but it may also be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor.

[0083] Those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by hardware related to computer program instructions. The program can be stored in a computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. The storage medium can be a magnetic disk, optical disk, read-only memory (ROM), or random access memory (RAM), etc.

[0084] In the above embodiments, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0085] The above-disclosed examples are merely preferred embodiments of this application and should not be construed as limiting the scope of this application. Therefore, any equivalent variations made in accordance with the claims of this application shall fall within the scope of this application.

Claims

1. A method for processing search results in a RAG system, characterized in that, The method for processing search results obtained from searching a document chunk retrieval database based on user questions includes: S1, assign a unique reference number to each document block in the search results; S2, Generate basic citation data, which includes basic citation context and basic citation list. The basic citation context is generated by formatting the content of each document block and attaching its corresponding citation number. The basic citation list is constructed based on the mapping relationship between each document block and each citation number, as well as the metadata and content of the document block. S3, use a large language model to analyze the content of each document block to extract existing knowledge points and form a set of existing knowledge points; S4, the large language model determines whether there are any missing knowledge points in the set of existing knowledge points.

2. The retrieval result processing method according to claim 1, characterized in that, The large language model determines whether there are any missing knowledge points in the existing set of knowledge points, including: S41, using the large language model to generate a set of expected knowledge points based on the user's question; S42, based on the set of existing knowledge points and the set of expected knowledge points, determine whether there are any missing knowledge points. If there are missing knowledge points, generate a missing knowledge point prompt.

3. The retrieval result processing method according to claim 1 or 2, characterized in that, After step S4, the method further includes: S51, determine the corresponding document type based on the user question and the basic reference materials; S52, based on the determined document type, select the standard structure template corresponding to the document type from the preset standard structure templates, wherein each standard structure template is preset according to the document type; S53, Generate a corresponding standard document structure outline based on the selected standard structure template and the user question. The standard document structure outline includes each chapter structure unit and the knowledge points under each chapter structure unit. S54, Match the existing knowledge points with all knowledge points under each chapter structure unit; S55, Based on the matching results, calculate the coverage rate of knowledge points in each chapter structure unit; S56, determine whether the coverage rate has reached a set threshold. If it has not reached the set threshold, proceed to step S57. If it has reached the set threshold, proceed to step S58. S57, using the uncovered knowledge points of the chapter structure unit, search the document block retrieval database to obtain new search results, supplement the basic reference materials based on the new search results, extract knowledge points based on the supplemented content, and then return to step S54; S58, Return the final cited materials, which include the final cited context, the final cited reference list, and the knowledge point coverage report.

4. The retrieval result processing method according to claim 3, characterized in that, The step of utilizing the uncovered knowledge points of the chapter structure unit to obtain new search results based on the document block retrieval database includes: For each knowledge point that is not covered in the structural units of each chapter, a targeted query is constructed; For each of the targeted queries, a search is performed in the document block retrieval database to obtain new search results.

5. The retrieval result processing method according to claim 3, characterized in that, The supplementation of the basic reference materials based on the new search results includes: For all the document blocks that have been obtained so far, reassign the reference numbers; The latest citation data is generated, which includes the latest citation context and the latest citation list. Specifically, the content of each document block is formatted and its corresponding reallocated citation number is appended to generate the latest citation context. The latest citation list is constructed based on the mapping relationship between all document blocks and the reallocated citation numbers, as well as the metadata and content of all document blocks.

6. The retrieval result processing method according to claim 3, characterized in that, In step S51, if the corresponding document type cannot be determined based on the user question and the basic reference materials, the preset general template is used as the selected standard structure template to generate the standard document structure outline.

7. The method for processing search results according to any one of claims 1 to 6, characterized in that, The method further includes: Assign a Uniform Resource Identifier to each document block retrieved and create a corresponding citation page; The large language model analyzes the user question and the final output references to generate a search answer including several sentences. Each sentence in the search answer that comes from a document segment in the final output references is appended with the citation number of that document segment. When the citation number appended to the sentence in the search answer is clicked, it will redirect to the corresponding reference page through the corresponding Uniform Resource Identifier.

8. A retrieval result processing device for a RAG system, characterized in that, The apparatus for processing search results obtained from searching a document chunk retrieval database based on user questions includes: The allocation module is used to assign a unique reference number to each document block in the search results; A generation module is used to generate basic citation data, which includes basic citation context and basic citation list. The basic citation context is generated by formatting the content of each document block and attaching its corresponding citation number. The basic citation list is constructed based on the mapping relationship between each document block and each citation number, as well as the metadata and content of the document block. The extraction module is used to analyze the content of each document block using a large language model to extract existing knowledge points and form a set of existing knowledge points. The judgment module is used by the large language model to determine whether there are any missing knowledge points in the set of existing knowledge points.

9. A computer device, characterized in that, Including memory and processor; The memory is connected to the processor, the memory is used to store computer programs, and the processor is used to call the computer programs so that the computer device executes the retrieval result processing method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program adapted to be loaded by a processor and execute the retrieval result processing method according to any one of claims 1 to 7.

Images