A keyword coverage perception-based search enhancement generation context construction method

Abstract

This application relates to the field of artificial intelligence technology, specifically to a method for enhancing retrieval context construction based on keyword coverage awareness. The method includes: obtaining a query statement and extracting multiple query keywords; performing a retrieval operation based on the query keywords to obtain an initial candidate document set; statistically analyzing the occurrence of query keywords in each initial candidate document and calculating the coverage value of each initial candidate document for the query keywords; sorting the initial candidate document set and selecting several candidate documents; determining whether the overall coverage value of the candidate documents for the query keywords meets a coverage threshold; if not, identifying uncovered query keywords and re-performing the retrieval operation; extracting text fragments from the candidate documents, constructing a context, and inputting it into a preset large language model to generate an answer statement for the query statement. This application improves the completeness of retrieval results and the accuracy of context construction.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology, and specifically to a method for constructing a search-enhanced context based on keyword coverage awareness. Background Technology

[0002] Retrieval-enhanced generative techniques have become an important research direction in the field of artificial intelligence in recent years. The core idea is to retrieve relevant documents from external knowledge bases to provide contextual information to large language models, thereby generating more accurate and reliable answers. This technique effectively alleviates the shortcomings of large language models in terms of knowledge timeliness, domain specialization, and factual accuracy.

[0003] Existing retrieval enhancement systems typically employ semantic similarity-based retrieval strategies, ranking and filtering candidate documents based on the overall semantic match between the query and the document. However, this approach has significant limitations: when a user query contains multiple key information points, traditional retrieval methods often only focus on the document most semantically similar to the query, neglecting comprehensive coverage of all keywords in the query. This may result in retrieved documents that only partially answer the query, omitting other important information points of interest to the user, leading to incomplete or missing information in the final generated answer.

[0004] Furthermore, existing technologies lack a mechanism for perceiving and responding to keyword coverage when constructing context, making it impossible to dynamically adjust retrieval strategies to supplement missing information. When candidate documents in the initial retrieval fail to cover all key concepts in the query, the system struggles to identify and remedy this problem, directly resulting in the large language model lacking necessary contextual support when generating responses.

[0005] Therefore, how to achieve comprehensive coverage of query keywords and construct a more complete and accurate context in the retrieval enhancement generation system has become an urgent technical problem to be solved. Summary of the Invention

[0006] In view of the aforementioned problems, this application is proposed to provide a keyword coverage-aware retrieval enhancement context construction method that overcomes or at least partially solves the aforementioned problems, comprising: The above-mentioned objective of this application is achieved through the following technical solution: A method for constructing a search enhancement context based on keyword coverage awareness includes the following steps: Get the query statement entered by the user and extract multiple query keywords from the query statement; Based on a pre-defined knowledge base, a retrieval operation is performed on the query keywords to obtain an initial candidate document set including at least one initial candidate document; The occurrence of query keywords in each initial candidate document is statistically analyzed, and the coverage value of each initial candidate document to the query keywords is calculated based on the occurrence. The initial candidate document set is sorted according to the coverage value, and several candidate documents are selected. Determine whether the overall coverage value of the selected candidate documents for the query keywords meets the preset coverage threshold. If not, identify the uncovered query keywords and re-execute the search operation for the uncovered query keywords, and add the supplementary documents obtained from the re-search to the candidate documents. Extract text fragments from candidate documents and construct a context for generating an answer; The constructed context is input into a pre-defined large language model to generate an answer statement for the query statement.

[0007] The second objective of this invention is achieved through the following technical solution: A retrieval enhancement context building system based on keyword coverage awareness includes: The keyword extraction module is used to obtain the query statement input by the user and extract multiple query keywords from the query statement; The knowledge base retrieval module is used to perform retrieval operations on query keywords based on a preset knowledge base to obtain an initial candidate document set including at least one initial candidate document; The coverage calculation module is used to count the occurrence of query keywords in each initial candidate document and calculate the coverage value of each initial candidate document to the query keywords based on the occurrence. The coverage sorting module is used to sort the initial candidate document set according to the coverage value and select a number of candidate documents; The coverage judgment module is used to determine whether the overall coverage value of the selected candidate documents for the query keywords meets the preset coverage threshold. If it does not meet the threshold, it identifies the uncovered query keywords and re-executes the retrieval operation for the uncovered query keywords, adding the supplementary documents obtained from the re-retrieval to the candidate documents. The context building module is used to extract text fragments from candidate documents and build a context for generating answers; The context input module is used to input the constructed context into a preset large language model to generate an answer statement for the query statement.

[0008] This application also relates to a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps of the above-described method for constructing a search enhancement generation context based on keyword coverage awareness.

[0009] This application has the following advantages: In the embodiments of this application, a search enhancement generation context construction method based on keyword coverage awareness is provided. By quantitatively calculating keyword coverage and dynamically triggering a supplementary search mechanism, it can effectively solve the problems of information loss caused by the inability to finely cover query keywords and the inability to identify uncovered keywords in the prior art. It can realize dynamic monitoring of the coverage status of query keywords and automatic identification of uncovered keywords and triggering supplementary search, which has the effect of improving the completeness of search results and the accuracy of context construction. Attached Figure Description

[0010] Figure 1 This is a flowchart of an embodiment of a keyword coverage-aware retrieval enhancement generation context construction method according to this application; Figure 2 This is a schematic block diagram of a computer device provided in an embodiment of this application. Detailed Implementation

[0011] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0012] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0013] All terms used in this application, including technical or scientific terms, have the same meaning as understood by one of ordinary skill in the art to which this application pertains, unless otherwise specifically defined. It should also be understood that terms defined in general dictionaries, such as those in common dictionaries, should be interpreted as having a meaning consistent with their meaning in the context of the relevant art, and not as having an idealized or highly formalized meaning, unless expressly defined herein.

[0014] Techniques, methods, and equipment known to those skilled in the art may not be discussed in detail, but where appropriate, such techniques, methods, and equipment shall be considered part of the specification.

[0015] First, some nouns or terms that appear in the description of the embodiments of this application shall be interpreted as follows: A query statement is a natural language text entered by a user through a client to express their information needs or intent to ask a question, serving as the starting point for the entire information retrieval and generation process.

[0016] Query keywords refer to words or phrases extracted from a query statement that represent the user's core query intent.

[0017] A knowledge base is a collection of large amounts of structured or unstructured information that is stored and organized in advance. It can contain data in various forms such as text, images, and videos, and provides a data source for retrieval operations.

[0018] Initial candidate documents refer to documents retrieved from the knowledge base based on query keywords during the preliminary retrieval stage that may be related to the query statement.

[0019] The initial candidate document set refers to a collection of multiple initial candidate documents, representing the result of the preliminary retrieval operation.

[0020] Coverage value is a numerical value used to quantitatively evaluate the extent to which a document or a group of documents covers a specific set of query keywords, reflecting the comprehensiveness of the query keywords contained in the documents.

[0021] Coverage threshold refers to a preset numerical standard used to judge whether the keyword coverage meets the requirements. When the overall coverage value reaches or exceeds the coverage threshold, the keyword coverage can be considered to be sufficient.

[0022] Supplementary documents are documents obtained by re-searching for the missing keywords when the initial search results fail to fully cover all the query keywords, and are used to supplement contextual information.

[0023] Candidate documents refer to the collection of documents that are ultimately selected to build the context after initial retrieval, sorting, filtering, and possible supplementary retrieval.

[0024] A text fragment refers to a block of text with specific semantic or informational units extracted from a candidate document, serving as a direct component of the constructed context.

[0025] Context refers to the text content that provides background information and relevant knowledge to the large language model, aiming to guide the large language model to generate accurate, coherent and informative answers.

[0026] A large language model is a computational model built on deep learning technology that has the ability to understand and generate natural language. It can generate corresponding answers based on the input context and query statement.

[0027] The response statement refers to the natural language text generated and output by the large language model based on the input query statement and context, which is intended to answer the user's query.

[0028] In one embodiment, such as Figure 1 As shown, this application discloses a method for constructing a search enhancement context based on keyword coverage awareness, which specifically includes the following steps: S10: Obtain the query statement input by the user and extract multiple query keywords from the query statement; In one embodiment of this application, all non-stop words in the query statement can be used as query keywords, or core words in the statement can be identified through manual annotation. In another implementation, a term frequency-inverse document frequency (TNF) method can be used to calculate the weights of words in the query statement and select words with higher weights as query keywords.

[0029] S20: Perform a retrieval operation on the query keywords based on a preset knowledge base to obtain an initial candidate document set including at least one initial candidate document; In one embodiment of this application, the retrieval operation can employ a Boolean retrieval model, that is, combining query keywords using logical operators to search for documents containing these keywords in the knowledge base. Alternatively, a vector space model can be used, representing query keywords and documents in the knowledge base as vectors, and calculating the similarity between these vectors to obtain initial candidate documents.

[0030] S30: Calculate the occurrence of query keywords in each initial candidate document and calculate the coverage value of each initial candidate document for the query keywords based on the occurrence. In one embodiment of this application, the statistics of the occurrence of query keywords can be accomplished by counting the number of times each query keyword appears in a document. For example, the number of query keywords contained in each document can be counted and used as a coverage value.

[0031] S40: Sort the initial candidate document set according to the coverage value and select several candidate documents; In one embodiment of this application, the initial candidate document set can be sorted according to coverage values ​​from high to low, prioritizing documents containing more query keywords. For example, a fixed number can be set, selecting the N documents with the highest coverage values ​​as candidate documents.

[0032] S50: Determine whether the overall coverage value of the selected candidate documents for the query keywords meets the preset coverage threshold. If not, identify the uncovered query keywords and re-execute the retrieval operation for the uncovered query keywords, and add the supplementary documents obtained from the re-retrieval to the candidate documents. In one embodiment of this application, the judgment can be made through manual review, whereby an operator assesses whether the current candidate documents sufficiently cover all query keywords. If any keywords are found to be uncovered, the uncovered keywords can be entered, and a supplementary search can be performed.

[0033] Furthermore, in the embodiments of this application, the coverage threshold can be adjusted according to the specific needs of the application scenario. As one implementation, the coverage threshold can be set to a fixed value, such as 0.8 or 0.9, indicating that at least 80% or 90% of the query keyword weights are required to be covered by candidate documents. As another implementation, an adaptive threshold setting strategy can be adopted. The coverage threshold is dynamically adjusted based on the number of query keywords. For example, when the number of query keywords is small (e.g., 2-3), a higher coverage threshold (e.g., 0.9) is set to ensure that all key information points are covered; when the number of query keywords is large (e.g., more than 5), the coverage threshold is appropriately reduced (e.g., 0.7) to balance coverage completeness and retrieval efficiency. As yet another implementation, the coverage threshold can be set based on the complexity of the query statement. For complex queries containing multiple parallel information requirements, a higher coverage threshold is set; for simple factual queries, a lower coverage threshold can be set.

[0034] S60: Extract text fragments from candidate documents and construct a context for generating an answer; In one embodiment of this application, text fragment extraction can be achieved by treating the entire candidate document as a text fragment. For example, the entire content of all selected candidate documents can be concatenated to form a long text as context.

[0035] S70: Input the constructed context into the preset large language model to generate an answer statement for the query statement.

[0036] In one embodiment of this application, the context text and query statement can be directly submitted to the large language model. Upon receiving the context and query statement, the large language model will directly output an answer to the query statement based on its internal language understanding and generation capabilities.

[0037] For example, as a specific implementation, suppose user A enters the query "What is quantum computing? What are its applications?".

[0038] First, the system obtains the query statement and extracts the query keywords from it. For example, through lexical analysis, query keywords such as "quantum computing" and "application scenarios" can be extracted.

[0039] Based on this, the system performs a search operation on a preset knowledge base using query keywords. This preset knowledge base may contain a large number of documents related to computer science, physics, and other fields. In the initial search phase, the system may retrieve some documents about the basic concepts of "quantum computing" and some general documents about "computing technology," forming an initial set of candidate documents.

[0040] Based on this, the system statistically analyzes the occurrence of the query keywords "quantum computing" and "application scenarios" in each document in the initial candidate document set, and calculates the coverage value of each document. For example, document 1 may explain the principles of "quantum computing" in detail, but rarely mentions "application scenarios," so its coverage value may be low; while document 2 may briefly mention "quantum computing" and list some "application scenarios," so its coverage value may be high.

[0041] At this point, based on the coverage value, the system sorts the initial set of candidate documents and selects several candidate documents. For example, the system may prioritize document 1 and document 2.

[0042] Furthermore, the system determines whether the overall coverage of the selected documents 1 and 2 for the query keywords "quantum computing" and "application scenarios" meets the preset coverage threshold. Suppose that after evaluation, it is found that while documents 1 and 2 cover most of the information on "quantum computing," their coverage of "application scenarios" is insufficient, and the overall coverage value does not meet the preset threshold. In this case, the system will identify "application scenarios" as an uncovered query keyword. Subsequently, for the uncovered keywords, the system will re-execute the search operation, for example, by specifically searching the knowledge base for documents related to "application scenarios," thereby obtaining supplementary documents and adding them to the current candidate document set.

[0043] The system then extracts relevant text fragments from the final set of candidate documents and constructs a context for generating an answer, which will include detailed information about the definition, principles, and various application scenarios of "quantum computing".

[0044] Finally, the constructed context and user A's query are input into a pre-defined large language model. The large language model performs semantic understanding and reasoning on the query based on the input context, and finally generates a complete answer to user A's query. This answer not only explains "what quantum computing is", but also further details its "application scenarios".

[0045] In the example above where user A queries "What is quantum computing? What are its application scenarios?", existing retrieval methods based on semantic similarity may only return a large number of documents about the definition and principles of "quantum computing", while the coverage of "application scenarios" is insufficient. This is because traditional methods tend to match the overall semantics of the query statement rather than ensuring comprehensive coverage of each key information point.

[0046] In contrast, the solution in this embodiment first extracts multiple query keywords such as "quantum computing" and "application scenarios" from the query statement, and then performs a preliminary search based on the extracted query keywords. More importantly, the solution in this embodiment introduces statistics on the coverage of query keywords in the initial candidate documents and calculates the coverage value. For example, after the preliminary search, the system can identify that although there are documents explaining "quantum computing," the coverage of "application scenarios" is low, and this awareness of keyword coverage is lacking in existing technologies.

[0047] Furthermore, this embodiment dynamically adjusts the context construction process by determining whether the overall coverage value of the selected candidate documents for the query keywords meets a preset threshold. When it is found that the keyword "application scenario" is not fully covered, the system can proactively identify the uncovered keywords and re-execute the search operation to obtain supplementary documents. Through this feedback mechanism, the shortcomings of existing technologies in dynamically supplementing missing information can be effectively compensated for, and by adding supplementary documents to the candidate documents, it is ensured that the final constructed context can comprehensively include all key information points in the user's query.

[0048] Therefore, the solution in this embodiment can construct a more complete and accurate context than existing technologies. For example, in the example above, the final context not only includes the definition of "quantum computing" but also supplements it with detailed information about its "application scenarios." After inputting this comprehensive context into the large language model, the large language model can generate a complete and informative answer to the query, effectively solving the problem of incomplete or missing answers in existing technologies.

[0049] In summary, the solution in this embodiment improves the performance and answer quality of the retrieval enhancement generation system through keyword coverage awareness, dynamic supplementary retrieval, and context optimization.

[0050] The following will further explain a method for constructing a search enhancement generation context based on keyword coverage awareness in this exemplary embodiment.

[0051] In one embodiment, step S10, "obtaining the query statement input by the user and extracting multiple query keywords from the query statement," specifically includes: The query statement is segmented and stopped word filtered to obtain several candidate words; In the embodiments of this application, the query statement is segmented and stopped word filtered to obtain several candidate words. The aim is to divide a continuous text sequence into word units with independent semantics and remove words that frequently appear in the language but contribute little to the semantics of the text, thereby reducing noise and improving the efficiency and accuracy of subsequent processing. This processing can employ dictionary-based segmentation methods, such as maximum matching or inverse maximum matching, combined with a pre-defined stop word list for filtering. Alternatively, statistical model-based segmentation methods, such as Hidden Markov Models (HMMs) or Conditional Random Fields (CRFs), or deep learning-based segmentation models can be used, with stop word filtering rules applied on top of these.

[0052] Candidate words are semantically labeled, and each candidate word is divided into a first keyword and a second keyword according to its semantic role. The first keyword corresponds to the main semantic component in the query statement, and the second keyword corresponds to the modifying semantic component and the limiting semantic component in the query statement. In the embodiments of this application, candidate words are semantically labeled, and each candidate word is divided into a first keyword and a second keyword based on its semantic role. This aims to identify the arguments of predicates in a sentence and their semantic roles, such as agent, patient, time, and place. Through semantic role labeling, a deeper understanding of the underlying semantic relationships and functions of each word in a query statement can be achieved, thereby distinguishing the core subject information (first keyword) from the auxiliary modifying or limiting information (second keyword). This labeling can be implemented using a rule-based semantic role labeling system, which predefines grammatical patterns and semantic rules to identify the semantic roles of different words and performs keyword segmentation accordingly. Alternatively, a machine learning or deep learning-based semantic role labeling model can be used, such as fine-tuning pre-trained language models like Transformer or BERT, to automatically identify the semantic roles of candidate words and map them to the first keyword (subject semantic component) and the second keyword (modifying semantic component, limiting semantic component).

[0053] Based on the semantic role of each candidate word, keyword weights are assigned to each candidate word to obtain the query keywords.

[0054] In the embodiments of this application, keyword weights are assigned to each candidate word based on its semantic role to obtain query keywords, aiming to quantify the importance of keywords in the query statement. By combining semantic role information, weights can be allocated more reasonably, giving higher weights to keywords representing the main semantic components and relatively lower weights to keywords modifying or limiting semantic components, thereby more accurately reflecting the user's query intent. This weight setting can be achieved through preset weight allocation rules, for example, assigning higher fixed weight values ​​to all candidate words labeled as "main semantic components," while assigning lower fixed weight values ​​to candidate words "modifying semantic components" and "limiting semantic components." Alternatively, a learning-based weight allocation method can be used, training a model to learn the mapping relationship between different semantic roles and keyword weights. For example, a supervised learning model can be used, inputting semantic role features and outputting corresponding weight values.

[0055] Furthermore, in the embodiments of this application, after setting keyword weights for each candidate word based on its semantic role, it is necessary to normalize the weights of all query keywords so that the sum of the weights of all query keywords is 1.

[0056] Specifically, the normalization process includes the following steps: Calculate the sum of the weights of all query keywords; Divide the weight of each query keyword by the sum of the weights to obtain the normalized keyword weights.

[0057] For example, if the weight of the first keyword is set to 0.8 and the weight of the second keyword is set to 0.5, then after normalization, the weights of each keyword are adjusted to 0.8 / (total weight) and 0.5 / (total weight), respectively, thereby ensuring that the calculation of the overall coverage value has a unified reference standard.

[0058] For example, as a specific implementation, suppose the user inputs the query "Admissions Brochure of Peking University". First, the system will perform word segmentation and stop word filtering on this query. After word segmentation, the possible results are "Beijing", "university", "of", "admissions", and "brochure". After stop word filtering, the word "of" is removed, thus obtaining the candidate words: "Beijing", "university", "admissions", and "brochure".

[0059] Based on this, semantic role labeling is performed on the obtained candidate words. For example, through semantic analysis, "Peking University" is identified as the subject of the query. Therefore, "Beijing" and "university" in "Peking University" can be classified as the first keyword, corresponding to the subject semantic component. "Admissions Brochure" is identified as a specific information request for "Peking University" and can be classified as the second keyword, corresponding to the limiting semantic component.

[0060] Finally, keyword weights are assigned to each candidate word based on the semantic roles described above. For example, "Beijing" and "university" can be assigned higher weights, such as 0.9; while "admissions" and "brochure" can be assigned relatively lower weights, such as 0.6. Based on this, a set of weighted query keywords can be obtained, for example: {"Beijing": 0.9, "university": 0.9, "admissions": 0.6, "brochure": 0.6}. These weighted keywords will then be used for subsequent knowledge base retrieval, ensuring that the search results focus more on the core subject of "Peking University" while also taking into account the limiting information of "admissions brochure".

[0061] Through the above technical solution, this application can perform refined processing of keywords in query statements, not only identifying the keywords themselves but also further distinguishing their semantic roles within the query statement and assigning them different weights accordingly. This semantic role-based weighting mechanism enables the system to accurately capture the core intent and auxiliary information of the user's query, avoiding the semantic ambiguity and uneven weight distribution problems that may result from simple keyword extraction. Therefore, in subsequent knowledge base retrieval, the system can prioritize matching core keywords with higher weights, thereby improving the relevance of search results to the user's true intent and reducing interference from irrelevant documents.

[0062] In one embodiment, step S20, "performing a retrieval operation on the query keywords based on a preset knowledge base to obtain an initial candidate document set including at least one initial candidate document," specifically includes: Construct the search query expressions corresponding to each query keyword; In the embodiments of this application, retrieval query expressions corresponding to each query keyword are constructed respectively. The aim is to transform each keyword in the user query into a specific format that can be retrieved by the knowledge base. Its function is to customize retrieval instructions for each keyword to improve its recall and accuracy in the knowledge base. Specifically, based on the keyword type, semantic role, or preset retrieval strategy, it can be constructed as a Boolean query expression, such as using AND, OR, and NOT logical operators to combine keywords; or as a phrase query expression to precisely match keyword sequences; alternatively, it can be constructed as a fuzzy query expression, allowing for a certain degree of spelling errors or variations; or as a semantic query expression based on vector similarity. For example, for an entity keyword, a phrase query with exact matching can be constructed; for a concept keyword, a Boolean query including its synonyms or related words can be constructed.

[0063] By constructing the search query expression, a multi-path parallel search is performed in the preset knowledge base to obtain a preliminary search result set corresponding to each query keyword; In the embodiments of this application, multi-path parallel retrieval is performed in a preset knowledge base using constructed retrieval query expressions to obtain preliminary retrieval result sets corresponding to each query keyword. The aim is to efficiently execute retrieval tasks in the knowledge base using retrieval query expressions constructed for each query keyword. Its function is to accelerate retrieval speed through parallel processing and allow different retrieval paths or algorithms to be used for different keywords, thereby improving the breadth and depth of the retrieval. As one implementation, the system can start an independent retrieval thread or process for each query keyword. These threads or processes can run simultaneously on different indexes or different retrieval models in the knowledge base, such as the BM25 model based on inverted indexes or the ANN search model based on semantic vectors. Another implementation is that, for each keyword, query requests can be sent simultaneously to multiple heterogeneous knowledge sources or retrieval systems to obtain more comprehensive preliminary retrieval results.

[0064] The initial search result sets are deduplicated and merged to obtain a candidate document pool. The inverse ranking fusion score is then calculated based on the ranking position of each document in the candidate document pool in the initial search result set. In the embodiments of this application, the preliminary search result sets are deduplicated and merged to obtain a candidate document pool. A reciprocal ranking fusion score is calculated based on the ranking position of each document in the candidate document pool within the preliminary search result sets. This aims to integrate preliminary search results from different keywords and provide a unified ranking basis for subsequent screening. Its function is to eliminate duplicate documents, ensuring that each document is considered only once, and to generate a more stable and comprehensive document importance assessment result by fusing ranking information from different search results. Specifically, deduplication and merging can be achieved by aggregating and manipulating the document identifiers in all preliminary search result sets to form a candidate document pool without duplicate documents. Based on this, a reciprocal ranking fusion algorithm can be used to calculate the fusion score for each document. This algorithm assigns a reciprocal weight, such as 1 / (rank+C), to each document's ranking in each preliminary search result set, where C is a constant. Then, the reciprocal weights of a document in all its occurrence sets are summed to obtain its final reciprocal ranking fusion score. This method effectively combines ranking information from different search sources, achieving robust fusion results even if the scoring criteria of different search sources are inconsistent.

[0065] Several documents were selected based on the inverse ranking fusion score to form an initial candidate document set.

[0066] In the embodiments of this application, a number of documents are selected based on the inverse ranking fusion score as an initial candidate document set. The aim is to select the most relevant and high-quality documents from the fused and sorted candidate document pool as the initial candidate document set for subsequent processing. This reduces the number of documents, ensuring that subsequent keyword coverage analysis is performed on a high-quality subset of documents, thereby improving efficiency and accuracy. One implementation method is to set a fixed document quantity threshold N and select the N documents with the highest inverse ranking fusion score. Another implementation method is to set a threshold for the inverse ranking fusion score and select all documents with scores higher than that threshold. Alternatively, a combination of both methods can be used; for example, first selecting the N documents with the highest scores, and if the total score of these documents does not reach a certain preset standard, then appropriately relaxing the conditions or performing supplementary screening.

[0067] For example, as a specific implementation, suppose the user enters the query "How to use Python for data visualization", the system extracts the query keywords "Python" and "data visualization".

[0068] First, the system will construct search query expressions for these two query keywords respectively. For example, for "Python", an exact phrase query "Python" can be constructed; for "data visualization", a Boolean query containing synonyms "data visualization OR data chart OR data graph" can be constructed.

[0069] Based on this, the system will perform multi-path parallel searches in a pre-defined knowledge base using the constructed search query expressions. For example, for a query about "Python," the system might perform a BM25-based search in a programming tutorial document library, obtaining a preliminary search result set 1, which includes document A ranked 1st, document B ranked 2nd, and document C ranked 3rd. Simultaneously, for a query about "data visualization," the system might perform a semantic vector-based search in a data science paper library, obtaining a preliminary search result set 2, which includes document D ranked 1st, document B ranked 2nd, and document E ranked 3rd. These two search processes can be performed in parallel.

[0070] Furthermore, the system performs deduplication and merging on the preliminary search result set 1 and the preliminary search result set 2 to obtain a candidate document pool, which may include documents A, B, C, D, and E.

[0071] Based on this, the system calculates the bottom-ranked merge score for each document. For example, assuming the bottom-ranked merge constant C is 60: Document A's score is 1 / (1+60) = 1 / 61; Document B's score is 1 / (2+60) + 1 / (2+60) = 2 / 62; Document C's score is 1 / (3+60) = 1 / 63; Document D's score is 1 / (1+60) = 1 / 61; and Document E's score is 1 / (3+60) = 1 / 63. Then, the system sorts the documents in descending order based on the calculated bottom-ranked merge scores.

[0072] Finally, the system selects several documents as an initial candidate document set based on the above scores. For example, if it is set to select 3 documents, it may select the documents with the highest scores, A, B and D, as the initial candidate document set.

[0073] Through the above technical solutions, this application can improve the quality of the initial candidate document set obtained from the knowledge base. Specifically, by constructing a retrieval query expression for each query keyword and performing multi-path parallel retrieval, the system can more comprehensively capture the semantic information of the query keywords, avoiding document omissions that may be caused by a single retrieval strategy, thereby improving the breadth of document recall. Simultaneously, the reciprocal ranking fusion scoring mechanism can effectively integrate ranking information from different retrieval paths, accurately sorting documents so that documents highly relevant to multiple query keywords can be prioritized, thus ensuring the high relevance of the initial candidate document set. This method overcomes the problems of incomplete recall or insufficient relevance that may occur when traditional retrieval methods handle multi-keyword queries, providing a higher-quality document foundation for subsequent keyword coverage calculations, thereby improving the accuracy and completeness of the final generated context and enhancing the quality of the response statements generated by the large language model.

[0074] In one embodiment, the step "performing multi-path parallel retrieval in a preset knowledge base using the constructed retrieval query expression to obtain a preliminary retrieval result set corresponding to each query keyword" specifically includes: Identify the semantic roles corresponding to each query keyword, and determine the retrieval range parameters corresponding to the query keywords based on the semantic roles; In the embodiments of this application, identifying the semantic roles corresponding to each query keyword and determining the retrieval scope parameters corresponding to the query keywords based on the semantic roles refers to analyzing the grammatical function and semantic meaning of each query keyword extracted from the query statement, such as its role as a subject, object, modifier, or qualifier. Based on these semantic roles, the system can set a specific retrieval strategy or scope for each keyword. For example, subject keywords may require a broader retrieval scope to ensure comprehensiveness, while modifier or qualifier keywords may require a more precise retrieval scope to improve relevance. This process can be implemented using natural language processing techniques, such as using dependency parsing or semantic role annotation tools to automatically identify the semantic roles of keywords. Furthermore, a set of rules or mapping tables can be predefined to associate different semantic roles with different retrieval scope parameters, such as search fields, document types, relevance thresholds, and the maximum number of documents returned.

[0075] The search query expressions for each query keyword are matched and calculated with the stored documents in the knowledge base to obtain the matching score between each query keyword and each stored document; In the embodiments of this application, the retrieval query expressions of each query keyword are matched with stored documents in the knowledge base to obtain a matching score between each query keyword and each stored document. This aims to quantify the relevance between each query keyword and each stored document in the knowledge base; a higher matching score indicates a stronger relevance between the keyword and the document. The matching calculation can be implemented using various methods. For example, it can be based on a term frequency-inverse document frequency model, evaluating the matching degree by calculating the frequency of keyword occurrences in documents and their rarity in the entire knowledge base. Alternatively, a vector space model or deep learning model, such as BM25 or BERT, can be used to represent keywords and documents as vectors, and the matching score is obtained by calculating the similarity between vectors.

[0076] The stored documents are sorted in descending order according to the matching score of each query keyword, and a number of stored documents are selected according to the retrieval range parameters corresponding to the query keywords. In the embodiments of this application, sorting the stored documents in descending order according to the matching scores of each query keyword, and selecting a number of stored documents based on the retrieval scope parameters corresponding to the query keywords, means that after calculating the matching score between each query keyword and all stored documents, for each keyword, the documents in the knowledge base are sorted from high to low according to their matching scores. Subsequently, a certain number of documents are selected from the sorted document list according to the retrieval scope parameters previously determined for the query keywords. For example, for keywords identified as "subject," the retrieval scope parameters may allow for the selection of more documents to ensure comprehensive coverage; while for keywords identified as "limited," the retrieval scope parameters may require the selection of documents with extremely high matching scores and a small number of documents to ensure accuracy.

[0077] By aggregating several stored documents selected based on each query keyword, preliminary search result sets corresponding to each query keyword are obtained.

[0078] In the embodiments of this application, summarizing several stored documents selected based on each query keyword to obtain a preliminary search result set corresponding to each query keyword means combining the documents independently selected for each query keyword to form a dedicated preliminary search result set for each query keyword. This process embodies multi-path parallel retrieval, meaning that each keyword is independently searched and documents are selected, ultimately forming its own search result set.

[0079] For example, as a specific implementation, suppose the query entered by the user is "What are the admission policies of Peking University in 2023?".

[0080] First, the system extracts query keywords from the query statement, such as "2023", "Peking University", and "admissions policy", and identifies the semantic role of each query keyword to determine the retrieval scope parameters: For "Peking University", its semantic role is the subject, and its search scope parameter can be determined as "broad search", that is, matching multiple fields such as document title, abstract and body, and allowing a large number of documents to be selected.

[0081] For "enrollment policy", its semantic role is object, and its search scope parameter can be determined as "exact search", that is, mainly matching in the document body or specific policy-related fields, and prioritizing documents with extremely high matching degree.

[0082] For "2023", its semantic role is time-limited, so its search scope parameter can be determined as "time-limited search", that is, only matching fields that contain year information in the publication date or content of the document, and strictly filtering.

[0083] Then, the search query expression of each query keyword is subjected to matching calculation with all stored documents in the knowledge base respectively, so as to obtain a matching score between each keyword and each document.

[0084] On this basis, for each keyword, all stored documents are sorted in descending order according to their matching scores with the keyword, and a number of documents are selected from the sorted documents according to the previously determined search scope parameter. For example, for "Peking University", the top N1 documents with higher matching scores are selected according to the "broad search" parameter; for "admission policy", the top N2 documents with higher matching scores are selected according to the "exact search" parameter, where N2<N1; for "year 2023", N3 documents with extremely high matching scores and containing the information of "year 2023" are selected according to the "time-limited search" parameter.

[0085] Finally, the N1 documents selected for "Peking University" are aggregated to form a preliminary search result set for "Peking University", the N2 documents selected for "admission policy" are aggregated to form a preliminary search result set for "admission policy", and the N3 documents selected for "year 2023" are aggregated to form a preliminary search result set for "year 2023".

[0086] Through the above technical solutions, since in the multi-channel parallel search stage, the system can dynamically adjust the search strategy and scope according to the semantic role of the query keyword, so that each keyword can be searched in the manner most suitable for its semantic characteristics, thereby improving the quality and relevance of each preliminary search result set, effectively avoiding search deviations caused by semantic differences of keywords, and providing more accurate and high-quality input for subsequent document deduplication and merging, reciprocal rank fusion score calculation, and screening of the initial candidate document set. On this basis, it is helpful to construct a more accurate and comprehensive context, thereby improving the quality of the response sentences generated by the large language model and user satisfaction.

[0087] In one embodiment, the "counting the occurrence of the query keyword in each initial candidate document, and calculating the coverage value of each initial candidate document to the query keyword based on the occurrence situation" described in step S30 specifically includes: Detecting whether each query keyword appears in each initial candidate document, and generating a keyword coverage vector based on the detection result, wherein each position in the keyword coverage vector corresponds to a query keyword, if the query keyword appears in the initial candidate document, the corresponding position is assigned a value of 1, otherwise the value is assigned 0; In the embodiments of this application, detecting whether each query keyword appears in each initial candidate document aims to determine the inclusion relationship between each query keyword extracted from the query statement and each initial candidate document. This can be achieved by: using an exact string matching algorithm, such as the Boyer-Moore algorithm or the Rabin-Karp algorithm, to directly search for keywords in the document text; or by using inverted indexing technology to pre-establish a mapping between keywords and documents in the knowledge base, thereby quickly determining whether a keyword exists in a specific document. A keyword coverage vector is generated to represent the occurrence of query keywords in documents in a structured numerical form. The keyword coverage vector is a binary vector with a length equal to the number of query keywords. The generation of this keyword coverage vector can be based on the detection results, by traversing all query keywords and filling each position of the vector according to their occurrence in the current initial candidate document. Furthermore, each position in the keyword coverage vector corresponds to a query keyword, clarifying the structure of the keyword coverage vector and ensuring that each dimension of the vector is associated with a specific query keyword. For example, if there are N query keywords, the vector will have N positions. The first position corresponds to the first query keyword, the second position to the second query keyword, and so on. This one-to-one correspondence ensures the semantic clarity and interpretability of the keyword coverage vector. If a query keyword appears in the initial candidate documents, the corresponding position is assigned a value of 1; otherwise, it is assigned a value of 0. This defines the assignment rule for each position in the keyword coverage vector. When a query keyword is detected to exist in an initial candidate document, its corresponding position in the vector is marked as 1, indicating that the keyword coverage vector is covered by the document; conversely, if the keyword does not appear, it is marked as 0, indicating that the keyword coverage vector is not covered.

[0088] Extract the query keywords corresponding to the positions assigned a value of 1 in the keyword coverage vector to obtain the initial candidate document coverage keyword set; In the embodiments of this application, the query keywords corresponding to the positions marked with a value of 1 in the keyword coverage vector are extracted. This aims to identify the query keywords actually covered by the initial candidate documents from the keyword coverage vector. The implementation may include: traversing the keyword coverage vector, and for all positions marked with a value of 1, retrieving the corresponding query keywords based on a preset mapping relationship between keywords and vector positions. Obtaining the coverage keyword set of the initial candidate documents refers to aggregating all the extracted query keywords covered by the documents into a set containing all query-related keyword information that the current initial candidate documents can provide.

[0089] Obtain the keyword weights corresponding to each query keyword in the covered keyword set, and sum the keyword weights to obtain the coverage values ​​of the initial candidate documents for the query keywords.

[0090] In the embodiments of this application, the keyword weights corresponding to each query keyword in the covered keyword set are obtained, aiming to assign an importance level to the identified covered keywords in the query statement. Keyword weights can be determined in advance in various ways, such as: based on the semantic role of the keyword in the query statement (e.g., subject words usually have higher weights, modifiers have lower weights); based on the keyword's term frequency-inverse document frequency value; or obtained through training a machine learning model based on historical query data. Summing the keyword weights involves accumulating the weights of all covered keywords, aiming to aggregate the importance of individual keywords into a comprehensive metric. The coverage value of the initial candidate documents for the query keywords is the final result of the above summation operation, which quantifies the overall coverage of the initial candidate documents for the query keywords. This coverage value comprehensively considers the appearance of keywords and their relative importance in the query, providing a quantitative basis for subsequent document ranking and filtering.

[0091] For example, as a specific implementation method, suppose the user enters the query "How is the camera function of the latest iPhone 15 Pro Max?". After processing, the following query keywords and their corresponding weights are extracted: Query keyword 1: "iPhone 15 Pro Max", weight 3; Query keyword 2: "Camera function", weight 2; Query keyword 3: "Latest model", weight 1. Now there is an initial candidate document A, whose content mainly discusses the various parameters and user experience of the "camera function" of the "iPhone 15 Pro Max", but does not directly mention the modifier "latest model".

[0092] First, the system checks whether these three query keywords appear in the initial candidate document A: it detects that "iPhone 15 Pro Max" appears in document A; it detects that "camera function" appears in document A; and it detects that "latest model" does not appear in document A. Based on the above detection results, a keyword coverage vector is generated.

[0093] Since the query keywords are in the order of "iPhone 15 Pro Max", "camera function", and "latest model", the keyword coverage vector is [1,1,0]. The first 1 indicates that "iPhone 15 Pro Max" is covered, the second 1 indicates that "camera function" is covered, and the third 0 indicates that "latest model" is not covered.

[0094] Based on this, the system will extract the query keywords corresponding to the positions assigned a value of 1 in the keyword coverage vector, thereby obtaining the initial candidate document A's coverage keyword set, namely {"iPhone 15 Pro Max", "camera function"}.

[0095] Then, the system obtains the keyword weights corresponding to each query keyword in the covered keyword set: the weight of "iPhone15 Pro Max" is 3; the weight of "camera function" is 2.

[0096] Finally, the keyword weights obtained above are summed: 3 + 2 = 5. Based on this, the coverage value of the initial candidate document A for the query keywords is 5, and this coverage value will be used to sort and filter the initial candidate document set in order to prioritize documents that more comprehensively cover the core semantics of the user's query.

[0097] Through the above technical solution, this application can more accurately evaluate initial candidate documents. Specifically, by introducing a keyword coverage vector, the document's coverage of the query keywords is structurally represented, and a weighted sum is performed based on keyword weights. This allows the calculated coverage value to comprehensively reflect the document's completeness and importance in matching the query intent, effectively solving the problems of crude evaluation and inability to distinguish differences in keyword importance caused by simply counting the number of keywords or judging only the presence or absence of keywords in traditional methods. Therefore, in the subsequent document sorting and filtering process, documents that not only contain the query keywords but also have more sufficient coverage of high-weight core keywords can be prioritized, thereby improving the quality of the constructed context.

[0098] In one embodiment, the step of "determining whether the overall coverage value of the selected candidate documents for the query keywords meets a preset coverage threshold; if not, identifying the uncovered query keywords, re-performing the retrieval operation for the uncovered query keywords, and adding the supplementary documents obtained from the re-retrieval to the candidate documents" specifically includes: Perform a bitwise OR operation on the keyword coverage vectors of the selected candidate documents to obtain the overall coverage vector; Extract the query keywords corresponding to the positions assigned a value of 1 in the overall coverage vector, obtain the keyword weights corresponding to each extracted query keyword, and sum the keyword weights to obtain the overall coverage weights; Calculate the ratio of the overall coverage weight to the sum of the weights of all query keywords to obtain the overall coverage value; Determine whether the overall coverage value is greater than or equal to the preset coverage threshold. If so, end the coverage determination process. If the overall coverage value is less than the preset coverage threshold, the query keywords corresponding to the positions in the overall coverage vector that are assigned a value of 0 are extracted as the uncovered query keywords. Based on each uncovered query keyword, perform a single keyword search operation in the knowledge base to obtain a supplementary document set corresponding to each uncovered query keyword; Select several documents from each supplementary document set as supplementary documents and add them to the candidate documents.

[0099] In the embodiments of this application, a keyword coverage vector is a binary vector used to represent the occurrence of query keywords in a document, with each position corresponding to a specific query keyword. If a query keyword appears in a document, the value at the corresponding position in the keyword coverage vector is set to 1; otherwise, it is set to 0. This vectorized representation can clearly and quantitatively reflect the document's coverage of the query keyword. Bitwise OR is a logical operation that, when applied to multiple keyword coverage vectors, can merge these vectors. Specifically, if any input vector has a value of 1 at a certain position, the output vector will also have a value of 1 at the same position. Through this operation, keyword coverage information from multiple documents can be effectively aggregated to determine the overall coverage of a document set. The overall coverage vector is the result of a bitwise OR operation on the keyword coverage vectors of multiple candidate documents, reflecting which query keywords are covered by at least one document in the selected candidate document set. The overall coverage weight is obtained by summing the weights of the query keywords corresponding to the positions in the overall coverage vector that are assigned a value of 1, quantifying the overall coverage of the query keyword by the current candidate document set. The coverage threshold is a pre-set value used to measure whether the overall coverage meets the expected requirements. This threshold can be configured according to the requirements for answer quality and comprehensiveness in the actual application scenario; for example, it can be set to 0.8 or 0.9 to ensure that most key information is covered. Uncovered query keywords refer to query keywords whose corresponding position in the overall coverage vector is 0. These keywords indicate that the currently selected candidate document set fails to provide relevant information. Single-keyword retrieval is a targeted search performed on a single query keyword, aiming to find documents highly relevant to that specific keyword from the knowledge base. The supplementary document set is the set of documents obtained after performing a single-keyword retrieval operation on uncovered query keywords, designed to compensate for the insufficient coverage of the existing candidate document set. Supplementary documents are documents selected from the supplementary document set and added to the existing candidate document set to enhance overall keyword coverage.

[0100] For example, as a specific implementation method, suppose the user query is "latest mobile phone camera function review". After keyword extraction, the query keywords and their weights are obtained, such as: "latest mobile phone" (weight 0.4), "camera function" (weight 0.3), "review" (weight 0.3).

[0101] At this point, the system retrieves and selects several initial candidate documents based on these keywords, such as document A, document B, and document C. The keyword coverage vector of document A may be [1,1,0] (covering "latest mobile phone" and "camera function"), the keyword coverage vector of document B may be [1,0,1] (covering "latest mobile phone" and "review"), and the keyword coverage vector of document C may be [1,0,0] (covering only "latest mobile phone").

[0102] At this point, a bitwise OR operation is performed on the keyword coverage vectors of documents A, B, and C to obtain the overall coverage vector [1,1,1], indicating that all query keywords have been covered. The overall coverage weight is calculated as 0.4 + 0.3 + 0.3 = 1.0. If the preset coverage threshold is 0.9, since 1.0 is greater than 0.9, the coverage of the current candidate document set can be considered to meet the requirements, and the process ends.

[0103] In another scenario, suppose the bitwise OR operation of documents A, B, and C results in an overall coverage vector [1,1,0], meaning the keyword "evaluation" is not covered. In this case, the overall coverage weight is 0.4 + 0.3 = 0.7. If the preset coverage threshold is still 0.9, since 0.7 is less than 0.9, the system will identify "evaluation" as an uncovered query keyword.

[0104] Subsequently, the system performs a single-keyword search in the knowledge base based on the keyword "evaluation," retrieving a supplementary document set, such as document D. Document D might be an article specifically evaluating the mobile phone's camera function. The system selects document D from the supplementary document set as a supplementary document and adds it to the current candidate document set. In this way, the final candidate document set will include documents A, B, C, and D, ensuring comprehensive coverage of the three query keywords: "latest mobile phone," "camera function," and "evaluation," providing a more complete context for the large language model to generate the answer.

[0105] Through the above technical solution, this application can effectively solve the problem of insufficient overall keyword coverage caused by filtering based solely on the coverage of a single document. Specifically, by introducing an overall coverage judgment mechanism and specifically supplementing the search for uncovered keywords, it can ensure that the final constructed context can comprehensively and evenly cover all key information in the user's query. This improves the accuracy, completeness, and relevance of the responses generated by the search enhancement model, avoids answer bias caused by missing information, and thus provides users with higher quality query results.

[0106] Furthermore, the step of selecting several documents from each supplementary document set as supplementary documents and adding them to the candidate documents may specifically include: Calculate the relevance score between each document in each supplementary document set and the corresponding uncovered query keyword; The documents in each supplementary document set are sorted according to their relevance scores; Select the K documents with the highest relevance scores from each supplementary document set as supplementary documents, where K is a preset threshold for the number of supplementary documents; Alternatively, supplementary documents can be selected sequentially from highest to lowest relevance score until the total length of the selected supplementary documents reaches a preset length threshold.

[0107] In the embodiments of this application, in order to control the quantity and quality of supplementary documents and avoid introducing too much redundant information or exceeding the context length limit of the large language model, it is necessary to screen the supplementary documents.

[0108] One approach is to use information retrieval algorithms such as TF-IDF and BM25 to calculate the relevance score between each supplementary document and the corresponding uncovered query keyword, and then sort the documents from highest to lowest score, selecting the top K documents as supplementary documents. For example, K=3 can be set, meaning that for each uncovered query keyword, a maximum of 3 supplementary documents can be added.

[0109] As another implementation method, selection can be based on length constraints. For example, the total length of supplementary documents can be set to no more than 2000 characters, and supplementary documents can be added in descending order of relevance score. Addition can be stopped when the cumulative length approaches or reaches the threshold.

[0110] As another implementation method, it can be combined with coverage gain for selection. Calculate the increase in overall coverage value after adding a supplementary document, and prioritize documents with larger coverage gains to maximize coverage with the fewest documents.

[0111] In one embodiment, step S70, "inputting the constructed context into a preset large language model to generate an answer statement for the query statement," specifically includes: The query statement is concatenated with the constructed context to generate input prompt text that includes the query statement and context content; In the embodiments of this application, the query statement is concatenated with the constructed context to generate input prompt text containing the query statement and context content. This aims to integrate the user's original query intent with relevant information retrieved from the knowledge base to form a unified input, enabling the large language model to simultaneously understand the user's needs and available background knowledge. The concatenation operation can be implemented through string concatenation, for example, using the query statement as a prefix and the context content as the body, separated by a specific delimiter (such as a newline character or a specific marker). Alternatively, a structured prompt template can be used, filling the query statement and context content into predefined placeholders, such as "Please answer the question based on the following information: [Context content]\nQuestion: [Query statement]".

[0112] Input the input prompt text into the preset large language model, which then performs semantic understanding and reasoning on the query statement based on the context. In the embodiments of this application, the input prompt text is fed into a preset large language model. The large language model performs semantic understanding and reasoning on the query statement based on the context. The large language model is the core generation component; it receives the integrated information and utilizes its language understanding and generation capabilities to extract relevant information from the context and perform logical reasoning in conjunction with the query statement to generate a natural and accurate answer. The large language model can be a pre-trained model based on the Transformer architecture, such as the GPT series, BERT series, or LLaMA series. These models are trained on massive amounts of data and possess powerful text understanding and generation capabilities. Alternatively, it can be a large language model fine-tuned for a specific domain to improve the professionalism and accuracy of its answers in that domain.

[0113] Obtain the text content output by the large language model as the answer to the query statement; In the embodiments of this application, the text content output by the large language model is obtained as the answer to the query statement. This is the direct product of the large language model after completing its generation task, and serves as the system's final response to the user's query. This text content can directly receive the text string returned by the large language model API. Alternatively, the original text output by the large language model can be post-processed, such as removing redundant information or formatting the output, to improve the quality and readability of the answer statement.

[0114] The system checks whether there is any reference content in the response statement that corresponds to a text fragment in a candidate document. If so, it extracts the candidate document identifier corresponding to the reference content and appends the candidate document identifier as the reference source to the response statement.

[0115] In embodiments of this application, the system detects whether there is any cited content in the answer statement that corresponds to a text fragment in a candidate document. If so, it extracts the candidate document identifier corresponding to the cited content and appends the candidate document identifier as the source of the citation to the answer statement. This step aims to improve the credibility and transparency of the generated answer by identifying which original documents' information is directly or indirectly cited in the answer and providing the identifiers of these documents, enabling users to trace and verify the source of information. This detection can be achieved by comparing fragments of the answer statement with text fragments in the candidate documents using a text similarity matching algorithm, thereby identifying the cited content. Alternatively, when generating the answer, the large language model's internal mechanism can mark the context fragments on which its generated content is based and return the corresponding document identifier.

[0116] For example, as a specific implementation, suppose a user query is "What are the products of photosynthesis?", and using the method described above, the system retrieves a candidate document D1 from the knowledge base, which contains the text fragment "The products of photosynthesis are glucose and oxygen."

[0117] First, the system will concatenate the query statement with the text fragment of the candidate document D1 to form the input prompt text, such as: "Please answer the question based on the following information: [D1 content: The products of photosynthesis are glucose and oxygen.] Question: What are the products of photosynthesis?"

[0118] The input prompt text is then fed into a pre-defined large language model, which generates the response statement "The products of photosynthesis are glucose and oxygen" based on the provided context.

[0119] Furthermore, the system will detect the answer statement and, through a text similarity algorithm, such as calculating the cosine similarity between the sentence embedding of the answer statement and the sentence embeddings in candidate document D1, find that the answer statement is highly similar to the text fragments in candidate document D1. Therefore, the system will identify that the content in the answer statement originates from candidate document D1 and extract D1 as the candidate document identifier.

[0120] Ultimately, the answer can be presented as: "The products of photosynthesis are glucose and oxygen. [Source: D1]", thus clearly indicating the source of the information.

[0121] Through the above technical solution, this application effectively solves the problem of opaque information sources after the generated answer statement. Specifically, by adding source citations to the answer statement, users can clearly understand the basis of the answer content, thereby increasing the credibility and trust of the generated answer. In addition, this traceability also facilitates users to verify the accuracy of the answer and, when necessary, to delve into the original document, thus enhancing the practicality and reliability of the retrieval and generation system.

[0122] In one embodiment, the step of "detecting whether there is any cited content in the answer statement that corresponds to a text fragment in a candidate document, and if so, extracting the candidate document identifier corresponding to the cited content and appending the candidate document identifier as the citation source to the answer statement" specifically includes: The response statement is segmented to obtain several response fragments, and the text similarity between each response fragment and the text fragments in each candidate document is calculated. In the embodiments of this application, the response statement is segmented to obtain several response fragments, and the text similarity between each response fragment and the text fragments in each candidate document is calculated. This aims to refine the response statement generated by the large language model into smaller, independently analyzable units, facilitating subsequent comparison with the text fragments in the candidate documents. This segmentation operation can employ sentence splitting based on punctuation marks or specific conjunctions, dividing the response statement into multiple independent sentences or phrases as response fragments. Alternatively, a sliding window approach can be used to extract overlapping or non-overlapping text segments from the response statement with fixed or dynamic lengths. The calculation of text similarity can be implemented using traditional text similarity algorithms such as the bag-of-words model or TF-IDF combined with cosine similarity. Alternatively, deep learning-based semantic similarity models, such as word vectors or sentence vectors generated by pre-trained language models like BERT and RoBERTa, can be used to measure the semantic closeness of the text by calculating the cosine similarity between the vectors.

[0123] Determine if there are candidate document text fragments with a text similarity greater than a preset similarity threshold. If so, determine that the corresponding answer fragment is the cited content and record the corresponding candidate document identifier. In the embodiments of this application, it is determined whether there are candidate document text fragments with a text similarity greater than a preset similarity threshold. If so, the corresponding answer fragment is identified as cited content, and the corresponding candidate document identifier is recorded. This identifier is used to identify the association between the answer fragment and the candidate document text fragment, thereby determining which answer fragments directly or indirectly cite the content of the candidate document. The preset similarity threshold is a key parameter for determining citation relationships. This determination can be achieved by traversing all answer fragments and text fragments in all candidate documents, calculating the similarity between each pair, and marking answer fragments with similarity exceeding the preset threshold as cited content. Simultaneously, the unique identifier of the candidate document with the highest matching degree is recorded. Alternatively, inverted indexes or approximate nearest neighbor search techniques can be used to quickly find candidate document text fragments with high similarity to the answer fragment, thereby improving matching efficiency.

[0124] Summarize the candidate document identifiers corresponding to each cited content to generate a list of cited sources; In the embodiments of this application, candidate document identifiers corresponding to each cited content are aggregated to generate a list of cited sources. The aim is to integrate the candidate document identifiers corresponding to all answer fragments identified as cited content into a unified set of cited sources. This aggregation operation can deduplicate all recorded candidate document identifiers and then arrange them sequentially or by document importance to form a list. Alternatively, the identifiers can be aggregated along with their corresponding document titles or summary information to provide users with a more intuitive understanding of the cited sources.

[0125] Append a list of cited sources to your answer, or insert a candidate document identifier as a citation at the corresponding location in each cited content.

[0126] In the embodiments of this application, two methods are provided to present the sources of citation: appending a list of cited sources to the answer statement or inserting candidate document identifiers as citation marks at the corresponding positions of each cited content. This aims to improve the credibility and traceability of the answer statement and enable users to easily access the original information. One method is to add a list of "Reference Sources: [Document Identifier 1], [Document Identifier 2]..." to the end of the answer statement. Alternatively, in the answer statement, after each answer segment identified as cited content, the corresponding candidate document identifier can be inserted in parentheses or as a superscript, for example, "...This is cited content [1]...".

[0127] For example, as a specific implementation, suppose a user queries "What is quantum entanglement?", and the large language model generates the answer: "Quantum entanglement is a peculiar physical phenomenon in which, when two or more particles interact, even if they are far apart, a change in the state of one particle will instantly change the state of the other particle accordingly. Einstein called it 'spooky action at a distance'."

[0128] First, the system will segment the answer statement. For example, based on a period, it can be divided into two answer segments: "Quantum entanglement is a peculiar physical phenomenon in which when two or more particles interact, even if they are far apart, a change in the state of one particle will instantly change the state of the other particle accordingly." and "Einstein once called it 'spooky action at a distance'."

[0129] Based on this, the system calculates the semantic similarity between these two answer fragments and text fragments from all candidate documents in the knowledge base (e.g., from Wikipedia, physics textbooks, etc.). For example, the Sentence-BERT model can be used to encode the answer fragments and candidate document fragments into vectors, and then cosine similarity can be calculated. Suppose the first answer fragment has a similarity of 0.85 with a text fragment in candidate document A (e.g., a popular science article about quantum mechanics), exceeding the preset similarity threshold of 0.7, then the first answer fragment is identified as cited content, and the identifier of candidate document A is recorded, such as "DocA". Suppose the second answer fragment has a similarity of 0.90 with a text fragment in candidate document B (e.g., a biography of Einstein or a book on the history of physics), also exceeding the similarity threshold, then the second answer fragment is identified as cited content, and the identifier of candidate document B is recorded, such as "DocB". Subsequently, the system summarizes these identifiers to generate a list of cited sources, such as "[DocA,DocB]".

[0130] Finally, you can append the list of citations to the end of your answer, forming: "Quantum entanglement is a peculiar physical phenomenon in which, when two or more particles interact, even if they are far apart, a change in the state of one particle will instantly change the state of the other particle. Einstein called it 'spooky action at a distance.' References: [DocA, DocB]." Alternatively, you can insert citations within your answer, for example: "Quantum entanglement is a peculiar physical phenomenon in which, when two or more particles interact, even if they are far apart, a change in the state of one particle will instantly change the state of the other particle. [DocA] Einstein called it 'spooky action at a distance.' [DocB]."

[0131] Through the above technical solution, this application can identify the cited content in the answer statements generated by the large language model and establish a clear association between it and the original candidate documents. This allows users to not only obtain fluent answers generated by the large language model but also clearly trace the original source of key information in the answer, thereby improving the credibility, transparency, and reliability of the answer. Simultaneously, by presenting citation sources in a flexible manner, such as supplementary lists or inline annotations, it can improve users' understanding of the source of the answer content and their browsing experience, effectively avoiding user doubts caused by unclear information sources and providing users with a way to further understand relevant knowledge.

[0132] It should be understood that the sequence number of each step in the above embodiments does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.

[0133] In one embodiment, a keyword-coverage-aware retrieval enhancement context generation system is provided, which corresponds one-to-one with the keyword-coverage-aware retrieval enhancement context generation method described in the previous embodiment. The keyword-coverage-aware retrieval enhancement context generation system includes: The keyword extraction module is used to obtain the query statement input by the user and extract multiple query keywords from the query statement; The knowledge base retrieval module is used to perform retrieval operations on query keywords based on a preset knowledge base to obtain an initial candidate document set including at least one initial candidate document; The coverage calculation module is used to count the occurrence of query keywords in each initial candidate document and calculate the coverage value of each initial candidate document to the query keywords based on the occurrence. The coverage sorting module is used to sort the initial candidate document set according to the coverage value and select a number of candidate documents; The coverage judgment module is used to determine whether the overall coverage value of the selected candidate documents for the query keywords meets the preset coverage threshold. If it does not meet the threshold, it identifies the uncovered query keywords and re-executes the retrieval operation for the uncovered query keywords, adding the supplementary documents obtained from the re-retrieval to the candidate documents. The context building module is used to extract text fragments from candidate documents and build a context for generating answers; The context input module is used to input the constructed context into a preset large language model to generate an answer statement for the query statement.

[0134] For specific limitations regarding the keyword coverage-aware retrieval enhancement context generation system, please refer to the limitations of the keyword coverage-aware retrieval enhancement context generation method described above, which will not be repeated here. Each module in the aforementioned keyword coverage-aware retrieval enhancement context generation system can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device, or stored in the memory of a computer device as software, so that the processor can call and execute the corresponding operations of each module.

[0135] In one embodiment, a computer device is provided, which may be a server, and its internal structure diagram may be as follows. Figure 2 As shown, the computer device includes a processor, memory, network interface, and database connected via a system bus. The processor provides computing and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system, computer programs, and database. The internal memory provides the environment for the operation of the operating system and computer programs in the non-volatile storage media. The database is used for data storage, data processing, and data analysis. The network interface is used for communication with external terminals via a network connection. When the computer program is executed by the processor, it implements a keyword-coverage-aware retrieval enhancement context construction method.

[0136] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed by a processor, implements a keyword-coverage-aware retrieval enhancement generation context construction method.

[0137] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the embodiments of the above methods. Any references to memory, storage, databases, or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in a variety of forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), RAMbus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0138] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is used as an example. In practical applications, the above functions can be assigned to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above.

[0139] The above-described embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application, and should all be included within the protection scope of this application.

Claims

1. A method for constructing a search enhancement generation context based on keyword coverage awareness, characterized in that, Including the following steps: Get the query statement entered by the user and extract multiple query keywords from the query statement; Based on a pre-defined knowledge base, a retrieval operation is performed on the query keywords to obtain an initial candidate document set including at least one initial candidate document; The occurrence of query keywords in each initial candidate document is statistically analyzed, and the coverage value of each initial candidate document to the query keywords is calculated based on the occurrence. The initial candidate document set is sorted according to the coverage value, and several candidate documents are selected. Determine whether the overall coverage value of the selected candidate documents for the query keywords meets the preset coverage threshold. If not, identify the uncovered query keywords and re-execute the search operation for the uncovered query keywords, and add the supplementary documents obtained from the re-search to the candidate documents. Extract text fragments from candidate documents and construct a context for generating an answer; The constructed context is input into a pre-defined large language model to generate an answer statement for the query statement.

2. The method for constructing a search-enhanced context based on keyword coverage awareness according to claim 1, characterized in that, The step of obtaining the query statement input by the user and extracting multiple query keywords from the query statement specifically includes: The query statement is segmented and stopped word filtered to obtain several candidate words; Candidate words are semantically labeled, and each candidate word is divided into a first keyword and a second keyword according to its semantic role. The first keyword corresponds to the main semantic component in the query statement, and the second keyword corresponds to the modifying semantic component and the limiting semantic component in the query statement. Based on the semantic role of each candidate word, keyword weights are assigned to each candidate word to obtain the query keywords.

3. The method for constructing a search enhancement generation context based on keyword coverage awareness according to claim 1, characterized in that, The step of performing a retrieval operation on the query keywords based on a preset knowledge base to obtain an initial candidate document set including at least one initial candidate document specifically includes: Construct the search query expressions corresponding to each query keyword; By constructing the retrieval query expression, a multi-path parallel retrieval is performed in the preset knowledge base to obtain a preliminary retrieval result set corresponding to each query keyword; The initial search result sets are deduplicated and merged to obtain a candidate document pool. The inverse ranking fusion score is then calculated based on the ranking position of each document in the candidate document pool in the initial search result set. Several documents were selected based on the inverse ranking fusion score to form an initial candidate document set.

4. The method for constructing a search enhancement generation context based on keyword coverage awareness according to claim 3, characterized in that, The step of performing multi-path parallel retrieval in a preset knowledge base using the constructed retrieval query expression to obtain a preliminary retrieval result set corresponding to each query keyword specifically includes: Identify the semantic roles corresponding to each query keyword, and determine the retrieval range parameters corresponding to the query keywords based on the semantic roles; The search query expressions for each query keyword are matched and calculated with the stored documents in the knowledge base to obtain the matching score between each query keyword and each stored document; The stored documents are sorted in descending order according to the matching score of each query keyword, and a number of stored documents are selected according to the retrieval range parameters corresponding to the query keywords. By aggregating several stored documents selected based on each query keyword, preliminary search result sets corresponding to each query keyword are obtained.

5. The method for constructing a search enhancement generation context based on keyword coverage awareness according to claim 1, characterized in that, The step of statistically analyzing the occurrence of query keywords in each initial candidate document and calculating the coverage value of each initial candidate document for the query keywords based on the occurrence of keywords specifically includes: The system detects whether each query keyword appears in each initial candidate document and generates a keyword coverage vector based on the detection results. Each position in the keyword coverage vector corresponds to a query keyword. If the query keyword appears in the initial candidate document, the corresponding position is assigned a value of 1; otherwise, it is assigned a value of 0. Extract the query keywords corresponding to the positions assigned a value of 1 in the keyword coverage vector to obtain the initial candidate document coverage keyword set; Obtain the keyword weights corresponding to each query keyword in the covered keyword set, and sum the keyword weights to obtain the coverage values ​​of the initial candidate documents for the query keywords.

6. The method for constructing a search enhancement generation context based on keyword coverage awareness according to claim 5, characterized in that, The step of determining whether the overall coverage value of the selected candidate documents for the query keywords meets a preset coverage threshold, and if not, identifying the uncovered query keywords, re-performing the search operation for the uncovered query keywords, and adding the supplementary documents obtained from the re-search to the candidate documents, specifically includes: Perform a bitwise OR operation on the keyword coverage vectors of the selected candidate documents to obtain the overall coverage vector; Extract the query keywords corresponding to the positions assigned a value of 1 in the overall coverage vector, obtain the keyword weights corresponding to each extracted query keyword, and sum the keyword weights to obtain the overall coverage weights; Calculate the ratio of the overall coverage weight to the sum of the weights of all query keywords to obtain the overall coverage value; Determine whether the overall coverage value is greater than or equal to the preset coverage threshold. If so, end the coverage determination process. If the overall coverage value is less than the preset coverage threshold, the query keywords corresponding to the positions in the overall coverage vector that are assigned a value of 0 are extracted as the uncovered query keywords. Based on each uncovered query keyword, perform a single keyword search operation in the knowledge base to obtain a supplementary document set corresponding to each uncovered query keyword; Select several documents from each supplementary document set as supplementary documents and add them to the candidate documents.

7. The method for constructing a search enhancement generation context based on keyword coverage awareness according to claim 1, characterized in that, The step of inputting the constructed context into a preset large language model to generate an answer statement for the query statement specifically includes: The query statement is concatenated with the constructed context to generate input prompt text that includes the query statement and context content; Input the input prompt text into the preset large language model, which then performs semantic understanding and reasoning on the query statement based on the context. Obtain the text content output by the large language model as the answer to the query statement; The system checks whether there is any reference content in the response statement that corresponds to a text fragment in a candidate document. If so, it extracts the candidate document identifier corresponding to the reference content and appends the candidate document identifier as the reference source to the response statement.

8. The method for constructing a search enhancement generation context based on keyword coverage awareness according to claim 7, characterized in that, The step of detecting whether there is any cited content in the response statement that corresponds to a text fragment in the candidate document, and if so, extracting the candidate document identifier corresponding to the cited content and appending the candidate document identifier as the citation source to the response statement, specifically includes: The response statement is segmented to obtain several response fragments, and the text similarity between each response fragment and the text fragments in each candidate document is calculated. Determine if there are candidate document text fragments with a text similarity greater than a preset similarity threshold. If so, determine that the corresponding answer fragment is the cited content and record the corresponding candidate document identifier. Summarize the candidate document identifiers corresponding to each cited content to generate a list of cited sources; Append a list of cited sources to your answer, or insert a candidate document identifier as a citation at the corresponding location in each cited content.

9. A retrieval enhancement and context generation system based on keyword coverage awareness, characterized in that, include: The keyword extraction module is used to obtain the query statement input by the user and extract multiple query keywords from the query statement; The knowledge base retrieval module is used to perform retrieval operations on query keywords based on a preset knowledge base to obtain an initial candidate document set including at least one initial candidate document; The coverage calculation module is used to count the occurrence of query keywords in each initial candidate document and calculate the coverage value of each initial candidate document to the query keywords based on the occurrence. The coverage sorting module is used to sort the initial candidate document set according to the coverage value and select a number of candidate documents; The coverage judgment module is used to determine whether the overall coverage value of the selected candidate documents for the query keywords meets the preset coverage threshold. If it does not meet the threshold, it identifies the uncovered query keywords and re-executes the retrieval operation for the uncovered query keywords, adding the supplementary documents obtained from the re-retrieval to the candidate documents. The context building module is used to extract text fragments from candidate documents and build a context for generating answers; The context input module is used to input the constructed context into a preset large language model to generate an answer statement for the query statement.

10. A computer-readable storage medium, characterized in that, A computer program is stored on the computer-readable storage medium, which, when executed by a processor, implements the steps of a keyword-coverage-aware retrieval enhancement generation context construction method as described in any one of claims 1 to 8.

Images