Hybrid search enhancement generation method and apparatus, computer device, and storage medium
By segmenting and preprocessing the knowledge base content, and combining semantic retrieval with keyword retrieval, the false detection problem caused by vector similarity dependence in the RAG system is solved, thereby improving the accuracy and relevance of retrieval results.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- BEIJING JIZHI DIGITAL TECH CO LTD
- Filing Date
- 2026-04-08
- Publication Date
- 2026-06-12
AI Technical Summary
Existing search augmentation generation (RAG) systems rely primarily on vector similarity in the similarity retrieval stage, which carries the risk of false detections and fails to effectively combine keyword matching, resulting in insufficient accuracy and relevance of search results.
By segmenting the knowledge base content into blocks, text vectors and keyword indexes of text blocks are obtained. Combined with the vectors and keyword indexes of user questions, a hybrid retrieval strategy is adopted, combining semantic retrieval and keyword retrieval, to determine the target text blocks.
It improves the accuracy and relevance of search results, enhances the stability and precision of recall, and reduces the possibility of false positives and false negatives.
Smart Images
Figure CN122196139A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of search enhancement generation technology, and in particular to a hybrid search enhancement generation method, apparatus, computer equipment, and storage medium. Background Technology
[0002] Retrieval-Enhanced Generation (RAG) is an architecture that combines an external knowledge base with a generative language model. This method uses a retrieval mechanism to filter query-relevant text fragments from a large number of documents, inputting the retrieved context along with the original query into a language model to generate an answer based on external knowledge.
[0003] In the similarity retrieval phase of the RAG system, retrieval mainly relies on vector similarity, which carries the risk of false detection. Summary of the Invention
[0004] The embodiments described in this specification aim to at least partially solve one of the technical problems in the related art. To this end, the embodiments described in this specification propose a hybrid search enhancement generation method, apparatus, computer device, and storage medium.
[0005] This specification provides a hybrid retrieval enhancement generation method, the method comprising: The knowledge base content is divided into blocks to obtain multiple text blocks; Preprocessing is performed on the multiple text blocks to obtain the text vector and keyword index corresponding to each text block; Based on the user's question, a question vector and the keyword index corresponding to the user's question are obtained; Semantic retrieval and keyword retrieval are performed based on the text vector corresponding to each text block, the keyword index corresponding to each text block, the question vector, and the keyword index corresponding to the user question to determine the target text block.
[0006] In one implementation, the step of performing semantic retrieval and keyword retrieval based on the text vector corresponding to each text block, the keyword index corresponding to each text block, the question vector, and the keyword index corresponding to the user question to determine the target text block includes: Semantic retrieval is performed based on the text vector and question vector corresponding to each text block to determine candidate text blocks; The target text block is determined by using a relevance scoring function to perform keyword retrieval based on the keyword index corresponding to the candidate text block and the keyword index corresponding to the user question.
[0007] In one implementation, the step of performing semantic retrieval and keyword retrieval based on the text vector corresponding to each text block, the keyword index corresponding to each text block, the question vector, and the keyword index corresponding to the user question to determine the target text block includes: Candidate text blocks are determined by using a relevance scoring function to perform keyword retrieval based on the keyword index corresponding to each text block and the keyword index corresponding to the user question. Semantic retrieval is performed based on the text vector corresponding to the candidate text block and the question vector to determine the target text block.
[0008] In one embodiment, the method further includes: Based on the user question and the target text block, a prediction result is generated.
[0009] In one implementation, the step of segmenting the knowledge base content to obtain multiple text blocks includes: A fixed-size block strategy is adopted to divide the knowledge base content according to a preset block size, and an overlap of a preset length is set between adjacent text blocks to obtain multiple text blocks.
[0010] In one implementation, preprocessing is performed on the plurality of text blocks to obtain a keyword index corresponding to each text block, including: calling a function to preprocess the plurality of text blocks to obtain inverse document frequency, term frequency and document length, and determining a keyword index corresponding to each text block based on the inverse document frequency, the term frequency and the document length; The process of processing user questions to obtain the keyword index corresponding to the user questions includes: calling a function to process the user questions, obtaining the total word frequency, and determining the keyword index corresponding to the user questions based on the total word frequency.
[0011] In one implementation, after preprocessing the plurality of text blocks to obtain the text vector corresponding to each text block and the keyword index corresponding to each text block, the method further includes: Store the text vector corresponding to each text block into an array to obtain a text vector array; Store the text vector array into a vector database; The keyword index corresponding to each text block is stored in the database.
[0012] This specification provides a combined search enhancement generation apparatus, the apparatus comprising: The knowledge base segmentation module is used to segment the knowledge base content into multiple text blocks. The text block preprocessing module is used to preprocess the multiple text blocks to obtain the text vector and keyword index corresponding to each text block. The user issue processing module is used to process user issues and obtain issue vectors and keyword indexes corresponding to the user issues. The semantic and keyword retrieval module is used to perform semantic and keyword retrieval based on the text vector corresponding to each text block, the keyword index corresponding to each text block, the question vector, and the keyword index corresponding to the user question, to determine the target text block.
[0013] This specification provides a computer device comprising: a memory, and one or more processors communicatively connected to the memory; the memory stores instructions executable by the one or more processors, the instructions being executed by the one or more processors to cause the one or more processors to perform the steps of the method described in any of the above embodiments.
[0014] This specification provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the method described in any of the above embodiments.
[0015] This specification provides a computer program product that includes instructions that, when executed by a processor of a computer device, enable the computer device to perform the steps of the method described in any of the above embodiments.
[0016] In the above-described implementation method, firstly, the knowledge base content is segmented into multiple text blocks. Then, preprocessing is performed on these text blocks to obtain a text vector and a keyword index for each text block. Next, processing is performed based on user questions to obtain a question vector and a keyword index for each user question. Finally, semantic retrieval and keyword retrieval are performed based on the text vector, keyword index, question vector, and keyword index for each user question to determine the target text block. By introducing keyword retrieval to optimize similarity retrieval, and combining semantic retrieval with keyword retrieval, a hybrid retrieval strategy is formed to improve the accuracy and relevance of the retrieval results. Attached Figure Description
[0017] Figure 1 A flowchart illustrating the process of obtaining visual embedding vectors of multiple pages and summary text information of multiple pages corresponding to a multimodal document, as provided in the embodiments of this specification. Figure 2A flowchart illustrating the hybrid search enhancement generation method provided in the embodiments of this specification; Figure 3 A flowchart illustrating the process of determining a target text block as provided in the embodiments of this specification; Figure 4 A flowchart illustrating the process of determining a target text block as provided in the embodiments of this specification; Figure 5 A flowchart illustrating the storage vectors and indexes provided for the implementation of this specification; Figure 6 A schematic diagram of the hybrid search enhancement generation apparatus provided in the embodiments of this specification; Figure 7 An internal structural diagram of a computer device provided for embodiments of this specification. Detailed Implementation
[0018] Embodiments of the present invention are described in detail below, examples of which are illustrated in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and intended to explain the present invention, and should not be construed as limiting the present invention.
[0019] Retrieval-Enhanced Generation (RAG) is an architecture that combines an external knowledge base with a generative language model. This method uses a retrieval mechanism to filter query-relevant text fragments from a large number of documents, inputting the retrieved context along with the original query into a language model to generate an answer based on external knowledge.
[0020] In the similarity retrieval phase of the RAG system, retrieval primarily relies on vector similarity, neglecting the importance of keyword matching and thus posing a risk of false positives. For example, please refer to [link to example]. Figure 1 , Figure 1 The flowchart of RAG is shown.
[0021] Single retrieval methods have significant limitations: relying solely on vector similarity is prone to false positives, while relying solely on keyword matching may result in false negatives. Therefore, combining keyword matching with semantic similarity to improve the accuracy of the similarity retrieval stage has become a key technical issue in the optimization of the RAG system.
[0022] Based on the above analysis, this specification provides a hybrid retrieval enhancement generation method. First, the knowledge base content is segmented into multiple text blocks. Then, preprocessing is performed on these text blocks to obtain a text vector and a keyword index for each text block. User questions are then processed to obtain a question vector and a keyword index for each user question. Finally, semantic retrieval and keyword retrieval are performed based on the text vector, keyword index, question vector, and keyword index for each user question to determine the target text block. By introducing keyword retrieval to optimize similarity retrieval, and combining semantic retrieval with keyword retrieval, a hybrid retrieval strategy is formed to improve the accuracy and relevance of retrieval results.
[0023] This specification provides a hybrid retrieval enhancement generation method. Please refer to [link / reference]. Figure 2 The hybrid retrieval enhancement generation method may include the following steps: S210. Divide the knowledge base content into blocks to obtain multiple text blocks.
[0024] S220. Preprocess based on multiple text blocks to obtain the text vector corresponding to each text block and the keyword index corresponding to each text block.
[0025] Specifically, the data sources for the knowledge base can be various knowledge resources belonging to a specified category and obtained from the Internet. The storage format of the knowledge base is not limited to a specific format; for example, it can be a document stored in PDF format or any other text format file. In other words, any electronic file capable of carrying text information can serve as the knowledge base used in this method, and its applicability is not particularly limited by the file format.
[0026] Specifically, the content in the knowledge base is segmented according to preset rules or semantic boundaries, dividing the overall content into multiple text blocks with semantic units. Then, two parallel preprocessing operations are performed on each text block. First, each text block is vectorized using a pre-selected embedding model (e.g., text2vec), mapping it to a vector representation in space to obtain a text vector that represents its semantic information, thus obtaining the text vector corresponding to each text block. Second, a keyword index is calculated for each text block; that is, representative keywords are extracted from each text block using methods such as word segmentation, word frequency statistics, and keyword extraction, and a keyword index is built for each text block based on these keywords.
[0027] S230. Process user questions to obtain question vectors and keyword indexes corresponding to user questions.
[0028] Specifically, the process begins by acquiring the user's input question and then executing two parallel processing flows based on that question. First, a pre-selected embedding model (e.g., text2vec) is used to vectorize the user question, mapping it into a vector representation in space, thus obtaining a question vector that represents its semantic information. Second, a keyword index is calculated for the user question; that is, representative keywords are extracted from the user question using methods such as word frequency statistics, and a keyword index corresponding to the user question is built based on these keywords.
[0029] S240. Based on the text vector corresponding to each text block, the keyword index corresponding to each text block, the question vector, and the keyword index corresponding to the user question, perform semantic retrieval and keyword retrieval to determine the target text block.
[0030] Specifically, a hybrid retrieval mechanism can be adopted: on the one hand, vector similarity calculation is performed between the question vector and the text vector of each text block to achieve semantic content-based retrieval recall. On the other hand, keyword matching is performed between the keyword index corresponding to the user question and the keyword index of each text block to achieve keyword-level retrieval. The above two retrieval methods are executed in a predetermined order. For example, semantic retrieval is used to initially filter out candidate text blocks with high similarity, and then keyword retrieval is used to further filter the candidate results. Alternatively, the order can be adjusted according to the actual scenario to gradually narrow down the scope and accurately locate the final target text block.
[0031] In the above implementation, firstly, the knowledge base content is segmented into multiple text blocks. Then, preprocessing is performed on these text blocks to obtain a text vector and a keyword index for each text block. Next, processing is performed based on user questions to obtain a question vector and a keyword index for each user question. Finally, semantic retrieval and keyword retrieval are performed based on the text vector, keyword index, question vector, and keyword index for each user question to determine the target text block. By introducing keyword retrieval to optimize similarity retrieval, and combining semantic retrieval with keyword retrieval, a hybrid retrieval strategy is formed to improve the accuracy and relevance of the retrieval results.
[0032] In some implementations, please refer to Figure 3 Based on the text vector corresponding to each text block, the keyword index corresponding to each text block, the question vector, and the keyword index corresponding to the user question, semantic retrieval and keyword retrieval are performed to determine the target text block. This may include the following steps: S310. Perform semantic retrieval based on the text vector and question vector corresponding to each text block to determine candidate text blocks.
[0033] Specifically, the semantic similarity between the text vector corresponding to each text block and the question vector generated from the user's question is calculated. By quantifying the distance or similarity between the two in a high-dimensional semantic space, such as using cosine similarity or dot product, semantic-level matching and retrieval are achieved. Based on the calculated semantic similarity score, a pre-set filtering strategy is used, such as setting a similarity threshold to filter out text blocks exceeding the threshold, or selecting the top-ranked text blocks after sorting them by similarity from high to low. These text blocks are then identified as candidate text blocks for subsequent processing stages.
[0034] S320. Use the relevance scoring function to perform keyword retrieval based on the keyword index corresponding to the candidate text block and the keyword index corresponding to the user question to determine the target text block.
[0035] In some cases, considering that relying solely on vector similarity for semantic matching may result in false detections, that is, although some text blocks are semantically similar to the user's question, there may be deviations in the direct correspondence of key information, a relevance scoring mechanism based on keyword matching is further introduced on the basis of obtaining candidate text blocks to refine the screening of search results.
[0036] Specifically, after obtaining candidate text blocks, a relevance scoring function is used to quantify the relevance between the keyword index of the user's question and the keyword index of the corresponding candidate text block by calculating the degree of matching between the two, thus obtaining a relevance score for each candidate text block. Based on this score, according to preset screening conditions, such as setting a scoring threshold or selecting a specified number of text blocks from high to low scores, multiple text blocks that meet the conditions are selected from the candidate text blocks and identified as target text blocks for subsequent processing. The relevance scoring function can be either BM25 or TF-IDF algorithms in progress.
[0037] In the above implementation, semantic retrieval is performed based on the text vector and question vector corresponding to each text block to determine candidate text blocks. Then, a relevance scoring function is used to perform keyword retrieval based on the keyword index corresponding to the candidate text blocks and the keyword index corresponding to the user's question to determine the target text block, thereby enhancing the stability and relevance of the recall.
[0038] In some implementations, please refer to Figure 4 Based on the text vector corresponding to each text block, the keyword index corresponding to each text block, the question vector, and the keyword index corresponding to the user question, semantic retrieval and keyword retrieval are performed to determine the target text block. This may include the following steps: S410. Use the relevance scoring function to perform keyword retrieval based on the keyword index corresponding to each text block and the keyword index corresponding to the user question to determine candidate text blocks.
[0039] Specifically, a relevance scoring function is used to quantify the relevance between the keyword index of the user's question and the keyword index of each text block by calculating the degree of matching between the two, thus obtaining a relevance score for each text block. Based on this score, according to preset filtering conditions, such as setting a scoring threshold or selecting a specified number of text blocks from high to low scores, multiple text blocks that meet the conditions are selected from all text blocks and identified as candidate text blocks for subsequent processing stages.
[0040] S420. Perform semantic retrieval based on the text vector and question vector corresponding to the candidate text block to determine the target text block.
[0041] In some cases, when relying solely on keyword retrieval, the semantic connections may not be captured due to differences between the way users express their questions and the wording in the text blocks, leading to potential missed retrievals. To overcome this limitation, a semantic retrieval step is introduced after filtering candidate text blocks through keyword retrieval.
[0042] Specifically, in the constructed vector library, text vectors corresponding to candidate texts are obtained through retrieval. Then, semantic similarity is calculated between the text vectors corresponding to the candidate text blocks and the question vectors generated from the user's question. By quantifying the distance or similarity between the two in a high-dimensional semantic space, semantic matching and retrieval are achieved. Based on the calculated semantic similarity score, a pre-set filtering strategy is used, such as setting a similarity threshold to filter out text blocks exceeding that threshold, or selecting the top-ranked text blocks after sorting them by similarity from high to low, to determine the final target text block from the candidate text blocks, which is then output as the retrieval result.
[0043] For example, if the retrieval target is to obtain the five text blocks with the highest semantic similarity to the user's question, the implementation method of this specification constructs a two-stage retrieval process. First, a preliminary recall is performed from the full dataset using a relevance scoring function (such as the BM25 algorithm) based on keyword matching, obtaining approximately 100 candidate text blocks. Subsequently, within a relatively small range comprised of these 100 candidate text blocks, a secondary retrieval is performed by calculating the semantic similarity between the question vector and the text vectors corresponding to each candidate text block, and the blocks are sorted from high to low similarity. Finally, the five text blocks with the highest similarity are selected as the target text blocks for output.
[0044] In the above implementation, a relevance scoring function is used to perform keyword retrieval based on the keyword index corresponding to each text block and the keyword index corresponding to the user question to determine candidate text blocks. Semantic retrieval is then performed based on the text vector and question vector corresponding to the candidate text blocks to determine the target text blocks, thereby enhancing the stability and relevance of the recall.
[0045] In some implementations, the method may further include: making predictions based on user questions and target text blocks, and generating prediction results.
[0046] Specifically, following a pre-defined prompt template, the user question and target text block are structurally assembled, for example, by adding specific delimiters or instruction prefixes through concatenation, to form a complete prompt that conforms to the input format of a large language model. This prompt is then passed as input to a large language model (e.g., Qwen-7B-Chat). Upon receiving the prompt, the model performs inference calculations based on its internal parameters and pre-trained knowledge, ultimately generating a prediction result that semantically matches the prompt. This result can be the answer to the user question.
[0047] It should be noted that after obtaining the prediction result, which is the answer to the user's question, it is necessary to use evaluation metrics to measure the consistency between the prediction result and the reference answer.
[0048] In some implementations, the evaluation metric can be obtained using BLEU (Bilingual Assessment Alternate). A higher BLEU value indicates higher accuracy of the prediction results in tasks such as translation. This metric focuses primarily on how much of the predicted content is correct, rather than whether the information in the reference answer is fully covered by the prediction results.
[0049] In other implementations, the evaluation metric can be obtained using ROUGE-L (Recall-Oriented Evaluation of Long Common Subsequences). A higher ROUGE-L value indicates a more complete coverage of key information in tasks such as summarizing. This metric focuses on measuring recall, i.e., how much content in the reference answer is covered by the predicted results, thus providing a more comprehensive evaluation of the prediction results' performance in terms of content completeness.
[0050] In the above implementation, prediction is made based on the user's question and the target text block determined by the retrieval, thereby improving the accuracy of the prediction results.
[0051] In some implementations, the knowledge base content is divided into blocks to obtain multiple text blocks. This may include: using a fixed-size block strategy to divide the knowledge base content according to a preset block size, and setting an overlap of a preset length between adjacent text blocks to obtain multiple text blocks.
[0052] Specifically, a fixed-size chunking strategy is used to segment the text content in the knowledge base. Specifically, based on a preset chunk size, the knowledge base content is divided into multiple text chunks of fixed length. To avoid potential information gaps or breaks in key semantics at chunk boundaries, a preset-length overlap region is established between adjacent text chunks; that is, the end of the current text chunk partially overlaps with the beginning of the next text chunk. By introducing this overlap mechanism, smooth semantic connections can be formed between chunks, effectively ensuring the continuity of cross-chunk context and the integrity and coverage of key information, ultimately generating multiple text chunks for subsequent processing or retrieval.
[0053] In the above implementation, a fixed-size block strategy is adopted to divide the knowledge base content according to a preset block size, and an overlap of a preset length is set between adjacent text blocks to obtain multiple text blocks, thereby improving recall and answer completeness in scenarios such as retrieval enhancement generation.
[0054] In some implementations, preprocessing based on multiple text blocks to obtain the keyword index corresponding to each text block may include: calling a function to preprocess multiple text blocks to obtain inverse document frequency, term frequency and document length, and determining the keyword index corresponding to each text block based on the inverse document frequency, term frequency and document length.
[0055] Specifically, this implementation uses the BM25 (Best Matching 25) algorithm for keyword retrieval. Therefore, it is necessary to pre-build and determine the corresponding keyword index. The core idea of the BM25 algorithm is to quantify the relevance between a document and a query by calculating the degree of matching between the query and the document at the keyword level. Specifically, the calculation of this relevance score mainly depends on the following three factors: Term Frequency (TF), Inverse Document Frequency (IDF), and document length.
[0056] The process requires calling a preprocessing function to sequentially clean multiple text blocks, followed by word segmentation to divide the continuous text into independent word sequences. Finally, word statistics are performed based on the segmentation results. For example, the relevant statistical functions in the open-source library rank_bm25 can be used to obtain the inverse document frequency, term frequency, and document length of each word in the entire text block set. Here, the text block set refers to the collection of all text blocks obtained after segmenting the knowledge base content.
[0057] Inverse document frequency (IDF) measures how rare a term is: the fewer text blocks a term appears in the entire set of text blocks (i.e., the rarer the term), the greater its role in distinguishing the topics of different text blocks, and therefore the higher its IDF value. In other words, IDF focuses on the scarcity of a term in the global set, rather than the number of times it appears locally within a single text block.
[0058] Term frequency refers to the number of times a term appears in a single text block. The more frequently a term appears in a text block, the more it is generally considered to represent the core content of that text block, and therefore it has a higher weight in relevance assessment. In other words, term frequency reflects the importance of a term in a local text block.
[0059] Document length, defined as the sum of the number of terms in a single text block, is used to balance the scores of long and short text blocks. Specifically, longer text blocks, containing more terms, may naturally receive higher word frequencies, leading to a higher relevance score. To prevent long text blocks from unfairly receiving excessively high scores, the BM25 algorithm introduces a normalization factor based on document length. This factor first calculates the average length of all text blocks in the current block set, i.e., the arithmetic mean of the number of terms in each text block, which serves as the benchmark for length comparison.
[0060] Finally, based on inverse document frequency, term frequency, and document length, combined with average length, a corresponding keyword index is determined for each text block, thereby supporting subsequent keyword retrieval operations based on the BM25 algorithm.
[0061] In the above implementation, a function is called to preprocess multiple text blocks to obtain inverse document frequency, term frequency, and document length. Based on the inverse document frequency, term frequency, and document length, the keyword index corresponding to each text block is determined, providing a data foundation for subsequent keyword retrieval.
[0062] In some implementations, processing based on user questions to obtain the keyword index corresponding to the user questions may include: calling a function to process the user questions, obtaining the total word frequency, and determining the keyword index corresponding to the user questions based on the total word frequency.
[0063] Specifically, a preprocessing function is invoked to clean the user questions, followed by word segmentation to divide consecutive user questions into independent word sequences. Finally, word statistics are performed based on the segmentation results. For example, the relevant statistical functions in the open-source library rank_bm25 can be called to count the total number of all words included in the user questions, obtaining the total word frequency. Ultimately, based on the total word frequency, a corresponding keyword index is determined for the user questions, thus supporting subsequent keyword retrieval operations based on the BM25 algorithm.
[0064] In the above implementation, the function is called to process the data and obtain the total word frequency. Based on the total word frequency, the keyword index corresponding to the user's question is determined, providing a data foundation for subsequent keyword retrieval.
[0065] It should be noted that after obtaining the keyword indexes for each text block and the user question, the BM25 algorithm can be used to calculate the relevance score between the user question and the text block. The specific calculation formula is as follows:
[0066]
[0067] in, For user questions, consisting of several terms constitute; For any text block; For user issues The total number of word frequencies; Asking users The first in 1 term; For terms Inverse document frequency; For terms In text block The frequency of a word's occurrence in a word is its frequency. For text blocks The document length; The average length of all text blocks; The total number of text blocks; For included terms The number of text blocks; , These are adjustable parameters; For user issues With text blocks Correlation score between them.
[0068] In some implementations, please refer to Figure 5After preprocessing multiple text blocks to obtain the text vector and keyword index for each text block, the following steps can be included: S510. Store the text vector corresponding to each text block into an array to obtain a text vector array.
[0069] S520. Store the text vector array into the vector database.
[0070] S530. Store the keyword index corresponding to each text block in the database.
[0071] Specifically, after obtaining the text vector corresponding to each text block, these text vectors are stored sequentially into an array structure to form a text vector array. Next, this text vector array is written as a dataset into a vector database (e.g., Faiss). This database supports selecting appropriate index types (e.g., IndexFlatL2, IndexFlatIP, IndexIVFFlat, etc.) based on the actual application scenario to achieve efficient vector similarity queries while ensuring retrieval accuracy. Furthermore, for each text block, its corresponding keyword index is stored separately in a relational database or key-value store system to support keyword-based retrieval operations.
[0072] In the above implementation, by using dual-path storage of vectors and keywords, both the depth of semantic retrieval and the accuracy of keyword retrieval are taken into account, providing efficient, flexible and scalable text retrieval and recall capabilities for upper-layer applications.
[0073] This specification provides a hybrid retrieval enhancement generation apparatus 600. Please refer to [link / reference]. Figure 6 The hybrid retrieval enhancement generation device 600 includes: a knowledge base block processing module 610, a text block preprocessing module 620, a user question processing module 630, and a semantic and keyword retrieval module 640.
[0074] The knowledge base segmentation module 610 is used to segment the knowledge base content into multiple text blocks. The text block preprocessing module 620 is used to preprocess the multiple text blocks to obtain the text vector and the keyword index corresponding to each text block. User question processing module 630 is used to process user questions and obtain question vectors and keyword indexes corresponding to the user questions; The semantic and keyword retrieval module 640 is used to perform semantic and keyword retrieval based on the text vector corresponding to each text block, the keyword index corresponding to each text block, the question vector, and the keyword index corresponding to the user question, to determine the target text block.
[0075] For a detailed description of the hybrid retrieval enhancement generation device, please refer to the description of the hybrid retrieval enhancement generation method above, which will not be repeated here.
[0076] In some embodiments, a computer device is provided, which may be a terminal, and its internal structure diagram may be as follows: Figure 7 As shown, the computer device includes a processor, memory, communication interface, display screen, and input devices connected via a system bus. The processor provides computing and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The communication interface is used for wired or wireless communication with external terminals; wireless communication can be achieved through Wi-Fi, carrier networks, NFC (Near Field Communication), or other technologies. When the computer program is executed by the processor, it implements a hybrid retrieval enhancement generation method. The display screen can be an LCD screen or an e-ink screen. The input devices can be a touch layer covering the display screen, buttons, a trackball, or a touchpad mounted on the computer device casing, or an external keyboard, touchpad, or mouse.
[0077] Those skilled in the art will understand that Figure 7 The structures shown are merely block diagrams of some structures related to the solutions disclosed in this specification, and do not constitute a limitation on the computer device to which the solutions disclosed in this specification are applied. Specifically, the computer device may include more or fewer components than shown in the figures, or combine certain components, or have different component arrangements.
[0078] In some embodiments, a computer device is provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the method steps described above.
[0079] This specification provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the method in any of the above embodiments.
[0080] One embodiment of this specification provides a computer program product including instructions that, when executed by a processor of a computer device, enable the computer device to perform the steps of the method described in any of the above embodiments.
[0081] It should be noted that the logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as a sequenced list of executable instructions for implementing logical functions, and can be specifically implemented in any computer-readable medium for use by, or in conjunction with, an instruction execution system, apparatus, or device (such as a computer-based system, a processor-included system, or other system that can fetch and execute instructions from, an instruction execution system, apparatus, or device). For the purposes of this specification, "computer-readable medium" can be any means that can contain, store, communicate, propagate, or transmit programs for use by, or in conjunction with, an instruction execution system, apparatus, or device. More specific examples (a non-exhaustive list) of computer-readable media include: an electrical connection having one or more wires (electronic device), a portable computer disk drive (magnetic device), random access memory (RAM), read-only memory (ROM), erasable and editable read-only memory (EPROM or flash memory), fiber optic devices, and portable optical disc read-only memory (CDROM). Furthermore, computer-readable media can even be paper or other suitable media on which programs can be printed, because programs can be obtained electronically, for example, by optically scanning the paper or other media, followed by editing, interpreting, or otherwise processing as necessary, and then stored in computer memory.
Claims
1. A hybrid retrieval enhancement generation method, characterized in that, The method includes: The knowledge base content is divided into blocks to obtain multiple text blocks; Preprocessing is performed on the multiple text blocks to obtain the text vector and keyword index corresponding to each text block; Based on the user's question, a question vector and the keyword index corresponding to the user's question are obtained; Semantic retrieval and keyword retrieval are performed based on the text vector corresponding to each text block, the keyword index corresponding to each text block, the question vector, and the keyword index corresponding to the user question to determine the target text block.
2. The method according to claim 1, characterized in that, The step of performing semantic retrieval and keyword retrieval based on the text vector corresponding to each text block, the keyword index corresponding to each text block, the question vector, and the keyword index corresponding to the user question to determine the target text block includes: Semantic retrieval is performed based on the text vector and question vector corresponding to each text block to determine candidate text blocks; The target text block is determined by using a relevance scoring function to perform keyword retrieval based on the keyword index corresponding to the candidate text block and the keyword index corresponding to the user question.
3. The method according to claim 1, characterized in that, The step of performing semantic retrieval and keyword retrieval based on the text vector corresponding to each text block, the keyword index corresponding to each text block, the question vector, and the keyword index corresponding to the user question to determine the target text block includes: Candidate text blocks are determined by using a relevance scoring function to perform keyword retrieval based on the keyword index corresponding to each text block and the keyword index corresponding to the user question. Semantic retrieval is performed based on the text vector corresponding to the candidate text block and the question vector to determine the target text block.
4. The method according to claim 1, characterized in that, The method further includes: Based on the user question and the target text block, a prediction result is generated.
5. The method according to claim 1, characterized in that, The process of segmenting the knowledge base content into blocks to obtain multiple text blocks includes: A fixed-size block strategy is adopted to divide the knowledge base content according to a preset block size, and an overlap of a preset length is set between adjacent text blocks to obtain multiple text blocks.
6. The method according to claim 1, characterized in that, Preprocessing is performed on the multiple text blocks to obtain the keyword index corresponding to each text block, including: calling a function to preprocess the multiple text blocks to obtain the inverse document frequency, term frequency and document length, and determining the keyword index corresponding to each text block based on the inverse document frequency, the term frequency and the document length; The process of processing user questions to obtain the keyword index corresponding to the user questions includes: calling a function to process the user questions, obtaining the total word frequency, and determining the keyword index corresponding to the user questions based on the total word frequency.
7. The method according to claim 1, characterized in that, After preprocessing the multiple text blocks to obtain the text vector and keyword index corresponding to each text block, the method further includes: Store the text vector corresponding to each text block into an array to obtain a text vector array; Store the text vector array into a vector database; The keyword index corresponding to each text block is stored in the database.
8. A hybrid retrieval enhancement generation device, characterized in that, The device includes: The knowledge base segmentation module is used to segment the knowledge base content into multiple text blocks. The text block preprocessing module is used to preprocess the multiple text blocks to obtain the text vector and keyword index corresponding to each text block. The user issue processing module is used to process user issues and obtain issue vectors and keyword indexes corresponding to the user issues. The semantic and keyword retrieval module is used to perform semantic and keyword retrieval based on the text vector corresponding to each text block, the keyword index corresponding to each text block, the question vector, and the keyword index corresponding to the user question, to determine the target text block.
9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 7.
10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 7.