Retrieval enhancement generation method integrating multi-type retrieval

Abstract

The invention discloses a retrieval enhancement generation method integrating multi-type retrieval. In the same system, multiple retrieval modes of abstracts, word vectors and knowledge trees are comprehensively utilized, the results of multiple types of retrieval are effectively integrated, and the efficient, accurate and intelligent retrieval enhancement generation method is provided. Meanwhile, according to business rules, the knowledge fragments are sorted, the associated knowledge fragments are organized into a knowledge tree, and the comprehensiveness and accuracy of retrieval are improved by associating and expanding a single entity through the knowledge tree. When expected knowledge nodes cannot be recalled, associated nodes can be dynamically added in the knowledge tree, and the knowledge retrieval accuracy can be continuously improved along with the increase of the use time.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to a retrieval enhancement generation method for comprehensive multi-type retrieval. Background Art

[0002] In the prior art, in the retrieval enhancement generation scheme, the accuracy of the generated content is low due to the low recall rate of knowledge fragmentation, which leads to low effective utilization of computing resources. With limited computing resources, it is necessary to improve the accuracy of the generated content to improve the effective utilization of resources. In the prior art, there are mainly two retrieval enhancement generation schemes, one is retrieval enhancement generation based on keyword search, and the other is retrieval enhancement generation based on vector retrieval. Among them:

[0003] 1) Enhanced search generation based on keyword search: This method is based on inverted index or relational database technology, and searches through direct matching of keywords. It is fast and accurate, but has poor understanding of semantics and often cannot fully meet complex user query requirements. This method relies solely on keywords and cannot capture the semantic intent behind user queries, resulting in low relevance of search results. When faced with complex situations such as synonyms, antonyms, and fuzzy expressions, the traditional keyword search method is not ideal. Because complete knowledge is often not retrieved, the accuracy of subsequent generated content is very low, resulting in low effective utilization of computing resources.

[0004] 2) Retrieval enhancement generation based on vector retrieval: Use the embedding model to vectorize the query documents and store them using vector database technology. Then use the same embedding model to vectorize the user's query requirements and use the vector retrieval algorithm to retrieve. This solution can retrieve content with similar semantics, but the semantic understanding effect is extremely dependent on the embedding model. In massive data scenarios, semantic misunderstandings often occur due to severe vector distortion. Although vector retrieval has achieved semantic retrieval to a certain extent, the output vector dimension of the embedding model it relies on supports up to 2048 dimensions. When the number of words in the knowledge segment is large, the required knowledge is often not retrieved. And it is impossible to retrieve knowledge with large semantic deviations but related. Due to the lack of knowledge segmentation, the accuracy of subsequent generated content is very low, resulting in low effective utilization of computing resources.

[0005] Therefore, the prior art has defects and needs to be improved. Summary of the invention

[0006] The purpose of the present invention is to overcome the deficiencies of the prior art and provide a retrieval enhancement generation method for comprehensive multi-type retrieval.

[0007] The technical solution of the present invention is as follows: a search enhancement generation method for comprehensive multi-type search is provided, comprising the following steps:

[0008] Step 1: Use the large model’s text summarization capabilities to summarize customer questions and extract summaries;

[0009] Step 2: Use the extracted summary and original question to pass into the multi-type retrieval module respectively, and return multiple retrieval results;

[0010] Step 3: Merge the multiple search results returned in step 2, remove duplicates, obtain the fragmented content corresponding to the knowledge node associated with the search result in the generated knowledge tree, merge it into the search result, and exit the multi-type search module;

[0011] Step 4: De-duplicate the fragmented content corresponding to the knowledge nodes returned by the multi-type retrieval module and integrate them into a complete knowledge tree;

[0012] Step 5: Use prompt word engineering technology to fill the integrated knowledge into the prompt word;

[0013] Step 6: Use the above prompt words to generate reply content in the large model content generation module.

[0014] Furthermore, the specific steps of step 2 are:

[0015] Step 2.1: Use inverted index and relational database technology to match the incoming summary and original question text respectively, and return the search results;

[0016] Step 2.2: Use the embedding model to convert the incoming summary and original question into vector values ​​respectively, use the vector database technology to retrieve the word vectors similar to them, and return the corresponding knowledge fragments.

[0017] Furthermore, in step 4, the knowledge fragments returned by the multi-type search module are sorted, and the related knowledge fragments are organized into a knowledge tree, and a single entity is associated and expanded through the knowledge tree.

[0018] Furthermore, when it is encountered that the expected knowledge node cannot be recalled, the associated node is dynamically added to the knowledge tree.

[0019] By adopting the above scheme, the present invention comprehensively utilizes multiple search methods such as abstracts, word vectors, and knowledge trees in the same system, and effectively integrates the results of multiple types of searches, providing an efficient, accurate, and intelligent search enhancement generation method. At the same time, according to business rules, the knowledge fragments are sorted, and the related knowledge fragments are organized into a knowledge tree. The knowledge tree is used to associate and expand a single entity to improve the comprehensiveness and accuracy of the search. When it is impossible to recall the expected knowledge node, the associated node can be dynamically added to the knowledge tree. As the usage time increases, the accuracy of knowledge retrieval can also be continuously improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] Figure 1 It is a schematic diagram of the process of the present invention. DETAILED DESCRIPTION

[0021] The present invention is described in detail below in conjunction with the accompanying drawings and specific embodiments.

[0022] See also Figure 1 The present invention provides a search enhancement generation method for comprehensive multi-type search, comprising the following steps:

[0023] Step 1: Use the large model’s text summarization capabilities to summarize customer questions and extract summaries.

[0024] Step 2: Use the extracted summary and original question to pass into the multi-type retrieval module and return multiple retrieval results. The specific steps are:

[0025] Step 2.1: Use inverted index and relational database technology to match the incoming summary and original question text respectively, and return the search results.

[0026] Step 2.2: Use the embedding model to convert the incoming summary and original question into vector values ​​respectively, use the vector database technology to retrieve the word vectors similar to them, and return the corresponding knowledge fragments.

[0027] Step 3: Merge the multiple search results returned in step 2, remove duplicates, obtain the fragmented content corresponding to the knowledge nodes associated with the search results in the generated knowledge tree, merge them into the search results, and exit the multi-type search module.

[0028] Step 4: De-duplicate the fragments corresponding to the knowledge nodes returned by the multi-type retrieval module and integrate them into a complete knowledge tree. Sort the knowledge fragments returned by the multi-type retrieval module, organize the related knowledge fragments into a knowledge tree, and associate and expand a single entity through the knowledge tree. When the expected knowledge node cannot be recalled, dynamically add the associated node to the knowledge tree.

[0029] Step 5: Use prompt word engineering technology to fill the integrated knowledge into the prompt word.

[0030] Step 6: Use the above prompt words to generate reply content in the large model content generation module.

[0031] In the specific embodiment provided by the present invention, the user sends a question to the system, and the system's large model summarizes the question text and extracts a summary from the text. The extracted summary and the original text are input into different retrieval modules for retrieval and recognition.

[0032] In one of the search branches of the multi-type search module, inverted index and relational database technology are used to match the incoming abstract and the text of the original question respectively, and the corresponding search results are returned.

[0033] An inverted index is an index structure used in information retrieval that allows the system to quickly find a list of documents containing a word. The inverted index consists of the following steps: Parse the original text, identify the document format, language, and encoding, and decompose the document content into a series of terms. Remove common, less meaningful terms from the terms. Then convert the terms to a standard form and convert the terms to their base form or roots. Based on the processed terms, build an inverted index.

[0034] Relational database technology is a data management technology based on the relational model, which organizes and stores data in the form of a two-dimensional table. Relational database technology includes the following steps: verify the correctness of the original question text and summary that are passed in, and generate the required full-text index for the attribute columns of the text type that needs to be published in the database. Store the schema information in the system database, and create a full-text catalog on the hard disk for the database to be published. Access each relationship according to the schema information, traverse the attribute columns, generate a full-text index for the attributes that need full-text retrieval, and fill the full-text index after all relationships are accessed. After completing the above preprocessing operations, enter the query processing stage, process the user's keyword query, and find the relationship data related to the original text and summary of the user's question that are passed in. The query process is completed by searching the inverted index and performing set operations. Construct a tuple set connection graph, explore the relationship between each tuple set, find the minimum connection subtree related to the user's keyword, and generate the final retrieval result from the candidate tuple set connection tree.

[0035] In another retrieval branch in the multi-type retrieval module, the embedding model is used to convert the incoming summary and the original question text into vector values, and the vector database technology is used to retrieve similar word vectors and return the corresponding knowledge fragments.

[0036] An embedding model is a model used to convert high-dimensional, discrete data (such as words, sentences, images, users, etc.) into low-dimensional, continuous vector representations (i.e., embeddings). Through this conversion, the semantics, structure, or features of the original data can be represented in digital form in the vector space, so that the representations of similar objects in this space are close to each other, which is convenient for computers to post-process and analyze. The specific steps of the embedding model for vector conversion usually include the following stages: preprocessing the incoming user original questions and summaries, including word segmentation, removal of stop words, stem extraction, etc., to prepare data for model training. Select a suitable embedding model according to the needs, and use the prepared data to train the selected embedding model to learn the vector representation of the vocabulary. The preprocessed text and summary are input into the trained model respectively. The model generates a fixed-dimensional vector representation for each vocabulary. These vectors capture the semantic relationship between words, and similar words are closer in the vector space.

[0037] The generated vector is represented in a vector database, and similar word vectors are retrieved from the vector database. Vector database technology is a database system specifically used to store and manage vector data. It can efficiently process large amounts of high-dimensional data and provide rich query and analysis functions. The specific steps for retrieving matching word vectors using vector database technology are: receiving a vector representation index, performing a search using a vector database, returning the most relevant results, and tuning the index structure and search algorithm according to specific application requirements to balance search accuracy, latency, and throughput. For large-scale data sets, a distributed processing solution is required to shard the data and process queries in parallel on multiple servers.

[0038] After searching in different branches, merge the search results of each branch, deduplicate the merged results, and generate a knowledge tree. Obtain the fragmented content corresponding to the knowledge node associated with the search result in the processed knowledge tree. Sort the knowledge fragments returned by the multi-type retrieval module, organize the related knowledge fragments into a knowledge tree, and associate and expand a single entity through the knowledge tree. When it is impossible to recall the expected knowledge node, dynamically add the related node to the knowledge tree. Use prompt word engineering technology to fill the integrated knowledge into the prompt word, generate the reply content in the large model content generation module and reply to the user.

[0039] In summary, the present invention comprehensively utilizes multiple search methods such as abstracts, word vectors, and knowledge trees in the same system, and effectively integrates the results of multi-type searches, providing an efficient, accurate, and intelligent search enhancement generation method. At the same time, according to business rules, knowledge fragments are sorted, and related knowledge fragments are organized into a knowledge tree. The knowledge tree is used to associate and expand a single entity to improve the comprehensiveness and accuracy of the search. When it is impossible to recall the expected knowledge node, the associated node can be dynamically added to the knowledge tree. As the usage time increases, the accuracy of knowledge retrieval can also be continuously improved.

[0040] The above are only preferred embodiments of the present invention and are not intended to limit the present invention. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A search enhancement generation method for comprehensive multi-type search, characterized in that: The steps include: Step 1: Use the large model’s text summarization capabilities to summarize customer questions and extract summaries; Step 2: Use the extracted summary and original question to pass into the multi-type retrieval module respectively, and return multiple retrieval results; Step 3: Merge the multiple search results returned in step 2, remove duplicates, obtain the fragmented content corresponding to the knowledge node associated with the search result in the generated knowledge tree, merge it into the search result, and exit the multi-type search module; Step 4: De-duplicate the fragmented content corresponding to the knowledge nodes returned by the multi-type retrieval module and integrate them into a complete knowledge tree; Step 5: Use prompt word engineering technology to fill the integrated knowledge into the prompt word; Step 6: Use the above prompt words to generate reply content in the large model content generation module.

2. The search enhancement generation method for comprehensive multi-type search according to claim 1 is characterized in that: The specific steps of step 2 are: Step 2.1: Use inverted index and relational database technology to match the incoming summary and original question text respectively, and return the search results; Step 2.2: Use the embedding model to convert the incoming summary and original question into vector values ​​respectively, use the vector database technology to retrieve the word vectors similar to them, and return the corresponding knowledge fragments.

3. The search enhancement generation method for comprehensive multi-type search according to claim 1 is characterized in that: In the step 4, the knowledge fragments returned by the multi-type search module are sorted, and the related knowledge fragments are organized into a knowledge tree, and a single entity is associated and expanded through the knowledge tree.

4. The search enhancement generation method for comprehensive multi-type search according to claim 3 is characterized in that: When an expected knowledge node cannot be recalled, an associated node is dynamically added to the knowledge tree.

Images