Multi-knowledge-base retrieval and answer integration optimization method, retrieval enhancement generation system, equipment and medium

Abstract

The invention relates to a multi-knowledge-base retrieval and answer integration optimization method, a retrieval enhancement generation system, equipment and a medium, and the method comprises the steps: respectively sending questions input by a user into N knowledge bases, and converting the questions into question vectors suitable for the specific semantic space of each knowledge base by using an embedding algorithm corresponding to each knowledge base; obtaining the first M most relevant results in each knowledge base by calculating the similarity between the problem vector and the document vector of each knowledge base; integrating the results of all the knowledge bases to obtain M * N results; and reordering the merged results by adopting a Rerank algorithm, and selecting the first m results with the highest scores as final recall results. Compared with the prior art, the method has the capability of improving the information retrieval efficiency and quality, and provides better query service for the user.

Description

Technical Field

[0001] The present invention relates to the field of database retrieval, and in particular to a multi-knowledge base retrieval and answer integration optimization method. Background Art

[0002] Retrieval-augmented generation (RAG) systems combine powerful information retrieval techniques with advanced language generation models to provide more accurate and detailed answers. Existing RAG systems usually rely on a single knowledge base or limited data sources to retrieve information. They still face some limitations when processing complex queries, especially when extensive and deep expertise is required. Using multiple knowledge bases can greatly expand the knowledge coverage and depth of the system. These systems usually integrate data from multiple sources, including professional databases, public paper collections, online encyclopedias, and professional knowledge bases, to process and answer various complex queries.

[0003] In order to effectively retrieve information from these diverse knowledge sources, the system requires powerful indexing and search technologies, as well as efficient algorithms to determine which information is most relevant. Generally, a single knowledge base in the system corresponds to a collection in the vector database, which contains all embedded vector values ​​and embedded objects under the knowledge base.

[0004] Existing solutions for retrieving all knowledge bases have the following drawbacks: The system queries all available knowledge bases simultaneously to maximize the search scope and increase the probability of finding relevant information. This usually involves complex query distribution mechanisms and result merging techniques to obtain and integrate answers from each knowledge base. This can maximize the probability of finding high-quality, relevant answers, especially when dealing with very specific or rare questions. However, different knowledge bases have different embedding algorithms, so the merging and sorting of recalled results can be very complicated. Summary of the invention

[0005] The purpose of the present invention is to overcome the defects of the above-mentioned prior art and provide a more efficient and accurate multi-knowledge base retrieval and answer integration optimization method, retrieval enhancement generation system, equipment and medium.

[0006] The purpose of the present invention can be achieved by the following technical solutions:

[0007] According to a first aspect of the present invention, a multi-knowledge base retrieval and answer integration optimization method is provided, the method comprising:

[0008] The questions input by the user are sent to N knowledge bases respectively, and the embedding algorithm corresponding to each knowledge base is used to convert the questions into question vectors that adapt to the specific semantic space of each knowledge base;

[0009] By calculating the similarity between the question vector and the document vector of each knowledge base, the top M most relevant results in each knowledge base are obtained;

[0010] Integrate the results of all knowledge bases to obtain M*N results;

[0011] The Rerank algorithm is used to re-rank the merged results, and the top m with the highest scores are selected as the final recall results.

[0012] Preferably, before the question input by the user is vectorized, the question input by the user is also preprocessed, and the preprocessing includes removing stop words, punctuation marks and irrelevant characters, and converting them into a standardized text format.

[0013] Preferably, the embedding algorithm is a word embedding algorithm based on a deep learning model.

[0014] Preferably, the word embedding algorithm includes a Word2Vec algorithm and a GloVe algorithm.

[0015] Preferably, for each knowledge base, a large language model is used to perform vector retrieval based on the question vector, and the top M most relevant results are obtained by calculating the similarity between the question vector and the document vector of each knowledge base.

[0016] Preferably, the similarity is cosine similarity.

[0017] Preferably, the adopting of the Rerank algorithm to reorder the merged results specifically comprises: adopting the Rerank algorithm to reorder the merged results based on result relevance, authority and novelty indicators.

[0018] According to a second aspect of the present invention, there is provided a search enhancement generation system, the system comprising an optimization integration module, the optimization integration module adopts the above-mentioned method to perform multi-knowledge base search and answer integration optimization output to generate results.

[0019] According to a third aspect of the present invention, there is provided an electronic device, comprising a memory and a processor, wherein a computer program is stored in the memory, and when the processor executes the program, any one of the methods described above is implemented.

[0020] According to a fourth aspect of the present invention, there is provided a computer-readable storage medium having a computer program stored thereon, wherein when the program is executed by a processor, any one of the methods described above is implemented.

[0021] Compared with the prior art, the present invention has the following beneficial effects:

[0022] 1) Improved comprehensiveness and accuracy of answers: By leveraging information from multiple knowledge bases, more comprehensive and precise answers can be provided, especially when dealing with complex or cross-domain problems.

[0023] 2) Enhanced adaptability and flexibility of the system: Dedicated embedding algorithms for different knowledge bases enable the system to better adapt to various types of data and query requirements.

[0024] 3) Optimized user experience: Through an efficient re-ranking mechanism, users can quickly get the most relevant answers, thereby improving satisfaction. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] Figure 1 The figure is a flow chart of the method of the present invention. DETAILED DESCRIPTION

[0026] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0027] Example 1

[0028] like Figure 1 As shown, this embodiment provides a multi-knowledge base retrieval and answer integration optimization method, the method comprising:

[0029] S1. Send the questions input by the user into N knowledge bases respectively, and use the embedding algorithm corresponding to each knowledge base to convert the questions into question vectors that adapt to the specific semantic space of each knowledge base;

[0030] S2. Obtain the top M most relevant results in each knowledge base by calculating the similarity between the question vector and the document vector of each knowledge base;

[0031] S3, integrate the results of all knowledge bases to obtain M*N results;

[0032] S4. Use the Rerank algorithm to reorder the merged results and select the top m with the highest scores as the final recall results.

[0033] Specifically, in this embodiment, before the question input by the user is vectorized, the question input by the user is also preprocessed, and the preprocessing includes removing stop words, punctuation marks and irrelevant characters, and converting them into a standardized text format.

[0034] The embedding algorithm used in this embodiment is a word embedding algorithm based on a deep learning model, and the word embedding algorithm includes the Word2Vec algorithm and the GloVe algorithm.

[0035] In this embodiment, for each knowledge base, a large language model is used to perform vector retrieval based on the question vector, and the first M most relevant results are obtained by calculating the cosine similarity between the question vector and the document vector of each knowledge base.

[0036] In this embodiment, the Rerank algorithm is used to re-rank the merged results based on the result relevance, authority and novelty indicators.

[0037] This embodiment also provides a search enhancement generation system, which includes an optimization integration module. The optimization integration module uses the above method to perform multi-knowledge base search and answer integration optimization output to generate results.

[0038] The electronic device of the present invention includes a central processing unit (CPU), which can perform various appropriate actions and processes according to computer program instructions stored in a read-only memory (ROM) or loaded from a storage unit into a random access memory (RAM). In the RAM, various programs and data required for device operation can also be stored. The CPU, ROM and RAM are connected to each other via a bus. An input / output (I / O) interface is also connected to the bus.

[0039] Multiple components in the device are connected to the I / O interface, including: input units, such as keyboards, mice, etc.; output units, such as various types of displays, speakers, etc.; storage units, such as disks, optical disks, etc.; and communication units, such as network cards, modems, wireless communication transceivers, etc. The communication unit allows the device to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunication networks.

[0040] The processing unit performs the various methods and processes described above, such as methods S1 to S4. For example, in some embodiments, methods S1 to S4 may be implemented as a computer software program, which is tangibly contained in a machine-readable medium, such as a storage unit. In some embodiments, part or all of the computer program may be loaded and / or installed on the device via a ROM and / or a communication unit. When the computer program is loaded into the RAM and executed by the CPU, one or more steps of methods S1 to S4 described above may be performed. Alternatively, in other embodiments, the CPU may be configured to execute methods S1 to S4 in any other appropriate manner (e.g., by means of firmware).

[0041] The functions described above herein may be performed at least in part by one or more hardware logic components. For example, without limitation, exemplary types of hardware logic components that may be used include: field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chip systems (SOCs), load programmable logic devices (CPLDs), and the like.

[0042] The program code for implementing the method of the present invention can be written in any combination of one or more programming languages. These program codes can be provided to a processor or controller of a general-purpose computer, a special-purpose computer or other programmable data processing device, so that the program code, when executed by the processor or controller, enables the functions / operations specified in the flow chart and / or block diagram to be implemented. The program code can be executed entirely on the machine, partially on the machine, partially on the machine as a stand-alone software package and partially on a remote machine, or entirely on a remote machine or server.

[0043] In the context of the present invention, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0044] The above is only a specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any technician familiar with the technical field can easily think of various equivalent modifications or replacements within the technical scope disclosed by the present invention, and these modifications or replacements should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention shall be based on the protection scope of the claims.

Claims

1. A multi-knowledge base retrieval and answer integration optimization method, characterized in that: The method includes: The questions input by the user are sent to N knowledge bases respectively, and the embedding algorithm corresponding to each knowledge base is used to convert the questions into question vectors that adapt to the specific semantic space of each knowledge base; By calculating the similarity between the question vector and the document vector of each knowledge base, the top M most relevant results in each knowledge base are obtained; Integrate the results of all knowledge bases to obtain M*N results; The Rerank algorithm is used to re-rank the merged results, and the top m with the highest scores are selected as the final recall results.

2. A multi-knowledge base retrieval and answer integration optimization method according to claim 1, characterized in that: Before the question input by the user is vectorized, the question input by the user is preprocessed, and the preprocessing includes removing stop words, punctuation marks and irrelevant characters, and converting them into a standardized text format.

3. A multi-knowledge base retrieval and answer integration optimization method according to claim 1, characterized in that: The embedding algorithm is a word embedding algorithm based on a deep learning model.

4. A multi-knowledge base retrieval and answer integration optimization method according to claim 3, characterized in that: The word embedding algorithms include Word2Vec algorithm and GloVe algorithm.

5. A multi-knowledge base retrieval and answer integration optimization method according to claim 1, characterized in that: For each knowledge base, a large language model is used to perform vector retrieval based on the question vector. The top M most relevant results are obtained by calculating the similarity between the question vector and the document vector of each knowledge base.

6. A multi-knowledge base retrieval and answer integration optimization method according to claim 1, characterized in that: The similarity is cosine similarity.

7. A multi-knowledge base retrieval and answer integration optimization method according to claim 6, characterized in that: The Rerank algorithm is used to re-rank the merged results, specifically: based on the result relevance, authority and novelty indicators, the Rerank algorithm is used to re-rank the merged results.

8. A search enhancement generation system, characterized in that: The system includes an optimization integration module, which uses the method described in any one of claims 1 to 7 to perform multi-knowledge base retrieval and answer integration optimization output to generate results.

9. An electronic device comprising a memory and a processor, wherein a computer program is stored in the memory, wherein: When the processor executes the program, the method according to any one of claims 1 to 7 is implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.

Images