Retrieval Methods for Multi-Type Databases

Abstract

The present application discloses a retrieval method for multiple types of databases, which stores and manages knowledge by using relational databases, tree-type databases, and graph-type databases at the same time. During the retrieval process, the appropriate database is dynamically selected for query according to different retrieval stages and retrieval targets to achieve efficient and accurate knowledge retrieval. Through personalized attribute extraction and preliminary screening, the system can effectively narrow the retrieval scope, and the hypothesis document generation and fusion further enhance the comprehensiveness and depth of the retrieval. Although the experimental group is slightly inferior to the control group in terms of retrieval efficiency, considering the significant improvement in accuracy, this increase in time cost is reasonable. Overall, the experimental results support the effectiveness and practicality of the retrieval method of the present application, and provide strong evidence for the future development of knowledge base management and retrieval technology.

Description

Technical Field

[0001] The present application relates to the technical field of information data processing, and in particular to a retrieval method for multiple types of databases. Background Art

[0002] In modern information technology, the design and implementation of knowledge bases are crucial for information retrieval and knowledge management. Traditional knowledge bases usually use a single type of database, such as a relational database or a graph database. However, with the rapid growth of data types and data volumes, a single type of database is unable to handle complex queries and diverse data.

[0003] Most existing knowledge bases rely on a single type of database. The integration of different types of databases has the following major problems:

[0004] 1. Retrieval efficiency problem: In a large knowledge base, a single type of database may face a retrieval speed bottleneck. For example, a relational database may be inefficient when processing complex queries, while a graph database is more efficient when processing a large number of relational retrievals.

[0005] 2. Data diversity problem: The types of data contained in the knowledge base are very diverse, including structured data, semi-structured data, and unstructured data. A single type of database has limitations and flexibility issues when processing different types of data.

[0006] 3. Scalability issues: As the amount of data continues to grow, the scalability of a single database is often unable to meet demand. A solution combining multiple types of databases may provide a better response strategy in this regard. Summary of the invention

[0007] In view of the deficiencies in the prior art, this application proposes a retrieval method for multiple types of databases.

[0008] The technical solution adopted by the present application to solve the technical problem is to include the following steps:

[0009] Step 1: Establish a mixed type knowledge base: the mixed type knowledge base includes a relational database, a tree type database, and a graph type database;

[0010] The definition of a mixed type knowledge base is a knowledge warehouse that uses multiple types of databases, such as relational, tree, and graph types. Each type of database retrieves different knowledge content at different retrieval stages and for different retrieval targets. Since the backend business system is very complex and involves a variety of databases, when building a knowledge base, you should consider including multiple types of databases rather than using a single type of database. Different types of databases have different functions during vector retrieval. According to the characteristics of their respective functions, they handle different types of retrieval targets at different retrieval stages.

[0011] Step 2: Personalized attribute screening

[0012] When a knowledge retrieval instruction initiated by a user is received, personalized attributes related to the user are screened out from the relational database according to the search keywords or query conditions entered by the user. Based on the extracted personalized attributes and combined with the user's search request, relevant knowledge items are preliminarily screened out to narrow the scope of subsequent searches.

[0013] Specifically, when a user initiates a knowledge search, it is first filtered through the relational database module. The specific process is as follows:

[0014] 1. Receive user search requests: Receive search keywords or query conditions entered by the user.

[0015] 2. Personalized attribute extraction: Filter out personalized attributes related to users from the relational database, including but not limited to the user's historical query records, interests and hobbies, geographic location, etc.

[0016] 3. Preliminary screening: Based on the extracted personalized attributes and combined with the user's search request, preliminarily screen out possible relevant knowledge items. The purpose of this step is to narrow the scope of subsequent searches and improve search efficiency.

[0017] Step 3: Perform precise queries in tree-based and knowledge graph knowledge bases

[0018] Based on the personalized attributes in step 2 and the selected related knowledge items, a more accurate search is performed in the tree-based knowledge base or knowledge graph to obtain question IDs similar to the user's query questions, and these questions are vectorized using a pre-trained natural language model and recorded as keyword similarity vectors.

[0019] Step 4: Generate “fake questions”

[0020] Send the real questions in the knowledge retrieval initiated by the user to the big language model in the background, and the big language model will generate several similar "fake questions";

[0021] Specifically, considering that similar questions queried in the third step may not be accurate enough at the semantic level, in order to improve the matching accuracy, this patent proposes sending the user's questions to the large language model in the background, and the large language model generates many similar "fake questions".

[0022] Step 5: Vectorize “fake questions” and real user questions

[0023] Using the pre-trained natural language model, “fake questions” and real user questions are vectorized to generate corresponding semantic vectors.

[0024] Step 6: Weighted average question vector

[0025] Take the weighted average of the keyword similarity vector generated in step 3, the "fake question" vector in step 5, and the semantic vector of the user's real question. The sum of the weights is 1, among which the user's real question vector has the highest weight, followed by the keyword similarity vector, and the "fake question" vector has the lowest weight.

[0026] Step 7: Fusion Problem Vector

[0027] The user's real question vector, keyword similarity vector, and "false question" vector obtained by weighted averaging in step 6 are superimposed on the channel dimension to generate a fused question vector, and the vectors with missing positions are filled with 0.

[0028] Step 8: Recall the knowledge base vector

[0029] Using the pre-trained natural language model, the similarity between the fused question vector and the full knowledge base vector is calculated, and the N knowledge base vectors closest to the fused question vector are recalled.

[0030] Hypothetical document embeddings

[0031] Hypothetical document embedding refers to the creation of several hypothetical documents in this vector retrieval technology. These hypothetical documents are trained based on the answers to questions generated by a large language model to fill in the gaps in the actual documents and improve the accuracy of retrieval.

[0032] Implementation process:

[0033] 1. Generate hypothetical documents: Generate possible hypothetical documents based on user questions and pre-trained large language models. These hypothetical documents simulate various scenarios that users may be interested in.

[0034] 2. Vectorize hypothetical documents: Use the pre-trained natural language model to vectorize the generated hypothetical documents.

[0035] 3. Fusion of hypothetical document embedding: The vector of the hypothetical document is weighted and fused with the previously generated keyword similarity vector, false question vector, and user real question vector to form a more accurate embedding vector.

[0036] Step 9: Calculate similarity and sort

[0037] Using the pre-trained natural language model, the N knowledge base vectors mentioned above are calculated for similarity with the keyword similarity vectors in step three, sorted from high to low, and the top three vectors with the highest similarity scores are output.

[0038] Step 10: Output the final knowledge results

[0039] Query the database for the question ID corresponding to the vector obtained in the previous step, and return the knowledge result corresponding to the question to the user.

[0040] The beneficial effect of the present application is that: a retrieval method for multiple types of databases is proposed, which stores and manages knowledge by using relational databases, tree-type databases and graph-type databases at the same time. During the retrieval process, the appropriate database is dynamically selected for query according to different retrieval stages and retrieval targets to achieve efficient and accurate knowledge retrieval. DETAILED DESCRIPTION

[0041] The technical solutions in the embodiments of the present application are described clearly and completely below. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in the field without creative work are within the scope of protection of the present application. In addition, the technical solutions between the various embodiments can be combined with each other, but must be based on the ability of ordinary technicians in the field to implement. When the combination of technical solutions is contradictory or cannot be implemented, it should be considered that the combination of such technical solutions does not exist and is not within the scope of protection of the present application.

[0042] Specific implementation method of the present application: a retrieval method for multiple types of databases.

[0043] The technical solution adopted by the present application to solve the technical problem is to include the following steps:

[0044] Step 1: Establish a mixed type knowledge base: the mixed type knowledge base includes a relational database, a tree type database, and a graph type database;

[0045] The definition of a mixed type knowledge base is a knowledge warehouse that uses multiple types of databases, such as relational, tree, and graph types. Each type of database retrieves different knowledge content at different retrieval stages and for different retrieval targets. Since the backend business system is very complex and involves a variety of databases, when building a knowledge base, you should consider including multiple types of databases rather than using a single type of database. Different types of databases have different functions during vector retrieval. According to the characteristics of their respective functions, they handle different types of retrieval targets at different retrieval stages.

[0046] Step 2: Personalized attribute screening

[0047] When a knowledge retrieval instruction initiated by a user is received, personalized attributes related to the user are screened out from the relational database according to the search keywords or query conditions entered by the user. Based on the extracted personalized attributes and combined with the user's search request, relevant knowledge items are preliminarily screened out to narrow the scope of subsequent searches.

[0048] Specifically, when a user initiates a knowledge search, it is first filtered through the relational database module. The specific process is as follows:

[0049] 1. Receive user search requests: Receive search keywords or query conditions entered by the user.

[0050] 2. Personalized attribute extraction: Filter out personalized attributes related to users from the relational database, including but not limited to the user's historical query records, interests and hobbies, geographic location, etc.

[0051] 3. Preliminary screening: Based on the extracted personalized attributes and combined with the user's search request, preliminarily screen out possible relevant knowledge items. The purpose of this step is to narrow the scope of subsequent searches and improve search efficiency.

[0052] Step 3: Perform precise queries in tree-based and knowledge graph knowledge bases

[0053] Based on the personalized attributes in step 2 and the selected related knowledge items, a more accurate search is performed in the tree-based knowledge base or knowledge graph to obtain question IDs similar to the user's query questions, and these questions are vectorized using a pre-trained natural language model and recorded as keyword similarity vectors.

[0054] Step 4: Generate “fake questions”

[0055] Send the real questions in the knowledge retrieval initiated by the user to the big language model in the background, and the big language model will generate several similar "fake questions";

[0056] Specifically, considering that similar questions queried in the third step may not be accurate enough at the semantic level, in order to improve the matching accuracy, this patent proposes sending the user's questions to the large language model in the background, and the large language model generates many similar "fake questions".

[0057] Step 5: Vectorize “fake questions” and real user questions

[0058] Using the pre-trained natural language model, “fake questions” and real user questions are vectorized to generate corresponding semantic vectors.

[0059] Step 6: Weighted average question vector

[0060] Take the weighted average of the keyword similarity vector generated in step 3, the "fake question" vector in step 5, and the semantic vector of the user's real question. The sum of the weights is 1, among which the user's real question vector has the highest weight, followed by the keyword similarity vector, and the "fake question" vector has the lowest weight.

[0061] Step 7: Fusion Problem Vector

[0062] The user's real question vector, keyword similarity vector, and "false question" vector obtained by weighted averaging in step 6 are superimposed on the channel dimension to generate a fused question vector, and the vectors with missing positions are filled with 0.

[0063] Step 8: Recall the knowledge base vector

[0064] Using the pre-trained natural language model, the fused question vector is similar to the full knowledge base vector, and the N knowledge base vectors closest to the fused question vector are recalled. The full knowledge base is a database management system specifically used to store and retrieve vector data. It is often used to process high-dimensional data such as embedded representations of images, audio, and text. Unlike traditional databases, vector databases do not rely on exact matching queries, but retrieve data by calculating the similarity between vectors. This makes them very suitable for applications such as recommendation systems and natural language processing. Vector databases use algorithms such as cosine similarity or Euclidean distance to compare vectors and quickly find the most similar items. The "full knowledge base vector" referred to in this application refers to a set (or multiple sets) of vector representations obtained by vectorizing all textual knowledge data in the full knowledge base (e.g., FAQ text, dishes and disease description documents, etc.) through a pre-trained natural language model. These vectors can characterize the semantic or content characteristics of each entry in the knowledge base, thereby providing a basis for subsequent similarity calculation, retrieval, and reasoning.

[0065] Hypothetical document embeddings

[0066] Hypothetical document embedding refers to the creation of several hypothetical documents in this vector retrieval technology. These hypothetical documents are trained based on the answers to questions generated by a large language model to fill in the gaps in the actual documents and improve the accuracy of retrieval.

[0067] Implementation process:

[0068] 1. Generate hypothetical documents: Generate possible hypothetical documents based on user questions and pre-trained large language models. These hypothetical documents simulate various scenarios that users may be interested in.

[0069] 2. Vectorize hypothetical documents: Use the pre-trained natural language model to vectorize the generated hypothetical documents.

[0070] 3. Fusion of hypothetical document embedding: The vector of the hypothetical document is weighted and fused with the previously generated keyword similarity vector, false question vector, and user real question vector to form a more accurate embedding vector.

[0071] Specifically:

[0072] 1. Use traditional SQL matching technology to find questions similar to the user's questions in the knowledge base, and use the pre-trained natural language model to vectorize them and record them as keyword similarity vectors Vkey.

[0073] 2. The user's question is sent to the large language model in the background, which imitates the question raised by the user and generates many similar "fake questions".

[0074] 3. Use the pre-trained natural language model to vectorize the “fake questions” and real user questions, denoted as Vfake and Vreal

[0075] 4. Weighted average of the keyword similarity vector, the "fake question" vector in step 5, and the vector of the user's real question. The sum of all weights is 1. The user's real question vector w1 has the highest weight, the keyword similarity vector w2 is the second, and the "fake question" vector w3 has the lowest weight. That is, Vmix = w1 * Vreal + w2 * Vkey + w3 * Vfake

[0076] In order to improve the final matching question similarity, this embodiment proposes a method of mixing three question vectors to improve the mapping effect of the semantic vector space. Among them, the user's real question vector weight w1 is the highest, accounting for 50%. The keyword similarity vector weight w2 is second, accounting for 30%. The weight w3 of the "fake question" vector is the lowest, accounting for 20%. The final vector is recorded as the fused question vector Vmix.

[0077] Step 9: Calculate similarity and sort

[0078] Using the pre-trained natural language model, the N knowledge base vectors mentioned above are calculated for similarity with the keyword similarity vectors in step three, sorted from high to low, and the top three vectors with the highest similarity scores are output.

[0079] Step 10: Output the final knowledge results

[0080] Query the database for the question ID corresponding to the vector obtained in the previous step, and return the knowledge result corresponding to the question to the user.

[0081] In order to verify the effectiveness of the proposed multi-type database hybrid knowledge base and its retrieval method, the applicant designed a series of experiments to evaluate the performance of the system in different scenarios. The following will introduce the experimental settings, data sets, evaluation indicators and experimental results in detail.

[0082] In this embodiment, the pre-trained language model uses the BERT pre-training model technology.

[0083] As an example of generating fake questions related to user questions, the following can be seen:

[0084] Question: Doctor, my blood pressure has been abnormal recently. How should I treat and maintain it? Please refer to the above question and generate 3 similar questions. The semantics must be accurate.

[0085] Reply: 1. My blood pressure measured recently is a little high. How should I adjust my diet and lifestyle habits to control my blood pressure?

[0086] 2. I find that my blood pressure is often unstable. What methods are there to help me stabilize my blood pressure and manage it effectively on a daily basis?

[0087] 3. I have noticed that my blood pressure has been high recently. What lifestyle adjustments and medical measures can help me lower and maintain normal blood pressure?

[0088] Continue asking questions: Please refer to the above questions and generate 3-5 similar questions. The semantics must be accurate and the content must be clear.

[0089] reply:

[0090] Of course, there are several similar ways of expressing the problem:

[0091] 1. I have been having headaches and irregular heartbeats recently. What should I do?

[0092] 2. I have recently had symptoms of headache and abnormal heartbeat. How should I seek medical treatment and take care of myself?

[0093] 3. Please tell me, if I experience headaches and a fast heartbeat, how can I get diagnosed and treated?

[0094] 4. I have a headache and an abnormal heartbeat. How should I get the correct treatment and maintenance?

[0095] 5. I have been having headaches and irregular heartbeats lately. What should I do?

[0096] These questions clearly express your symptoms and ask about relevant treatment and maintenance methods. I hope this helps you!

[0097] VI. Experimental Setup

[0098] Dataset

[0099] 1. Relational database: Contains 1 million pieces of user personal information, including historical query records, interests, hobbies, geographic locations, etc., for personalized attribute extraction.

[0100] 2. Tree-type knowledge base: contains 500,000 professional knowledge items, covering computer science, medicine, physics and other fields.

[0101] 3. Knowledge Graph: A knowledge network consisting of 2 million entity nodes and 3 million relationship edges was constructed, covering encyclopedic information.

[0102] 4. Document database: Contains 100,000 academic paper abstracts, which are used to generate hypothesis documents.

[0103] Evaluation Metrics

[0104] 1. Retrieval accuracy: measures the degree of match between retrieval results and user needs.

[0105] 2. Retrieval efficiency: evaluates the time consumption of the retrieval process.

[0106] 3. User satisfaction: Collect users’ subjective evaluation of the quality of search results through questionnaire surveys.

[0107] Experimental Results

[0108] Retrieval accuracy experiment

[0109] Experimental group: The retrieval method of the multi-type database of the present application was adopted.

[0110] Control group: Traditional search method using a single type of database.

[0111]

[0112] Retrieval efficiency experiment

[0113]

[0114] User satisfaction experiment

[0115]

[0116] The experimental results show that the multi-type database hybrid knowledge base and its retrieval method show significant advantages in retrieval accuracy and user satisfaction. Through personalized attribute extraction and preliminary screening, the system can effectively narrow the scope of retrieval, while the hypothesis document generation and fusion further improve the comprehensiveness and depth of retrieval. Although the experimental group is slightly inferior to the control group in terms of retrieval efficiency, considering the significant improvement in accuracy, this increase in time cost is reasonable. Overall, the experimental results support the effectiveness and practicality of the retrieval method of this application, and provide strong evidence for the future development of knowledge base management and retrieval technology.

[0117] So far, various embodiments of the present application have been described in detail. In order to avoid obscuring the concept of the present application, some details known in the art are not described. Based on the above description, those skilled in the art can fully understand how to implement the technical solution disclosed herein.

[0118] Finally, it should be noted that the above is only a preferred embodiment of the present application, and the aforementioned embodiments are only used to illustrate the technical solution of the present application, rather than to limit it. Although the present application has been described in detail with reference to the aforementioned embodiments, a person of ordinary skill in the art should understand that it is still possible to modify the technical solutions recorded in the aforementioned embodiments, or to make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not drive the essence of the corresponding technical solutions away from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A retrieval method for a multi-type database, characterized in that: The following steps are involved: Step 1: Establish a mixed type knowledge base: the mixed type knowledge base includes a relational database, a tree type database, and a graph type database; Step 2: When a knowledge search instruction initiated by a user is received, personalized attributes related to the user are screened out from the relational database according to the search keywords or query conditions input by the user. Based on the extracted personalized attributes and the user's search request, relevant knowledge items are preliminarily screened out to narrow the scope of subsequent searches. Step 3: Based on the personalized attributes in the above step 2 and the selected related knowledge items, further accurate search is performed in the tree type database and / or graph type database to obtain question IDs similar to the real questions in the knowledge retrieval initiated by the user, and these questions are vectorized using the pre-trained natural language model to generate keyword similarity vectors; Step 4: Send the real question in the knowledge search initiated by the user to the big language model in the background, and the big language model will generate several similar "fake questions"; Step 5: Use the pre-trained natural language model to vectorize the "fake questions" and real user questions to generate corresponding semantic vectors; Step 6: Perform weighted averaging on the keyword similarity vector generated in step 3 and the semantic vector generated in step 5, with the sum of the weights being 1; the user's real question vector has the highest weight, followed by the keyword similarity vector, and the "fake question" vector has the lowest weight; Step 7: Superimpose the user's real question vector, keyword similarity vector, and "false question" vector after weighted average in step 6 in channel dimension to generate a fused question vector, and fill the vectors with zeros in the missing positions; Step 8: Using the pre-trained natural language model, calculate the similarity between the fused question vector and the full knowledge base vector, and recall the N knowledge base vectors that are closest to the fused question vector; Step 9: Using the pre-trained natural language model, calculate the similarity between the N knowledge base vectors and the keyword similarity vectors in step 3, sort them from high to low, and output the top three vectors with the highest similarity scores; Step 10: Query the database for the question ID corresponding to the vector obtained in the previous step, and return the knowledge result corresponding to the question to the user.

2. The retrieval method of a multi-type database according to claim 1, characterized in that: The personalized attributes in step 2 include the user's historical query records, interests and hobbies, and geographic location.

3. The retrieval method of a multi-type database according to claim 1, characterized in that: The step eight also includes hypothetical document embedding, which includes the following steps: S1, generating possible hypothetical documents based on user questions and a pre-trained large language model, and these hypothetical documents simulate various scenarios that the user may be interested in; S2, using the pre-trained natural language model to vectorize the generated hypothetical documents; S3, weightedly fusing the vector of the hypothetical document with the previously generated keyword similarity vector, false question vector, and user real question vector to form a more accurate embedding vector.

Images