Method for improving recall rate of vector database by using text extension training

Abstract

The invention discloses a method for improving the recall rate of a vector database by using text extension training. The method comprises the following steps: S1, generating and extending questions; s2, performing similarity evaluation and problem screening; s3, linking and embedding questions and answers; s4, constructing and optimizing a vector database; and S5, performing real-time retrieval. In the preprocessing stage, not only can a plurality of similar problems be generated when the original problems are expanded, but also the similarity of each generated problem is evaluated by introducing a text neighbor algorithm review mechanism. Only questions passing the similarity evaluation criteria can be linked with original answers and embedded into a vector database. And during query, the problem of the user is directly matched with the optimally stored data, so that the load in a real-time query stage is reduced, and the overall efficiency of the system is improved.

Description

Technical Field

[0001] The present invention belongs to the technical field of data processing, and in particular relates to a method for improving the recall rate of a vector database by using text extension training. Background Art

[0002] In the field of Natural Language Processing (NLP), the development of Question Answering (QA) systems has always been an active research field, and its core challenge is how to achieve fast and accurate information retrieval. In traditional QA systems, document preprocessing is a key step. Word embedding technology is usually used to convert QA pairs into vector form and store them in a vector database for retrieval. In the query processing stage, in order to improve the recall rate, traditional technologies use a question expansion strategy, that is, expanding a single question into multiple related questions, querying them separately, sorting and merging the query results, and finally returning a comprehensive result.

[0003] In the existing technology, the mainstream method is to use the RAG-fusion (Retrieval-Augmented Generation with Fusion) architecture to expand retrieval. RAG-Fusion is an improved retrieval augmentation generation model that improves the quality and depth of text generated by large language models (LLM) by introducing multi-query generation and result re-ranking. Figure 1As shown in the figure, the main steps of RAG-Fusion include: S1, question reception, the user sends a question request, and the model receives this question as input; S2, question expansion, the model expands the original question into multiple related questions; S3, vector database retrieval, retrieve relevant documents or information in the vector database; S4, result ranking fusion: sort and fuse the retrieval results; S5, generate answers: use the sorted and fused information to generate answers to the original questions and provide them to the user. RAG-Fusion improves the quality and efficiency of the question answering system through these steps, especially in scenarios where complex queries need to be processed and deep text needs to be generated. However, this method has some obvious limitations: (1) High computing cost: RAG-Fusion requires a lot of computing resources in the process of question expansion and result fusion, resulting in high computing cost; (2) Long query time: RAG-Fusion prolongs the processing time in the process of converting a single question into multiple related questions, which affects the response speed of the system and causes the query time to be too long; (3) "Hallucination" problem: RAG-Fusion may cause the generated questions to have a lower relevance to the original questions during question expansion. This phenomenon is called "hallucination". The "hallucination" defect seriously reduces the quality of the answer.

[0004] Due to the above problems, the application scenarios of the QA system based on RAG-Fusion are limited, and it is difficult to achieve efficient query and real-time feedback. Therefore, it is urgent to provide a method for improving the recall rate of the vector database by using text expansion training to solve the shortcomings of the above existing technologies. Summary of the invention

[0005] In response to the problems in the related art, the present invention provides a method for improving the recall rate of the vector database by using text expansion training, so as to improve the speed and accuracy of vector database retrieval. In the preprocessing stage, this method can not only generate multiple similar questions when expanding the original question, but also introduce a text neighbor algorithm review mechanism to evaluate the similarity of each generated question. Only questions that pass the similarity evaluation criteria can be linked to the original answer and embedded in the vector database. When querying, the user's question will be directly matched with the optimized stored data, thereby reducing the load of the real-time query stage and improving the overall efficiency of the system.

[0006] The technical solution of the present invention is implemented as follows: a method for improving the recall rate of a vector database by using text expansion training, comprising the following steps:

[0007] S1. Question generation and expansion: Generate the original question and the original answer, and expand the original question to create multiple semantically similar questions;

[0008] S2. Similarity evaluation and question screening: Use the text nearest neighbor algorithm to evaluate similar questions, so that the expanded similar questions have a high degree of similarity with the original questions; by setting a similarity threshold, similar questions with insufficient similarity are eliminated;

[0009] S3. Question-answer linking and embedding: Link similar questions that have passed similarity evaluation with the original answers, and use deep learning algorithms to embed them to form preprocessed question-answer pairs;

[0010] S4. Construction and optimization of vector database: vectorize and store preprocessed question-answer pairs; optimize the storage structure to support efficient retrieval of a large number of question-answer pairs;

[0011] S5. Real-time retrieval: Use retrieval algorithms to quickly match user questions to preprocessed question-answer pairs in the vector database and return the most relevant answers.

[0012] In the preprocessing stage, the present invention can not only generate multiple similar questions when expanding the original question, but also introduce a text neighbor algorithm review mechanism to evaluate the similarity of each generated question. Only questions that pass the similarity evaluation criteria can be linked to the original answer and embedded in the vector database. When querying, the user's question will be directly matched with the optimized stored data, thereby reducing the load of the real-time query stage and improving the overall efficiency of the system.

[0013] As a further improvement of the above scheme, S1 includes the following steps:

[0014] S11. Clean and load the knowledge base to obtain preprocessed documents;

[0015] S12. Read the original question-original answer pair from the preprocessed document: Q 原 -A 原 ;

[0016] S13. Use NLP technology to solve the original question Q 原 Perform text expansion to generate multiple semantically similar questions: Q1, Q2...Q n ; Where n is a positive integer.

[0017] As a further improvement of the above scheme, S2 includes the following steps:

[0018] S21. Use vectorization tools to solve the original problem Q 原 And multiple similar questions Q1, Q2...Q n Perform vectorization processing to obtain the original problem vector q 原 and similar problem vectors q1,q2...q n ;

[0019] S22. Calculate each similar problem vector q1, q2...q n With the original problem vector q 原 The similarity angle of θ1, θ2, ... θ n ;

[0020] S23. Set similarity angle threshold θ 阈值 ;

[0021] S24. Compare similarity angles θ1, θ2…θ n The similarity angle threshold θ 阈值 The numerical value will make the similarity angle greater than θ 阈值 Corresponding similar questions are eliminated.

[0022] As a further improvement of the above solution, the vectorization tool is Word2Vec, GloVe, fastText, ELMo, BERT or TF-IDF.

[0023] As a further improvement of the above scheme, S3 includes the following processes:

[0024] The similarity angle is greater than θ 阈值 After the corresponding similar questions are removed, the remaining similar questions are the similar questions that have passed the similarity evaluation; the remaining similar questions are pointed to the same original answer, and the question-answer pairs are reconstructed to form the preprocessed question-answer pairs.

[0025] As a further improvement of the above solution, S4 includes the following steps:

[0026] S41. Divide the preprocessed question-answer pairs into blocks and vectorize and store them to form a vector model;

[0027] S42. Train the vector model to form a vector database.

[0028] As a further improvement of the above solution, in the block segmentation process of S41, the text cutting tool adopts RecursiveCharacterTextSplitter.

[0029] As a further improvement of the above solution, S5 includes the following steps:

[0030] S51. The user issues a target question, and the question-answering system receives the target question;

[0031] S52. The question-answering system vectorizes the target question to form a target vector model;

[0032] S53. Perform a similarity query on the target vector model in the vector database and sort the query results;

[0033] S54. Generate an answer to the target question using the ranking information and return it to the user.

[0034] As a further improvement of the above solution, S53 includes the following steps:

[0035] S531. For each vector in the vector database, calculate its similarity with the target vector model;

[0036] S532. Sort the vectors according to similarity, from the highest similarity to the lowest similarity;

[0037] S533. Select K vectors with the highest similarity from the sorted list.

[0038] As a further improvement of the above solution, S54 includes the following process: the question-answering system injects background knowledge into the sorting information, constructs a prompt word template, inputs it into the large language model, and the large model generates an answer to the target question and returns it to the user.

[0039] Compared with the traditional processing method, the present invention has the following advantages:

[0040] (1) Effective question review: The text nearest neighbor algorithm review mechanism is used to pre-screen high-quality question variants to ensure the similarity of questions, thereby improving the recall rate while ensuring accuracy.

[0041] (2) Resource optimization: In traditional Rag fusion technology, each query needs to be expanded into multiple questions, and then multiple queries are used to obtain answers and sort. This process not only increases the complexity of the calculation, but also prolongs the query response time. In contrast, the present invention completes all necessary computationally intensive tasks in the preprocessing stage, so that the pre-processed data can be directly called during the actual query, significantly improving the query efficiency.

[0042] (3) Reduced query latency: The present invention greatly simplifies the query process by directly retrieving pre-embedded question-answer pairs. This approach not only reduces the computational burden of the system during real-time queries, but also significantly shortens the user's waiting time. Compared with traditional Rag fusion technology, our solution only requires one calculation in the embedding stage and can achieve fast response in the query stage, which is particularly advantageous when processing a large number of queries.

[0043] (4) Efficient use of storage space: Although this method requires more storage space, due to the pre-screening mechanism, the stored data is more relevant, making the retrieval process faster and the storage space is used effectively. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] Figure 1 The retrieval architecture diagram of RAG-Fusion in the prior art;

[0045] Figure 2 A diagram of the search architecture of the present invention;

[0046] Figure 3 It is a flowchart of the steps of the present invention. DETAILED DESCRIPTION

[0047] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0048] Example

[0049] like Figure 2 , Figure 3 As shown, a method for improving the recall rate of a vector database by using text expansion training includes the following steps:

[0050] S1. Question generation and expansion: Generate the original question and the original answer, and expand the original question to create multiple semantically similar questions; specifically, S1 includes the following steps:

[0051] S11. Clean and load the knowledge base to obtain preprocessed documents;

[0052] S12. Read the original question-original answer pair from the preprocessed document: Q 原 -A 原 ;

[0053] For example: Original question: Who are you?; Original answer: I am Xiao Ming.

[0054] S13. Use NLP technology to solve the original question Q 原 Perform text expansion to generate multiple semantically similar questions: Q1, Q2...Q n ; Where n is a positive integer. For example: Who are you? / What is your name? / ... / What is your name?

[0055] S2. Similarity evaluation and question screening: Use the text nearest neighbor algorithm to evaluate similar questions so that the expanded similar questions have a high degree of similarity with the original questions; by setting a similarity threshold, similar questions with insufficient similarity are eliminated; in this embodiment, S2 includes the following steps:

[0056] S21. Use vectorization tools to solve the original problem Q 原 And multiple similar questions Q1, Q2...Q n Perform vectorization processing to obtain the original problem vector q 原 and similar problem vectors q1,q2...q n ; Specifically, the vectorization tool is Word2Vec, GloVe, fastText, ELMo, BERT or TF-IDF. In this embodiment, Word2Vec is preferably used.

[0057] S22. Calculate each similar problem vector q1, q2...q n With the original problem vector q 原 The similarity angle of θ1, θ2, ... θ n ;

[0058] S23. Set similarity angle threshold θ 阈值 ;

[0059] S24. Compare similarity angles θ1, θ2…θ n The similarity angle threshold θ 阈值 The numerical value will make the similarity angle greater than θ 阈值 Corresponding similar questions are eliminated.

[0060] It should be noted that this embodiment uses the method of calculating the cosine angle to measure the similarity between two vectors. The smaller the angle between the two vectors, the larger the cosine value of the angle, which means that the two vectors are more similar, thereby eliminating similar question vectors with different semantics. The text neighbor algorithm review mechanism pre-screens high-quality question variants to ensure the similarity of the questions, thereby improving the recall rate while ensuring accuracy. In other embodiments, methods such as Euclidean distance and Manhattan distance can also be used for similarity assessment.

[0061] S3. Question-answer linking and embedding: Link similar questions that have passed similarity evaluation with the original answers, and use deep learning algorithms to embed them to form preprocessed question-answer pairs; more specifically, 阈值 After the corresponding similar questions are removed, the remaining similar questions are the similar questions that have passed the similarity evaluation; the remaining similar questions are pointed to the same original answer, and the question-answer pair is reconstructed to form the preprocessed question-answer pair. For example: if only the similarity angle of Q1 is greater than θ 阈值 , then the remaining similar questions point to the same original answer, i.e. Q2->A 原 , Q3->A 原 ...Q n ->A 原. In traditional Rag fusion technology, each query needs to be expanded into multiple questions, and then multiple queries are used to obtain answers and sort. This process not only increases the complexity of calculations, but also prolongs the query response time. In contrast, the present invention significantly improves query efficiency by completing all necessary computationally intensive tasks in the preprocessing stage, so that pre-processed data can be directly called during actual queries.

[0062] S4. Construction and optimization of vector database: vectorize and store preprocessed question-answer pairs; optimize the storage structure to support effective retrieval of a large number of question-answer pairs; specifically, S4 includes the following steps:

[0063] S41. Divide the preprocessed question-answer pairs into blocks and store them in vector form to form a vector model; in the block division process, the text cutting tool adopts RecursiveCharacterTextSplitter.

[0064] S42. Train the vector model to form a vector database.

[0065] S5. Real-time retrieval: Use the retrieval algorithm to quickly match the questions raised by the user to the pre-processed question-answer pairs in the vector database, and return the most relevant answers. This embodiment greatly simplifies the query process by directly retrieving the pre-embedded question-answer pairs. This method not only reduces the computational burden of the system during real-time queries, but also significantly shortens the user's waiting time. Compared with traditional Rag fusion technology, our solution only requires one calculation in the embedding stage, and can achieve fast response in the query stage, which is particularly advantageous when processing a large number of queries.

[0066] Specifically, S5 includes the following steps:

[0067] S51. The user issues a target question, and the question-answering system receives the target question;

[0068] S52. The question-answering system vectorizes the target question to form a target vector model;

[0069] S53. Perform a similarity query on the target vector model in the vector database and sort the query results; specifically, S53 includes the following steps:

[0070] S531. For each vector in the vector database, calculate its similarity with the target vector model;

[0071] S532. Sort the vectors according to similarity, from the highest similarity to the lowest similarity;

[0072] S533. Select K vectors with the highest similarity from the sorted list.

[0073] S54. Generate an answer to the target question using the ranking information and return it to the user. Specifically, the question-answering system injects background knowledge into the ranking information, constructs a prompt word template, inputs it into the large language model, and the large model generates an answer to the target question and returns it to the user.

[0074] Through the above solution of the present invention, in specific applications: in the preprocessing stage, not only can multiple similar questions be generated when expanding the original question, but also a text neighbor algorithm review mechanism is introduced to evaluate the similarity of each generated question. Only questions that pass the similarity evaluation criteria can be linked to the original answer and embedded in the vector database. When querying, the user's question will be directly matched with the optimized stored data, thereby reducing the load of the real-time query stage and improving the overall efficiency of the system.

[0075] The present invention converts storage space into computing time, and has expanded similar questions in advance in the preprocessing stage. The expansion process is a one-time expansion, and then the query is completed directly at one time when the user recalls the query, without the need for additional expansion. Compared with the traditional RAG-Fusion retrieval process, each time the query is recalled, the expansion is done, and the results are recalled and sorted, which obviously takes less time, and the query is faster and more efficient. Although this method requires more storage space, due to the role of the pre-screening mechanism, the stored data is more relevant, which makes the retrieval process faster and the storage space is effectively used.

[0076] According to the disclosure and teaching of the above description, those skilled in the art to which the present invention belongs may also make changes and modifications to the above embodiments. Therefore, the present invention is not limited to the specific embodiments disclosed and described above, and some modifications and changes to the invention should also fall within the scope of protection of the claims of the present invention. In addition, although some specific terms are used in this specification, these terms are only for the convenience of description and do not constitute any limitation to the present invention.

Claims

1. A method for improving the recall rate of a vector database by using text expansion training, characterized in that: The following steps are involved: S1. Question generation and expansion: Generate the original question and the original answer, and expand the original question to create multiple semantically similar questions; S2. Similarity evaluation and question screening: Use the text nearest neighbor algorithm to evaluate similar questions, so that the expanded similar questions have a high degree of similarity with the original questions; by setting a similarity threshold, similar questions with insufficient similarity are eliminated; S3. Question-answer linking and embedding: Link similar questions that have passed similarity evaluation with the original answers, and use deep learning algorithms to embed them to form preprocessed question-answer pairs; S4. Construction and optimization of vector database: vectorize and store preprocessed question-answer pairs to support efficient retrieval of a large number of question-answer pairs; S5. Real-time retrieval: Use retrieval algorithms to quickly match user questions to preprocessed question-answer pairs in the vector database and return the most relevant answers.

2. The method for improving the recall rate of a vector database by using text extension training according to claim 1, characterized in that: S1 includes the following steps: S11. Clean and load the knowledge base to obtain preprocessed documents; S12. Read the original question-original answer pair from the preprocessed document: Q 原 -A 原 ; S13. Use NLP technology to solve the original question Q 原 Perform text expansion to generate multiple semantically similar questions: Q1, Q2...Q n ; Where n is a positive integer.

3. The method for improving the recall rate of a vector database by using text expansion training according to claim 2, characterized in that: S2 includes the following steps: S21. Use vectorization tools to solve the original problem Q 原 And multiple similar questions Q1, Q2...Q n Perform vectorization processing to obtain the original problem vector q 原 and similar problem vectors q1,q2...q n ; S22. Calculate each similar problem vector q1, q2...q n With the original problem vector q 原 The similarity angle of θ1, θ2, ... θ n ; S23. Set similarity angle threshold θ 阈值 ; S24. Compare similarity angles θ1, θ2…θ n The similarity angle threshold θ 阈值 The numerical value will make the similarity angle greater than θ 阈值 Corresponding similar questions are eliminated.

4. The method for improving the recall rate of a vector database by using text expansion training according to claim 3, characterized in that: The vectorization tool is Word2Vec, GloVe, fastText, ELMo, BERT or TF-IDF.

5. The method for improving the recall rate of a vector database by using text expansion training according to claim 4, characterized in that: S3 includes the following processes: The similarity angle is greater than θ 阈值 After the corresponding similar questions are removed, the remaining similar questions are the similar questions that have passed the similarity evaluation; the remaining similar questions are pointed to the same original answer, and the question-answer pairs are reconstructed to form the preprocessed question-answer pairs.

6. The method for improving the recall rate of a vector database by using text expansion training according to claim 5, characterized in that: S4 includes the following steps: S41. Divide the preprocessed question-answer pairs into blocks and vectorize and store them to form a vector model; S42. Train the vector model to form a vector database.

7. The method of improving the recall rate of a vector database by using text extension training according to claim 6, characterized in that: In the block segmentation process of S41, the text cutting tool adopts RecursiveCharacterTextSplitter.

8. The method for improving the recall rate of a vector database by using text expansion training according to claim 1, characterized in that: S5 includes the following steps: S51. The user issues a target question, and the question-answering system receives the target question; S52. The question-answering system vectorizes the target question to form a target vector model; S53. Perform a similarity query on the target vector model in the vector database and sort the query results; S54. Generate an answer to the target question using the ranking information and return it to the user.

9. The method for improving the recall rate of a vector database by using text expansion training according to claim 8, characterized in that: S53 includes the following steps: S531. For each vector in the vector database, calculate its similarity with the target vector model; S532. Sort the vectors according to similarity, from the highest similarity to the lowest similarity; S533. Select K vectors with the highest similarity from the sorted list.

10. The method for improving the recall rate of a vector database by using text extension training according to claim 9, characterized in that: S54 includes the following processes: The question-answering system injects background knowledge into the sorting information, builds a prompt word template, inputs it into the large language model, and the large model generates an answer to the target question and returns it to the user.

Images