Knowledge base file uploading consistency guaranteeing method and device based on RAG framework

By employing locking, segmentation and comparison, and database storage in the knowledge base file upload method of the RAG framework, the consistency problem of knowledge base file upload in the RAG framework is solved, ensuring data integrity and consistency, and improving the output accuracy of large language models.

CN119005163APending Publication Date: 2024-11-22ROCK AI
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202410879931.7
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-07-02
Publication Date
2024-11-22

AI Technical Summary

Technical Problem

In the RAG framework, it is difficult to guarantee the integrity and consistency of data during the file upload process of the enterprise private knowledge base, especially in the case of rapid updates. Information may be delayed or inconsistent, which will affect the accuracy of subsequent training of large language models and data retrieval.

Method used

By locking, splitting, and concatenating knowledge base documents, and combining keyword and vector database storage, real-time checks and periodic compensation are performed to ensure the consistency of file content, and retransmission and repair are performed in case of storage failure.

Benefits of technology

This ensures that the file content remains consistent with the original content, avoids information contradictions, improves the accuracy of the output results of the large language model, and ensures the security and integrity of the data.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119005163A_ABST
    Figure CN119005163A_ABST
Patent Text Reader

Abstract

The invention provides a knowledge base file uploading consistency guaranteeing method and device based on an RAG framework. The problem that the knowledge base file uploading consistency of the RAG framework is difficult to guarantee is solved. By guaranteeing the uploading consistency of the knowledge base file based on the RAG framework, it is ensured that the file content uploaded to the database is kept consistent with the original file content, misleading caused by information contradiction is avoided, the data accuracy of a follow-up large language model during training and data retrieval is guaranteed, and the data retrieval efficiency is improved. And the accuracy of the output result of the large language model is improved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of large language models, and in particular to a method and device for ensuring consistency of knowledge base file upload based on a RAG framework. Background Art

[0002] With the development of artificial intelligence technology, especially the application of large models in the field of knowledge retrieval and generation, enterprises are increasingly paying attention to using these technologies to improve the management and utilization efficiency of internal knowledge. The RAG (Retrieval-Augmented Generation) model is an ideal choice for building an intelligent knowledge base because of its ability to combine retrieval and generation, can efficiently retrieve knowledge in large-scale data sets, and supports real-time updates.

[0003] However, there are several challenges in the actual application of the RAG model, especially the security and data consistency issues of the enterprise's private knowledge base. Private knowledge bases often contain a large amount of sensitive information, such as proprietary technical documents, customer data, contract documents, etc. When uploading these data to the cloud or external knowledge base, the integrity and confidentiality of the data must be ensured. At the same time, with the rapid update of knowledge, how to ensure that the newly uploaded or updated file content can be integrated into the RAG system in a timely and accurate manner to avoid information lag or inconsistency has become a key issue. Summary of the invention

[0004] The present invention provides a method and device for ensuring consistency of knowledge base file upload based on a RAG framework, so as to solve the problem that the consistency of knowledge base file upload based on the RAG framework is difficult to ensure.

[0005] In a first aspect, the present invention provides a knowledge base file upload consistency assurance method based on a RAG framework, which specifically comprises the following steps:

[0006] Step S1, upload the knowledge base document T1 to the database;

[0007] Step S2: Check all knowledge base documents uploaded in the database within A hours.

[0008] Wherein, A is a positive integer.

[0009] Preferably, in step S1, uploading the knowledge base document T1 to the database specifically includes the following steps:

[0010] Step S101, upload the knowledge base document T1 to the server;

[0011] Step S102, locking T1 through L1 lock;

[0012] Step S103, extracting the text content in T1 to form C1;

[0013] Step S104, dividing C1 into a segment by taking each a characters in C1 as a segment, and obtaining an array C2 containing b small texts;

[0014] Step S105, concatenate the b small texts in C2 to form C3;

[0015] Step S106, compare C3 and C1 to determine whether the text content in C3 is complete. If the content in C1 is inconsistent with that in C3, mark the status as missing content, immediately terminate the subsequent steps and repair the inconsistent content in C2. Otherwise, execute step S107;

[0016] Step S107, store the content in C2 to the keyword database elasticsearch and the vector database milvus respectively. If the storage fails, re-upload it immediately until it succeeds. If the upload fails after re-uploading c times, mark the status as content missing, immediately end the subsequent steps and repair the content in C2. Otherwise, execute step S108;

[0017] Step S108: After the content in C2 is uploaded, if there is content that has not been successfully re-uploaded, roll back the uploaded C2 data and repair the content in C2;

[0018] Step S109: Release the L1 lock.

[0019] Wherein, a, b and c are all positive integers.

[0020] Preferably, in step S102, the L1 lock uses the knowledge base + file id as the primary key to lock the file, thereby preventing the same file from being repeatedly uploaded to the same knowledge base.

[0021] Preferably, in step S2, the knowledge document is checked regularly, specifically comprising the following steps:

[0022] Step S201, obtain the knowledge base documents uploaded within the last A hours to form C4;

[0023] Step S202: Compare C4 with the knowledge base document content uploaded within the last A hours to see if there is any missing data;

[0024] Step S203: If there is data missing in C4, the data of C4 in the database is compensated. If there is still data missing in the data of C4 after data compensation, the data of the missing part in C4 is set to an unavailable state and the data is repaired.

[0025] Preferably, a method for ensuring consistency of knowledge base file upload based on the RAG framework may further include step S3, performing regular random sampling checks on all knowledge base documents in the database.

[0026] Preferably, in step S3, a regular random sampling check is performed on all knowledge base documents in the database, specifically comprising the following steps:

[0027] Step S301, randomly extract D pieces of data every day to obtain data C5;

[0028] Step S302: Compare C5 with the original uploaded file content to determine whether C5 has any missing data;

[0029] Step S303: If there is data missing in C5, the data of C5 in the database is compensated. If there is still data missing in the data of C5 after data compensation, the data of the missing part in C5 is set to an unavailable state and the data is repaired.

[0030] Wherein, D is a positive integer.

[0031] In a second aspect, the present invention further provides a knowledge base file upload consistency assurance device based on the RAG framework, which specifically includes the following modules:

[0032] The document upload module is used to upload the knowledge base document T1 to the database;

[0033] The document checking module is used to check all knowledge base documents uploaded in the database within A hours.

[0034] Wherein, A is a positive integer.

[0035] Preferably, the document upload module specifically includes the following submodules:

[0036] The first submodule of document upload is used to upload the knowledge base document T1 to the server;

[0037] The second submodule of document upload is used to lock T1 through L1 lock;

[0038] The third submodule of document upload is used to extract the text content in T1 to form C1;

[0039] The fourth submodule of document upload is used to segment C1 by taking each a characters in C1 as a segment, and obtain an array C2 containing b small texts;

[0040] The fifth submodule of document upload is used to concatenate the b small texts in C2 to form C3;

[0041] The sixth submodule of document upload is used to compare C3 and C1 to determine whether the text content in C3 is complete. If the content in C1 and C3 is inconsistent, the status is marked as missing content, and the inconsistent content in C2 is immediately repaired;

[0042] The seventh submodule of document upload is used to store the content in C2 to the keyword database elasticsearch and the vector database milvus respectively. If the storage fails, it will be re-uploaded immediately until it succeeds. If it fails to upload after re-uploading c times, it will be marked as missing content and the content in C2 will be repaired immediately.

[0043] The eighth submodule of document upload is used to roll back the uploaded C2 data and repair the content in C2 after the content in C2 is uploaded, if there is content that has not been successfully re-uploaded;

[0044] The ninth submodule of document upload is used to release the L1 lock.

[0045] Wherein, a, b and c are all positive integers.

[0046] Preferably, in the second submodule of document upload, the L1 lock uses the knowledge base + file id as the primary key to lock the file, thereby preventing the same file from being repeatedly uploaded in the same knowledge base.

[0047] Preferably, the document checking module specifically includes the following submodules:

[0048] The first submodule of document checking is used to obtain knowledge base documents uploaded in the last A hours to form C4;

[0049] The second submodule of document check is used to compare the contents of C4 and the knowledge base documents uploaded within the last A hours to see if there is any missing data;

[0050] The third submodule of the document check is used to compensate the data of C4 in the database if there is data missing in C4. If there is still data missing in the data of C4 after data compensation, the data of the missing part in C4 is set to an unavailable state and the data is repaired.

[0051] Preferably, a knowledge base file upload consistency assurance device based on the RAG framework may further include a sampling inspection module for performing regular random sampling inspections on all knowledge base documents in the database.

[0052] Preferably, the sampling inspection module specifically includes the following submodules:

[0053] The first submodule of the sampling inspection is used to randomly select D pieces of data every day to obtain data C5;

[0054] The second submodule of the sampling check is used to compare the content of C5 with the original uploaded file to determine whether there is any missing data in C5;

[0055] The third submodule of the sampling check is used to compensate the data of C5 in the database if there is data missing in C5. If there is still data missing in the data of C5 after data compensation, the data of the missing part in C5 is set to an unavailable state and the data is repaired.

[0056] Wherein, D is a positive integer.

[0057] In a third aspect, the present invention further provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements a knowledge base file upload consistency assurance method based on a RAG framework as described in any one of the first aspects of the present application.

[0058] In a fourth aspect, the present invention further provides an electronic device, comprising: a memory storing a computer program; and a processor communicatively connected to the memory, for executing a knowledge base file upload consistency assurance method based on a RAG framework as described in any one of the first aspects of the present application when the computer program is called.

[0059] Compared with the prior art, the present invention has the following obvious outstanding substantive features and significant advantages:

[0060] The present invention provides a method and device for ensuring consistency of knowledge base file upload based on the RAG framework, which solves the problem that the consistency of knowledge base file upload based on the RAG framework is difficult to ensure. By ensuring the consistency of knowledge base file upload based on the RAG framework, it is ensured that the content of the file uploaded to the database is consistent with the original file content, avoiding misleading caused by information contradictions, ensuring the accuracy of data during subsequent large language model training and data retrieval, and thus improving the accuracy of the output results of the large language model. BRIEF DESCRIPTION OF THE DRAWINGS

[0061] The accompanying drawings, which constitute part of the present invention, are used to provide a further understanding of the present invention. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute improper limitations on the present invention. In the accompanying drawings:

[0062] Figure 1 It is a flow chart of a knowledge base file upload consistency assurance method based on a RAG framework according to a preferred embodiment of the present invention.

[0063] Figure 2It is a flowchart of uploading a knowledge base document T1 to a database in a knowledge base file upload consistency assurance method based on a RAG framework in a preferred embodiment of the present invention.

[0064] Figure 3 It is a flowchart of performing periodic checking on knowledge documents in a knowledge base file upload consistency assurance method based on a RAG framework in a preferred embodiment of the present invention.

[0065] Figure 4 It is a flowchart of performing regular random sampling checks on all knowledge base documents in a database in a knowledge base file upload consistency assurance method based on a RAG framework in a preferred embodiment of the present invention.

[0066] Figure 5 It is a structural diagram of a knowledge base file upload consistency assurance device based on a RAG framework according to a preferred embodiment of the present invention.

[0067] Figure 6 It is a structural diagram of a document uploading module in a knowledge base file uploading consistency assurance device based on a RAG framework in a preferred embodiment of the present invention.

[0068] Figure 7 It is a structural diagram of a document checking module in a knowledge base file upload consistency assurance device based on a RAG framework in a preferred embodiment of the present invention.

[0069] Figure 8 It is a structural diagram of a sampling inspection module in a knowledge base file upload consistency assurance device based on a RAG framework in a preferred embodiment of the present invention. DETAILED DESCRIPTION

[0070] The present invention provides a knowledge base file upload consistency guarantee method and device based on the RAG framework. In order to make the purpose, technical solution and effect of the present invention clearer and more specific, the present invention is further described in detail with reference to the accompanying drawings and examples. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.

[0071] It should be noted that the terms "first", "second", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged under appropriate circumstances. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0072] Embodiment 1:

[0073] like Figure 1-Figure 4 As shown, the knowledge base file upload consistency guarantee method based on the RAG framework described in this embodiment specifically includes the following steps:

[0074] Step S1, uploading the knowledge base document T1 to the database. In step S1, uploading the knowledge base document T1 to the database specifically includes the following steps:

[0075] Step S101, upload the knowledge base document T1 to the server.

[0076] Step S102: Lock T1 through L1 lock.

[0077] Optionally, in step S102, the L1 lock locks the file with the knowledge base + file id as the primary key to prevent the same file from being repeatedly uploaded in the same knowledge base.

[0078] Step S103: extract the text content in T1 to form C1.

[0079] Step S104 , dividing C1 into a segment with each a characters in C1 as a segment, and obtaining an array C2 containing b small texts.

[0080] Wherein, a and b are both positive integers. In this embodiment, a is set to 300.

[0081] Step S105: concatenate the b small texts in C2 to form C3.

[0082] Step S106, compare C3 and C1 to determine whether the text content in C3 is complete. If the contents in C1 and C3 are inconsistent, the status is marked as missing content, and the subsequent steps are immediately terminated and the inconsistent content in C2 is repaired. Otherwise, execute step S107.

[0083] Step S107, store the content in C2 to the keyword database elasticsearch and the vector database milvus respectively. If storage fails, re-upload it immediately until it succeeds. If the upload fails after re-uploading c times, mark the status as content missing, immediately end the subsequent steps and repair the content in C2. Otherwise, execute step S108.

[0084] Wherein, c is a positive integer. In this embodiment, c is set to 3.

[0085] Step S108: After the content in C2 is uploaded, if there is content that has not been successfully re-uploaded, the uploaded data of C2 is rolled back and the content in C2 is repaired.

[0086] Step S109: Release the L1 lock.

[0087] Step S2: Check all knowledge base documents uploaded in the database within A hours.

[0088] Wherein, A is a positive integer. In this embodiment, A is set to 24.

[0089] In step S2, the knowledge document is checked regularly, which specifically includes the following steps:

[0090] Step S201: Obtain knowledge base documents uploaded within the last A hours to form C4.

[0091] Step S202: Compare C4 with the knowledge base document content uploaded within the last A hours to see if there is any missing data.

[0092] Step S203: If there is data missing in C4, the data of C4 in the database is compensated. If there is still data missing in the data of C4 after data compensation, the data of the missing part in C4 is set to an unavailable state and the data is repaired.

[0093] Optionally, a method for ensuring consistency of knowledge base file upload based on the RAG framework further includes step S3, performing regular random sampling checks on all knowledge base documents in the database.

[0094] Among them, in step S3, a regular random sampling check is performed on all knowledge base documents in the database, which specifically includes the following steps:

[0095] Step S301, randomly extract D pieces of data at regular intervals every day to obtain data C5.

[0096] Wherein, D is a positive integer. In this embodiment, D is set to 10000.

[0097] Step S302: Compare C5 with the original uploaded file content to determine whether there is any missing data in C5.

[0098] Step S303: If there is data missing in C5, the data of C5 in the database is compensated. If there is still data missing in the data of C5 after data compensation, the data of the missing part in C5 is set to an unavailable state and the data is repaired.

[0099] Embodiment 2:

[0100] like Figure 5-Figure 8 As shown, the knowledge base file upload consistency assurance device based on the RAG framework described in this embodiment specifically includes the following modules:

[0101] The document upload module is used to upload the knowledge base document T1 to the database.

[0102] The document upload module specifically includes the following submodules:

[0103] The first submodule of document upload is used to upload the knowledge base document T1 to the server.

[0104] The second submodule of document upload is used to lock T1 through L1 lock.

[0105] The third submodule of document upload is used to extract the text content in T1 to form C1.

[0106] The fourth submodule of document upload is used to segment C1 by taking each a characters in C1 as a segment, and obtain an array C2 containing b small texts.

[0107] Wherein, a and b are both positive integers. In this embodiment, a is set to 300.

[0108] The fifth submodule of document upload is used to concatenate the b small texts in C2 to form C3.

[0109] The sixth submodule of document upload is used to compare C3 and C1 to determine whether the text content in C3 is complete. If the contents in C1 and C3 are inconsistent, the status is marked as missing content, and the inconsistent content in C2 is immediately repaired.

[0110] The seventh submodule of document upload is used to store the content in C2 to the keyword database elasticsearch and the vector database milvus respectively. If a storage failure occurs, it will be re-uploaded immediately until it succeeds. If the upload fails after c re-uploads, the status is marked as missing content and the content in C2 is repaired immediately.

[0111] Wherein, c is a positive integer, and in this embodiment, c is set to 3.

[0112] The eighth submodule of document upload is used to roll back the uploaded C2 data and repair the content in C2 after the content in C2 is uploaded, if there is content that has not been successfully re-uploaded.

[0113] The ninth submodule of document upload is used to release the L1 lock.

[0114] The document checking module is used to check all knowledge base documents uploaded in the database within A hours.

[0115] Wherein, A is a positive integer. In this embodiment, A is set to 24.

[0116] Optionally, in the second submodule of document upload, the L1 lock uses the knowledge base + file id as the primary key to lock the file to prevent the same file from being repeatedly uploaded in the same knowledge base.

[0117] Optionally, the document checking module specifically includes the following submodules:

[0118] The first submodule of document checking is used to obtain knowledge base documents uploaded within the last A hours to form C4.

[0119] The second submodule of document check is used to compare the contents of C4 and the knowledge base documents uploaded within the last A hours to see if there is any missing data.

[0120] The third submodule of the document check is used to compensate the data of C4 in the database if there is data missing in C4. If there is still data missing in the data of C4 after data compensation, the data of the missing part in C4 is set to an unavailable state and the data is repaired.

[0121] Optionally, a knowledge base file upload consistency assurance device based on the RAG framework further includes a sampling inspection module, and the sampling inspection module is used to perform regular random sampling inspection on all knowledge base documents in the database.

[0122] Optionally, the sampling inspection module specifically includes the following submodules:

[0123] The first submodule of the sampling inspection is used to randomly select D pieces of data at a fixed time every day to obtain data C5.

[0124] The second submodule of the sampling check is used to compare C5 with the content of the original uploaded file to determine whether there is any missing data in C5.

[0125] The third submodule of the sampling check is used to compensate the data of C5 in the database if there is data missing in C5. If there is still data missing in the data of C5 after data compensation, the data of the missing part in C5 is set to an unavailable state and the data is repaired.

[0126] Wherein, D is a positive integer. In this embodiment, D is set to 10000.

[0127] The specific embodiments of the present invention are described in detail above, but they are only examples, and the present invention is not limited to the specific embodiments described above. For those skilled in the art, any equivalent modifications and substitutions made to the present invention are also within the scope of the present invention. Therefore, the equalization changes and modifications made without departing from the spirit and scope of the present invention should be included in the scope of the present invention.

Claims

1. A method for ensuring consistency of knowledge base file upload based on RAG framework, characterized in that: The specific steps include: Step S1, upload the knowledge base document T1 to the database; Step S2: Check all knowledge base documents uploaded in the database within A hours; Wherein, A is a positive integer.

2. According to the method for ensuring consistency of knowledge base file upload based on RAG framework in claim 1, it is characterized in that: In step S1, the knowledge base document T1 is uploaded to the database, which specifically includes the following steps: Step S101, upload the knowledge base document T1 to the server; Step S102, locking T1 through L1 lock; Step S103, extracting the text content in T1 to form C1; Step S104, dividing C1 into a segment by taking each a characters in C1 as a segment, and obtaining an array C2 containing b small texts; Step S105, concatenate the b small texts in C2 to form C3; Step S106, compare C3 and C1 to determine whether the text content in C3 is complete. If the content in C1 is inconsistent with that in C3, mark the status as missing content, immediately terminate the subsequent steps and repair the inconsistent content in C2. Otherwise, execute step S107; Step S107, store the content in C2 to the keyword database elasticsearch and the vector database milvus respectively. If the storage fails, re-upload it immediately until it succeeds. If the upload fails after re-uploading c times, mark the status as content missing, immediately end the subsequent steps and repair the content in C2. Otherwise, execute step S108; Step S108: After the content in C2 is uploaded, if there is content that has not been successfully re-uploaded, roll back the uploaded C2 data and repair the content in C2; Step S109, releasing the L1 lock; Wherein, a, b and c are all positive integers.

3. According to the method for ensuring consistency of knowledge base file upload based on RAG framework in claim 2, it is characterized in that: In step S102, the L1 lock uses the knowledge base + file id as the primary key to lock the file to prevent the same file from being repeatedly uploaded to the same knowledge base.

4. According to the method for ensuring consistency of knowledge base file upload based on RAG framework in claim 1, it is characterized in that: In step S2, the knowledge document is checked regularly, which specifically includes the following steps: Step S201, obtain the knowledge base documents uploaded within the last A hours to form C4; Step S202: Compare C4 with the knowledge base document content uploaded within the last A hours to see if there is any missing data; Step S203: If there is data missing in C4, the data of C4 in the database is compensated. If there is still data missing in the data of C4 after data compensation, the data of the missing part in C4 is set to an unavailable state and the data is repaired.

5. The method for ensuring consistency of knowledge base file upload based on RAG framework according to claim 1 is characterized in that: The method further includes step S3 of performing regular random sampling checks on all knowledge base documents in the database.

6. A method for ensuring consistency of knowledge base file upload based on RAG framework according to claim 5, characterized in that: In step S3, a random sampling check is performed on all knowledge base documents in the database at a regular interval, specifically including the following steps: Step S301, randomly extract D pieces of data every day to obtain data C5; Step S302: Compare C5 with the original uploaded file content to determine whether C5 has any missing data; Step S303: If there is data missing in C5, the data of C5 in the database is compensated. If there is still data missing in the data of C5 after data compensation, the data of the missing part in C5 is set to an unavailable state, and the data is repaired; Wherein, D is a positive integer.

7. A device for ensuring consistency of knowledge base file upload based on RAG framework, characterized in that: The modules include: The document upload module is used to upload the knowledge base document T1 to the database; The document checking module is used to check all knowledge base documents uploaded in the database within A hours; Wherein, A is a positive integer.

8. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method for ensuring consistency of knowledge base file upload based on the RAG framework as described in any one of claims 1 to 6 is implemented.

9. An electronic device, characterized in that: It includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements a method for ensuring consistency of knowledge base file upload based on the RAG framework as described in any one of claims 1 to 6.