Correct rate verification method and device of RAG knowledge base and storage medium

Abstract

The invention provides a correct rate verification method and device of an RAG knowledge base and a storage medium, and solves the problems that a traditional correct rate verification mode of the RAG knowledge base is low in manual sampling inspection efficiency, and an auditing result is separated from a statistical caliber, the method comprises the following steps: importing a plurality of verification questions and a standard answer of each verification question into a unified file template, the field integrity is ensured; the multiple verification questions are input into a large language model of the RAG knowledge base in batches and concurrently for retrieval, the similarity between the structured JSON format answer of each batch of verification questions and the standard answer of each batch of verification questions is obtained and calculated, and whether the structured JSON format answer is correct or not is marked based on the similarity; displaying the data on a visual auditing interface, and configuring a result judgment button for each verification problem in each batch of verification problems; and in response to the verification operation of the trigger result judgment button, updating the global correct rate of the RAG knowledge base.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and more particularly to a method, apparatus and storage medium for verifying the accuracy of a RAG knowledge base. Background Technology

[0002] After a traditional RAG knowledge base is launched, operations staff typically randomly select a small number of questions for manual comparison to verify the accuracy of the knowledge base. This sampling method has limited knowledge coverage and is prone to leaving many potential errors undetected for a long time, which may eventually lead to business incidents.

[0003] Furthermore, the lengthy manual verification process, coupled with the acceptance of statistical data, makes the entire verification and testing process quite time-consuming. This results in the accuracy of the traditional RAG knowledge base not being updated in real time after manual verification, causing a disconnect between the metrics seen by management and the true accuracy of the traditional RAG knowledge base, thus impacting subsequent optimization decisions.

[0004] In summary, traditional methods for verifying the accuracy of RAG knowledge bases suffer from problems such as low efficiency of manual sampling and a disconnect between audit results and statistical standards. Summary of the Invention

[0005] This application provides a method, apparatus, and storage medium for verifying the accuracy of a RAG knowledge base, in order to solve the problems existing in related technologies. The technical solution is as follows: In a first aspect, embodiments of this application provide a method for verifying the accuracy of a RAG knowledge base, including: Import multiple validation questions and the standard answer for each validation question into a unified file template, and ensure the integrity of the fields in the unified file template; Multiple verification questions are input into the large language model of the RAG knowledge base in batches for retrieval, and the structured JSON format answers of each batch of verification questions are obtained. Then, the similarity between the structured JSON format answers and the standard answers of each batch of verification questions in the unified file template is calculated, and the correctness of the structured JSON format answers is marked based on the similarity. The structured JSON format answers, the standard answers to each batch of verification questions, and the similarity between the two are displayed on the visual review interface, and a result judgment button is provided for each verification question in each batch of verification questions. In response to the verification operation triggered by the result determination button, the global accuracy of the RAG knowledge base is updated.

[0006] In one implementation, importing multiple validation questions and the standard answer for each validation question into a unified document template, and ensuring the field integrity of the unified document template includes: Create a file template; Customize the question list and answer list on the file template to obtain the unified file template; The multiple verification questions and the standard answers for each verification question are imported into the question list and answer list of the unified file template, while the integrity of the fields in the unified file template is verified. If the verification shows that at least one line in the question list or answer list of the unified document template is empty or has a format error, a prompt message will be output. The prompt message is used to indicate that there is a field error in the question list or answer list of the unified document template, which needs to be supplemented or corrected. The process continues until the unified document template is verified to have no empty fields and the format is correct, thus confirming that the fields of the unified document template are complete.

[0007] In one embodiment, the method further includes: Generate a unique identifier for each import of the unified file template; The entire process is recorded based on the unique identifier, documenting the operation trajectory for each imported verification question and its standard answer.

[0008] In one implementation, multiple verification questions are input concurrently into a large language model of the RAG knowledge base for retrieval, and the structured JSON format answers for each batch of verification questions are obtained, including: Count the number of question lines N in the unified file template occupied by the multiple verification questions; Following the approach of grouping L verification questions into batches, the multiple verification questions are divided into M batches, where M = ; Using an asynchronous queue with backoff and retry functionality, each batch of verification questions is sequentially input into the large language model for retrieval, obtaining the structured JSON format answers for each batch of verification questions.

[0009] In one implementation, an asynchronous queue with backoff and retry functionality is used to sequentially input each batch of verification questions into the large language model for retrieval, obtaining the structured JSON format answers for each batch of verification questions, including: Using the asynchronous queue, each batch of verification questions is concurrently input into the large language model according to a preset interval, to obtain the structured JSON format answer for each set of verification questions. The preset interval is set based on the rate limiting threshold of the large language model.

[0010] In one implementation, the structured JSON format answer includes the retrieved answer for each verification question in each batch of verification questions; calculating the similarity between the structured JSON format answer and the standard answer for each batch of verification questions in the unified document template, and marking the correctness of the structured JSON format answer based on the similarity, includes: The retrieval answer and its corresponding standard answer for each verification question in each batch of verification questions are uniformly input into the vector model, and the similarity between the retrieval answer and its corresponding standard answer for each verification question is calculated. If the similarity between the search answer and the corresponding standard answer of each verification question exceeds a preset threshold, the search answer of each verification question is marked as the correct answer; otherwise, the search answer of each verification question is marked as the incorrect answer.

[0011] In one implementation, the visual review interface adopts a three-column layout, wherein the left column is used to display each verification question in each batch of verification questions, the right column is used to display the standard answer of each verification question, and the middle column is used to display the search answer of each verification question and its similarity with the standard answer; The visual review interface supports filtering and displaying verification questions based on similarity range and keywords.

[0012] Secondly, embodiments of this application also provide an accuracy verification device for a RAG knowledge base, comprising: The data import unit is used to import multiple verification questions and the standard answer of each verification question into a unified file template, and to ensure the integrity of the fields in the unified file template; A two-layer verification unit is used to input multiple verification questions into the large language model of the RAG knowledge base in batches for retrieval, obtain the structured JSON format answer of each batch of verification questions, calculate the similarity between the structured JSON format answer and the standard answer of each batch of verification questions in the unified file template, and mark the correctness of the structured JSON format answer based on the similarity. The visual review unit is used to display the structured JSON format answers, the standard answers of each batch of verification questions, and the similarity between the two on the visual review interface, and to equip each verification question in each batch of verification questions with a result judgment button; The visual review unit is also used to update the global accuracy of the RAG knowledge base in response to the verification operation triggered by the result judgment button.

[0013] In one implementation, the data import unit, when used to import multiple validation questions and the standard answer to each validation question into a unified document template, and ensuring the integrity of the fields in the unified document template, is specifically used for: Create a file template; Customize the question list and answer list on the file template to obtain the unified file template; The multiple verification questions and the standard answers for each verification question are imported into the question list and answer list of the unified file template, while the integrity of the fields in the unified file template is verified. If the verification shows that at least one line in the question list or answer list of the unified document template is empty or has a format error, a prompt message will be output. The prompt message is used to indicate that there is a field error in the question list or answer list of the unified document template, which needs to be supplemented or corrected. The process continues until the unified document template is verified to have no empty fields and the format is correct, thus confirming that the fields of the unified document template are complete.

[0014] In one implementation, the data import unit is further configured to: Generate a unique identifier for each import of the unified file template; The entire process is recorded based on the unique identifier, documenting the operation trajectory for each imported verification question and its standard answer.

[0015] In one implementation, the two-layer verification unit, when retrieving the structured JSON format answers for each batch of verification questions from a large language model that concurrently inputs multiple verification questions into the RAG knowledge base in batches, specifically performs the following: Count the number of question lines N in the unified file template occupied by the multiple verification questions; Following the approach of grouping L verification questions into batches, the multiple verification questions are divided into M batches, where M = ; Using an asynchronous queue with backoff and retry functionality, each batch of verification questions is sequentially input into the large language model for retrieval, obtaining the structured JSON format answers for each batch of verification questions.

[0016] In one implementation, the two-layer verification unit, when using an asynchronous queue with backoff and retry functionality to sequentially input each batch of verification questions concurrently into the large language model for retrieval and obtain structured JSON format answers for each batch of verification questions, is specifically used for: Using the asynchronous queue, each batch of verification questions is concurrently input into the large language model according to a preset interval, to obtain the structured JSON format answer for each set of verification questions. The preset interval is set based on the rate limiting threshold of the large language model.

[0017] In one implementation, the structured JSON format answer includes the retrieved answer for each verification question in each batch of verification questions; the two-layer verification unit, when calculating the similarity between the structured JSON format answer and the standard answer for each batch of verification questions in the unified document template, and marking the correctness of the structured JSON format answer based on the similarity, is specifically used for: The retrieval answer and its corresponding standard answer for each verification question in each batch of verification questions are uniformly input into the vector model, and the similarity between the retrieval answer and its corresponding standard answer for each verification question is calculated. If the similarity between the search answer and the corresponding standard answer of each verification question exceeds a preset threshold, the search answer of each verification question is marked as the correct answer; otherwise, the search answer of each verification question is marked as the incorrect answer.

[0018] In one implementation, the visual review interface adopts a three-column layout, wherein the left column is used to display each verification question in each batch of verification questions, the right column is used to display the standard answer of each verification question, and the middle column is used to display the search answer of each verification question and its similarity with the standard answer; The visual review interface supports filtering and displaying verification questions based on similarity range and keywords.

[0019] Thirdly, embodiments of this application also provide a computer device, which includes: a memory and a processor, wherein the memory stores instructions, the instructions are loaded and executed by the processor to implement the methods in any of the above embodiments, wherein the memory and the processor communicate with each other through an internal connection path.

[0020] Fourthly, embodiments of this application also provide a computer-readable storage medium storing a computer program that, when run on a computer, implements the methods in any of the above-described embodiments.

[0021] The advantages or beneficial effects of the above technical solutions include at least the following: This application imports multiple verification questions and their standard answers into a unified file template, ensuring the integrity of the template's fields. This facilitates accurate subsequent verification of the RAG knowledge base's accuracy and allows for the timely detection of potential errors. Furthermore, by batch-wise and concurrently inputting multiple verification questions into the RAG knowledge base's large language model for retrieval, and obtaining structured JSON-formatted answers for each batch, the similarity between these answers and the standard answers in the unified file template is calculated. Based on this similarity, the correctness of the structured JSON-formatted answers is determined. This intelligent batching and concurrency control of the imported verification questions allows for efficient and rapid retrieval and output of results. Simultaneously, it also enables intelligent batching and concurrency control of the RAG knowledge base. The RAG knowledge base employs a two-layer verification process: the first layer involves batch responses using a large language model, while the second layer calculates similarity to further ensure the accuracy of the RAG knowledge base's accuracy verification. This allows for broader question verification covering the RAG knowledge base, enabling the timely detection of numerous potential errors and preventing subsequent business incidents. Furthermore, by displaying the structured JSON-formatted answers, the standard answers for each batch of verification questions, and their similarity on a visual review interface, and providing a result judgment button for each verification question in each batch, the global accuracy of the RAG knowledge base is updated in real-time in response to the verification operation that triggers this button. This ensures that the metrics seen by management are closely correlated with the true accuracy of the RAG knowledge base, preventing any impact on subsequent optimization decisions. In short, this application addresses the problems of low efficiency in manual sampling and the disconnect between review results and statistical standards inherent in traditional RAG knowledge base accuracy verification methods.

[0022] The above overview is for illustrative purposes only and is not intended to be limiting in any way. In addition to the illustrative aspects, embodiments, and features described above, further aspects, embodiments, and features of this application will become readily apparent from the accompanying drawings and the following detailed description. Attached Figure Description

[0023] In the accompanying drawings, unless otherwise specified, the same reference numerals throughout the various drawings denote the same or similar parts or elements. These drawings are not necessarily drawn to scale. It should be understood that these drawings depict only some embodiments disclosed in this application and should not be construed as limiting the scope of this application.

[0024] Figure 1 A flowchart illustrating a method for verifying the accuracy of a RAG knowledge base, provided in an embodiment of this application; Figure 2A flowchart illustrating an execution step S110 provided in this application embodiment; Figure 3 A flowchart illustrating an execution step S120 provided in this application embodiment; Figure 4 A structural block diagram of a RAG knowledge base accuracy verification device provided in this application embodiment; Figure 5 This is a structural block diagram of a computer device provided in an embodiment of this application. Detailed Implementation

[0025] In the following description, only certain exemplary embodiments are briefly described. As those skilled in the art will recognize, the described embodiments can be modified in various ways without departing from the spirit or scope of this application. Therefore, the drawings and description are considered to be exemplary in nature and not restrictive.

[0026] To facilitate those skilled in the art to understand the relevant technical solutions provided in the embodiments of this application, the relevant terms involved in this application will be explained below.

[0027] RAG stands for Retrieval-Augmented Generation, a technique that combines information retrieval and generative language models. It first retrieves relevant information from a knowledge base through information retrieval, and then uses a generative language model to generate an answer based on the retrieved information.

[0028] Vector models: Models used to convert text (such as questions and answers) into vector representations to facilitate similarity calculations, such as Sentence-BERT and ERNIE.

[0029] Cosine similarity: This is an indicator used to measure the similarity between two vectors (such as the vector representation of text). The value ranges from -1 to 1, and the closer the value is to 1, the higher the similarity.

[0030] Asynchronous queues (such as Celery and RabbitMQ) are tools for handling asynchronous tasks and message queues. They enable asynchronous execution, distribution, and scheduling of tasks, avoiding system blocking under high concurrency and ensuring stable task execution.

[0031] The relevant technical solutions provided in the embodiments of this application will be described in detail below with reference to the accompanying drawings.

[0032] Figure 1 A flowchart illustrating a method for verifying the accuracy of a RAG knowledge base according to an embodiment of this application is shown. Figure 1 As shown, the method may include the following steps: S110. Import multiple validation questions and the standard answer for each validation question into a unified file template, and ensure the integrity of the fields in the unified file template.

[0033] In one implementation, these multiple verification questions can be pre-stored questions in the RAG knowledge base or questions set manually; this application does not specifically limit this.

[0034] In one implementation, combined with Figure 1 and Figure 2 As shown, the implementation process of step S110 may include the following steps: S111, Create a file template.

[0035] As an example, a file template can be created based on a CSV or EXCEL file, thus adapting the file template to data integration and batch import / export.

[0036] S112. Customize the question list and answer list on the file template to obtain a unified file template.

[0037] This can be understood as the unified file template containing a dual list of "questions + answers", thus supporting the batch import of a large number of verification questions and the standard answers for each verification question into the unified file template.

[0038] S113. Import multiple validation questions and their corresponding standard answers into the question list and answer list of the unified file template, and at the same time verify the field integrity of the unified file template.

[0039] As an example, regular expressions or third-party libraries (such as ExcelJS) can be used to validate fields in real time when importing multiple validation questions and their corresponding standard answers into the question list and answer list of a unified file template. This could include checking if a row in the unified file template contains an empty validation question and / or a row contains an empty standard answer, or checking if a row in the unified file template contains a formatting error in a validation question and / or a row contains an incorrect standard answer, and so on.

[0040] S114. If at least one line in the question list or answer list of the unified document template is found to be empty or has a format error, output a prompt message.

[0041] In practice, this prompt can be used to indicate that there are field errors (i.e., missing fields or incorrect formatting) in the question list or answer list in the unified document template, and that they need to be supplemented or corrected.

[0042] For example, if the verification shows that the xth line of the unified document template is empty, the prompt message "The xth line of the problem field is missing. Please complete it before importing" will be output immediately.

[0043] S115. Continue until the unified document template is verified to have no empty fields and the format is correct, to ensure that the fields of the unified document template are complete.

[0044] This can be understood as follows: if a field error is found in the unified document template, the user needs to correct it before proceeding. In other words, the completeness of the fields in the unified document template is a prerequisite for subsequent verification.

[0045] In this embodiment of the application, by executing step S110, a large number of verification questions and their standard answers can be uniformly imported, and the field integrity of the unified file template can be ensured. This facilitates the subsequent accurate verification of the accuracy of the RAG knowledge base and can also promptly detect a large number of potential errors that may exist in the RAG knowledge base.

[0046] In one applicable scenario provided by the embodiments of this application, a unique identifier (such as batch number batch_id) can be generated for each import of a unified file template. Then, based on the unique identifier, the operation trajectory of the verification questions and their standard answers for each import can be recorded throughout the process. In this way, it is convenient to trace the verification data of the RAG knowledge base for each time.

[0047] S120. Input multiple verification questions into the large language model of the RAG knowledge base in batches for retrieval, obtain the structured JSON format answer of each batch of verification questions, calculate the similarity between the structured JSON format answer and the standard answer of each batch of verification questions in the unified file template, and mark whether the structured JSON format answer is correct or not based on the similarity.

[0048] In one implementation, combined with Figures 1-3 As shown, the implementation process of step S120 may include the following steps: S121. Count the number of question lines N occupied by multiple validation questions in the unified file template.

[0049] In practice, after completing step S110, the number of question lines N occupied by multiple verification questions in the unified file template can be automatically counted. This can be understood as counting the total number of multiple verification questions imported into the unified file template this time.

[0050] S122. Divide multiple verification issues into M batches according to the method of L verification issues per batch.

[0051] In practice, L can be set according to actual needs; for example, L can be set to 20. M= , where M= This is represented as rounding up M.

[0052] For example, assuming the total number of validation questions is 125, then M = 7, which is the integer part of 125 / 20 = 6.25. In other words, the first six batches of the seven batches each contain 20 validation questions, and the last batch contains 5 validation questions.

[0053] S123. Using an asynchronous queue with backoff and retry functionality, input each batch of verification questions concurrently into the large language model for retrieval, and obtain the structured JSON format answers for each batch of verification questions.

[0054] In practice, this asynchronous queue can be used to concurrently input each batch of verification questions into the large language model for retrieval according to a preset interval, so as to obtain the structured JSON format answer for each set of verification questions. The preset interval is set according to the rate limiting threshold (such as TPS) of the large language model, so as to avoid triggering HTTP 429 errors due to excessive concurrency.

[0055] In this embodiment, the asynchronous queue is used to concurrently input a batch of verification questions into the large language model for retrieval at preset intervals. During this process, if a batch of verification questions fails to retrieve the data from the large language model, the asynchronous queue automatically delays the retry. If the large language model retryes the retrieval of that batch of verification questions, other batches of verification questions ranked after that batch continue to wait for retrieval, ensuring stable processing of requests.

[0056] S124. Input the retrieval answer and its corresponding standard answer for each verification question in each batch of verification questions into the vector model, and calculate the similarity between the retrieval answer and its corresponding standard answer for each verification question.

[0057] In practice, the structured JSON format answer for each batch of validation questions includes the retrieval answer for each validation question in each batch of validation questions.

[0058] For example, when the search answer and its corresponding standard answer for each verification question in each batch of verification questions are uniformly input into the vector model, the vector model converts them into 768-dimensional vectors and then calculates their cosine similarity, which is the similarity between the search answer and the corresponding standard answer for each verification question.

[0059] As an example, the structured JSON answer for each batch of validation questions can also include a usage token for each batch (providing details of the number of tokens used by the large language model for input and output). The usage token includes: input_tokens (the number of input tokens used by the large language model to retrieve each validation question), output_tokens (the number of output tokens used by the large language model to retrieve each validation question), and total_tokens (the total number of tokens used by the large language model to retrieve each validation question). The usage token can be used for subsequent billing.

[0060] S125. If the similarity between the search answer and the corresponding standard answer of each verification question exceeds a preset threshold, mark the search answer of each verification question as correct; otherwise, mark the search answer of each verification question as incorrect.

[0061] As an example, the preset threshold can be set to 0.85, or it can be set according to actual needs. This application embodiment does not limit this.

[0062] For example, taking a preset threshold of 0.85 as an example, if a verification question is "What is the starting price of the iPhone 15?", the standard answer is "The starting price of the iPhone 15 is 5999 yuan". If the large language model yields the retrieval answer "The starting price of the iPhone 15 is 5999 yuan", and the similarity calculated by the vector model is 0.92, which exceeds 0.85, the retrieval answer for the verification question is determined to be correct and can be marked as the correct answer. If the large language model yields the retrieval answer "The starting price of the iPhone 15 is 6000 yuan", and the similarity calculated by the vector model is 0.78, which is lower than 0.85, the retrieval answer for the verification question is determined to be incorrect and can be marked as the incorrect answer.

[0063] In step S125, the preset threshold can be dynamically configured without modifying the code, making it adaptable to different business lines. For example, financial businesses, which have extremely high accuracy requirements, can raise the preset threshold to 0.9, while entertainment businesses, which accept a certain degree of flexibility, can lower the preset threshold to 0.8.

[0064] In this embodiment, by executing step S120, intelligent batching and concurrent control of multiple verification questions imported from the unified file template can be achieved. This allows for efficient and rapid retrieval of these multiple verification questions and output of retrieval results. Simultaneously, a two-layer verification of the RAG knowledge base can be performed: the first layer of verification is batch answering using a large language model, and the second layer of verification is similarity calculation. This further ensures the accuracy of the RAG knowledge base's accuracy verification, thereby supporting verification of questions covering a wider range of knowledge in the RAG knowledge base. It can also promptly detect a large number of potential errors and avoid subsequent business incidents.

[0065] In another applicable scenario provided in this application embodiment, the request batch size (i.e., L mentioned above) and interval (i.e., the preset interval duration mentioned above) can be dynamically adjusted based on the real-time response status of the large language model (such as token usage rate and queue waiting time). In this way, intelligent adaptation of task scheduling can be achieved, ensuring efficient progress of verification tasks. For example, if L is initially set to 20, that is, 20 verification questions / batch, when the response delay of the large language model is detected to exceed 3 seconds, L can be automatically reduced to 15, that is, reduced to 15 verification questions / batch.

[0066] S130. Display the structured JSON format answer, the standard answer of each batch of verification questions, and the similarity between the two on the visual review interface, and equip each verification question in each batch of verification questions with a result judgment button.

[0067] In one implementation, the result judgment button includes a correct button and an incorrect button, allowing users to automatically trigger an update of the global accuracy of the RAG knowledge base by clicking the result judgment button.

[0068] In one implementation, the visual review interface adopts a three-column layout, wherein the left column is used to display each verification question in each batch of verification questions, the right column is used to display the standard answer of each verification question, and the middle column is used to display the search answer of each verification question and its similarity with the standard answer.

[0069] As an example, each validation question is equipped with a result judgment button that can be set to a suitable position on the visual review interface, such as in the middle column of the visual review interface corresponding to each validation question.

[0070] In one implementation, the visual review interface can support filtering and displaying verification questions according to similarity ranges (such as 0.8-0.85, 0.7-0.8, etc.) and keywords (such as question keywords and answer keywords), which can help manual reviewers accurately focus on key content.

[0071] S140. In response to the verification operation triggered by the result determination button, update the global accuracy of the RAG knowledge base.

[0072] For example, if a user discovers during the review process that a search answer deemed correct by the above process deviates from the actual answer, clicking the error button will immediately update the status of the corresponding question and simultaneously update the global accuracy rate of the RAG knowledge base. For instance, the global accuracy rate can be updated according to the calculation method of the number of correct answers / the total number of answers, thus intuitively reflecting the results of manual correction.

[0073] In one implementation, the visual audit interface can also perform triggered verification operations, thus preventing accidental triggering that could lead to incorrect updates to the global accuracy of the RAG knowledge base.

[0074] In this embodiment of the application, by executing steps S130 and S140, the global accuracy of the RAG knowledge base can be refreshed in real time, so that the indicators seen by the management are closely related to the true accuracy of the RAG knowledge base, thus avoiding affecting subsequent optimization decisions.

[0075] To further facilitate understanding of the accuracy verification method of the RAG knowledge base provided in this application (hereinafter referred to as this scheme), the following table compares this scheme with other existing schemes.

[0076] Table 1

[0077] As described above, the RAG knowledge base accuracy verification method provided in this application imports multiple verification questions and their standard answers into a unified file template, ensuring the integrity of the fields in the unified file template. This facilitates accurate subsequent verification of the RAG knowledge base's accuracy and allows for the timely detection of numerous potential errors. Furthermore, by inputting multiple verification questions into the large language model of the RAG knowledge base in batches for retrieval, obtaining structured JSON format answers for each batch of verification questions, calculating the similarity between these structured JSON format answers and the standard answers for each batch of verification questions in the unified file template, and marking the correctness of the structured JSON format answers based on the similarity, the method achieves intelligent batching and concurrent control of the multiple verification questions imported into the unified file template. This enables efficient and rapid completion of the retrieval and input of these multiple verification questions. The method not only displays search results but also performs dual-layer verification on the RAG knowledge base: the first layer involves batch responses using a large language model, and the second layer involves similarity calculation. This further ensures the accuracy of the RAG knowledge base's accuracy verification, enabling broader question verification and timely detection of potential errors to prevent subsequent business incidents. Furthermore, by displaying the structured JSON-formatted answers, the standard answers for each batch of verification questions, and their similarity on a visual review interface, and providing a result judgment button for each verification question in each batch, the method updates the global accuracy of the RAG knowledge base in real time upon triggering the verification operation of this button. This ensures that the metrics seen by management are closely related to the true accuracy of the RAG knowledge base, preventing any impact on subsequent optimization decisions. In other words, the RAG knowledge base accuracy verification method provided in this application solves the problems of low efficiency in manual sampling and the disconnect between review results and statistical standards inherent in traditional RAG knowledge base accuracy verification methods.

[0078] Figure 4 A structural block diagram of a RAG knowledge base accuracy verification device according to an embodiment of this application is shown. Figure 4 As shown, the device may include: Data import unit 210 is used to import multiple validation questions and the standard answer of each validation question into a unified file template, and to ensure the integrity of the fields in the unified file template; The two-layer verification unit 220 is used to input multiple verification questions into the large language model of the RAG knowledge base in batches for retrieval, obtain the structured JSON format answers of each batch of verification questions, calculate the similarity between the structured JSON format answers and the standard answers of each batch of verification questions in the unified file template, and mark the correctness of the structured JSON format answers based on the similarity. The visual review unit 230 is used to display structured JSON format answers, standard answers to each batch of verification questions, and the similarity between the two on the visual review interface, and to equip each verification question in each batch of verification questions with a result judgment button; The visual audit unit 230 is also used to update the global accuracy of the RAG knowledge base in response to the verification operation triggered by the result judgment button.

[0079] In one implementation, the data import unit 210, when used to import multiple validation questions and the standard answer for each validation question into a uniform document template, and ensuring the integrity of the fields in the uniform document template, is specifically used for: Create a file template; Customize the question and answer lists on the file template to obtain a unified file template; Import multiple validation questions and their corresponding standard answers into the question list and answer list of the unified file template, while verifying the integrity of the fields in the unified file template; If the verification shows that at least one line in the question list or answer list of the unified document template is empty or has a format error, a prompt message will be output. The prompt message is used to indicate that there are field errors in the question list or answer list in the unified document template, and that they need to be supplemented or corrected. The process continues until the unified document template is verified to have no empty fields and the format is correct, thus confirming that the fields in the unified document template are complete.

[0080] In one embodiment, the data import unit 210 is further configured to: Generate a unique identifier for each import of the unified file template; The entire process is recorded based on a unique identifier, documenting the operation trajectory for each imported verification question and its standard answer.

[0081] In one implementation, the two-layer verification unit 220, when retrieving structured JSON-formatted answers for each batch of verification questions from a large language model used to concurrently input multiple verification questions into the RAG knowledge base in batches, specifically performs the following: Count the number of lines N in a unified file template occupied by multiple validation questions; Following the approach of grouping multiple validation questions into batches of L, the validation questions are divided into M batches, where M = ; By using an asynchronous queue with backoff and retry functionality, each batch of validation questions is concurrently input into a large language model for retrieval, obtaining the structured JSON format answers for each batch of validation questions.

[0082] In one implementation, the two-layer verification unit 220, when using an asynchronous queue with backoff and retry functionality to sequentially input each batch of verification questions concurrently into a large language model for retrieval and obtain the structured JSON format answers for each batch of verification questions, specifically performs the following: Using an asynchronous queue, each batch of validation questions is concurrently input into the large language model according to a preset interval, and the structured JSON format answer for each set of validation questions is obtained. The preset interval is set based on the rate limiting threshold of the large language model.

[0083] In one implementation, the structured JSON format answer includes the retrieved answer for each validation question in each batch of validation questions; the two-layer validation unit 220, when calculating the similarity between the structured JSON format answer and the standard answer for each batch of validation questions in the uniform document template, and marking the correctness of the structured JSON format answer based on the similarity, is specifically used for: The search answer and its corresponding standard answer for each verification question in each batch of verification questions are uniformly input into the vector model, and the similarity between the search answer and its corresponding standard answer for each verification question is calculated. If the similarity between the search answer and the corresponding standard answer for each verification question exceeds a preset threshold, the search answer for each verification question will be marked as the correct answer; otherwise, the search answer for each verification question will be marked as the incorrect answer.

[0084] In one implementation, the visual review interface adopts a three-column layout, wherein the left column is used to display each verification question in each batch of verification questions, the right column is used to display the standard answer of each verification question, and the middle column is used to display the search answer of each verification question and its similarity with the standard answer. The visual review interface supports filtering and displaying verification issues based on similarity range and keywords.

[0085] The functions of each unit in the RAG knowledge base accuracy verification device of this application embodiment can be found in the corresponding description in the above method, and will not be repeated here.

[0086] Figure 5 A structural block diagram of a computer device according to an embodiment of this application is shown. Figure 5 As shown, the computer device includes a memory 310 and a processor 320. The memory 310 stores instructions, which are loaded and executed by the processor 320 to implement the RAG knowledge base accuracy verification method in the above embodiment. The number of memories 310 and processors 320 can be one or more.

[0087] The computer device also includes: The communication interface 330 is used to communicate with external devices and perform data exchange and transmission.

[0088] If the memory 310, processor 320, and communication interface 330 are implemented independently, they can be interconnected via a bus to communicate with each other. This bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. This bus can be divided into address bus, data bus, control bus, etc. For ease of representation, Figure 5 The bus is represented by a single thick line, but this does not mean that there is only one bus or one type of bus.

[0089] Optionally, in a specific implementation, if the memory 310, processor 320 and communication interface 330 are integrated on a single chip, the memory 310, processor 320 and communication interface 330 can communicate with each other through an internal interface.

[0090] This application provides a computer-readable storage medium storing a computer program. When the computer program is run on a computer, it implements the method provided in this application.

[0091] This application also provides a chip, which includes a processor for calling and executing instructions stored in a memory, causing a communication device with the chip installed to perform the method provided in this application.

[0092] This application also provides a chip, including: an input interface, an output interface, a processor, and a memory. The input interface, output interface, processor, and memory are connected through an internal connection path. The processor is used to execute code in the memory. When the code is executed, the processor is used to execute the method provided in the application embodiment.

[0093] It should be understood that the aforementioned processor can be a Central Processing Unit (CPU), or other general-purpose processors, Digital Signal Processors (DSPs), Application Specific Integrated Circuits (ASICs), Field-Programmable Gate Arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. General-purpose processors can be microprocessors or any conventional processor. It is worth noting that the processor can be a processor supporting Advanced Reduced Instruction Set Machines (ARM) architecture.

[0094] Further, optionally, the aforementioned memory may include read-only memory and random access memory, and may also include non-volatile random access memory. The memory may be volatile or non-volatile, or may include both. Non-volatile memory may include read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. Volatile memory may include random access memory (RAM), which serves as an external cache. Many forms of RAM are available by way of example, but not limitation. Examples include static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous linked dynamic random access memory (SLDRAM), and direct rambus RAM (DR RAM).

[0095] In the above embodiments, implementation can be achieved, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented in software, it can be implemented, in whole or in part, as a computer program product. A computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the flow or function according to this application is generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transferred from one computer-readable storage medium to another.

[0096] In the description of this specification, the references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of this application. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. Moreover, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification, as well as the features of those different embodiments or examples.

[0097] In the description of this application, "multiple" means two or more, unless otherwise expressly and specifically defined.

[0098] Any process or method description in the flowchart or otherwise herein can be understood as representing a module, segment, or portion of code comprising one or more executable instructions for implementing a particular logical function or process. Furthermore, the scope of the preferred embodiments of this application includes additional implementations in which functions may be performed not in the order shown or discussed, including substantially simultaneously or in reverse order depending on the functionality involved.

[0099] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as a sequenced list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by, or in conjunction with, an instruction execution system, apparatus or device (such as a computer-based system, a processor-included system or other system that can fetch and execute instructions from, an instruction execution system, apparatus or device).

[0100] It should be understood that various parts of this application can be implemented using hardware, software, firmware, or a combination thereof. In the above embodiments, multiple steps or methods can be implemented using software or firmware stored in memory and executed by a suitable instruction execution system. All or part of the steps of the methods in the above embodiments can be implemented by a program instructing related hardware, the program being stored in a computer-readable storage medium, which, when executed, includes one or a combination of the steps of the method embodiments.

[0101] Furthermore, the functional units in the various embodiments of this application can be integrated into a processing module, or each unit can exist physically separately, or two or more units can be integrated into a module. The integrated module can be implemented in hardware or as a software functional module. If the integrated module is implemented as a software functional module and sold or used as an independent product, it can also be stored in a computer-readable storage medium. This storage medium can be a read-only memory, a disk, or an optical disk, etc.

[0102] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any person skilled in the art can easily conceive of various variations or substitutions within the technical scope disclosed in this application, and these should all be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A method for checking the accuracy of a RAG knowledge base, characterized in that, The method comprises the following steps: Importing a plurality of verification questions and standard answers of each verification question into a unified file template and ensuring the field integrity of the unified file template; Batching and concurrently inputting the plurality of verification questions into a large language model of a RAG knowledge base for retrieval to obtain structured JSON format answers of each batch of verification questions, and then calculating the similarity between the structured JSON format answers and the standard answers of each batch of verification questions in the unified file template, and marking the correctness of the structured JSON format answers based on the similarity; Displaying the structured JSON format answers, the standard answers of each batch of verification questions and the similarity between them on a visual review interface, and providing a result determination button for each verification question in each batch of verification questions; In response to the verification operation of triggering the result determination button, updating the global accuracy of the RAG knowledge base.

2. The method of claim 1, wherein, The step of importing a plurality of verification questions and standard answers of each verification question into a unified file template and ensuring the field integrity of the unified file template comprises the following steps: Creating a file template; Customizing a question list and an answer list on the file template to obtain the unified file template; Correspondingly importing the plurality of verification questions and the standard answers of each verification question into the question list and the answer list of the unified file template, and checking the field integrity of the unified file template; If it is found that at least one row in the question list or the answer list of the unified file template is empty or has a format error, outputting a prompt information, wherein the prompt information is used to prompt that there is a field error in the question list or the answer list of the unified file template, and the question list or the answer list needs to be supplemented or corrected; Until there is no empty item in the unified file template and the format is correct, the field integrity of the unified file template is determined.

3. The method of claim 1, wherein, The method further comprises the following steps: Generating a unique identification code for each import of the unified file template; Based on the unique identification code, recording the operation track of the verification question and its standard answer for each import.

4. The method of claim 1, wherein, The step of batching and concurrently inputting the plurality of verification questions into a large language model of a RAG knowledge base for retrieval to obtain structured JSON format answers of each batch of verification questions comprises the following steps: Counting the number N of question rows occupied by the plurality of verification questions in the unified file template; In accordance with the manner of dividing L verification questions into batches, a plurality of the verification questions are divided into M batches, M ; Using an asynchronous queue with a back-off retry function to sequentially and concurrently input each batch of verification questions into the large language model for retrieval to obtain structured JSON format answers of each batch of verification questions.

5. The method of claim 4, wherein, The step of using an asynchronous queue with a back-off retry function to sequentially and concurrently input each batch of verification questions into the large language model for retrieval to obtain structured JSON format answers of each batch of verification questions comprises the following steps: Using the asynchronous queue to sequentially and concurrently input each batch of verification questions into the large language model according to a preset interval time length to obtain structured JSON format answers of each batch of verification questions, wherein the preset interval time length is set according to a flow limiting threshold of the large language model.

6. The method of claim 1, wherein, The structured JSON format answer includes the retrieved answer for each verification question in each batch of verification questions; calculating the similarity between the structured JSON format answer and the standard answer for each batch of verification questions in the unified file template, and marking the correctness of the structured JSON format answer based on the similarity includes: The retrieval answer and its corresponding standard answer for each verification question in each batch of verification questions are uniformly input into the vector model, and the similarity between the retrieval answer and its corresponding standard answer for each verification question is calculated. If the similarity between the search answer and the corresponding standard answer of each verification question exceeds a preset threshold, the search answer of each verification question is marked as the correct answer; otherwise, the search answer of each verification question is marked as the incorrect answer.

7. The method according to any one of claims 1 to 6, characterized in that, The visual review interface adopts a three-column layout. The left column is used to display each verification question in each batch of verification questions, the right column is used to display the standard answer of each verification question, and the middle column is used to display the search answer of each verification question and its similarity with the standard answer. The visual review interface supports filtering and displaying verification questions based on similarity range and keywords.

8. A device for checking the accuracy of a RAG knowledge base, characterized in that include: The data import unit is used to import multiple verification questions and the standard answer of each verification question into a unified file template, and to ensure the integrity of the fields in the unified file template; A two-layer verification unit is used to input multiple verification questions into the large language model of the RAG knowledge base in batches for retrieval, obtain the structured JSON format answer of each batch of verification questions, calculate the similarity between the structured JSON format answer and the standard answer of each batch of verification questions in the unified file template, and mark the correctness of the structured JSON format answer based on the similarity. The visual review unit is used to display the structured JSON format answers, the standard answers of each batch of verification questions, and the similarity between the two on the visual review interface, and to equip each verification question in each batch of verification questions with a result judgment button; The visual review unit is also used to update the global accuracy of the RAG knowledge base in response to the verification operation triggered by the result judgment button.

9. A computer apparatus, comprising: include: A memory and a processor, wherein the memory stores instructions which are loaded and executed by the processor to implement the method as described in any one of claims 1-7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when run on a computer, implements the method as described in any one of claims 1-7.

Images