Recall accuracy evaluation method, system, equipment, medium and product

Abstract

The invention discloses a recall accuracy evaluation method, system and device, a medium and a product, relates to the technical field of large model application, and discloses a recall accuracy evaluation method which comprises the steps that a preset knowledge base is divided into knowledge blocks, and questions corresponding to the knowledge blocks are generated through a large language model; the questions are input into the large language model, knowledge block tracing results corresponding to the questions are obtained, and the knowledge block tracing results are knowledge blocks recalled by the large language model based on the input questions; and automatically evaluating recall accuracy according to the knowledge blocks corresponding to the questions and the knowledge block traceability results corresponding to the questions. The technical scheme of the invention aims to solve the technical problem of how to reduce the cost required by evaluation when the recall accuracy is evaluated.

Description

Technical Field

[0001] The present application relates to the field of large model application technology, and in particular to a recall accuracy assessment method, system, device, medium and product. Background Art

[0002] Large language models are trained with massive amounts of data, using billions of parameters to generate raw output for tasks such as answering questions, translating languages, and completing sentences. Retrieval-augmented generation techniques are used to optimize the output of large language models, enabling them to reference knowledge bases outside of the training data source before generating answers. Building on the already powerful capabilities of large language models, retrieval-augmented generation techniques can extend them to access internal knowledge bases in specific domains or organizations. Specifically, the first step of retrieval-augmented generation techniques is to read external data, that is, new data outside of the original training dataset of the large language model. The new data can come from multiple data sources, such as APIs, databases, or document repositories. The second step is to convert the data into a digital representation and store it in a vector database through embedded language model technology. This process creates a knowledge base that a generative AI model can understand. The third step is to perform a relevance search, converting the user query into a vector representation and matching it with the vector database. For example, an application of retrieval-augmented generation techniques that can answer organizational human resources questions, if an employee searches: "How much annual leave do I have?", the system will retrieve annual leave policy documents and the employee's personal past leave records. These specific documents will be recalled because they are highly relevant to what the employee entered. Next, the retrieval-augmented generation technology model enhances the user input by adding the retrieved relevant data in context. Finally, the enhanced content is fed into the large language model to generate more relevant answers to the user query.

[0003] However, when injecting data search capabilities into large language model applications through retrieval enhancement generation technology, it is also necessary to measure the accuracy of the relevant recall content searched out. Nowadays, the accuracy of recall content is usually evaluated by human labor, but human evaluation requires multiple human participation and requires high costs.

[0004] Therefore, how to reduce the cost required for evaluation when evaluating recall accuracy is a technical problem that has yet to be solved by those skilled in the art. Summary of the invention

[0005] The main purpose of this application is to provide a recall accuracy assessment method, system, device, medium and product, aiming to solve the technical problem of how to reduce the cost required for assessment when assessing recall accuracy.

[0006] To achieve the above objectives, the present application proposes a recall accuracy assessment method, the method comprising:

[0007] Divide the preset knowledge base into various knowledge blocks, and generate questions corresponding to each of the knowledge blocks through a large language model;

[0008] Input each of the questions into the large language model respectively, and obtain the knowledge block tracing results corresponding to each of the questions, wherein the knowledge block tracing results are the knowledge blocks recalled by the large language model based on the input questions;

[0009] The recall accuracy is automatically evaluated based on the knowledge blocks corresponding to each of the questions and the tracing results of the knowledge blocks corresponding to each of the questions.

[0010] In one embodiment, the step of automatically evaluating the recall accuracy according to the knowledge blocks corresponding to the respective questions and the tracing results of the respective knowledge blocks includes:

[0011] For each of the above mentioned problems, the knowledge blocks corresponding to the same problem and the knowledge block tracing results are compared to obtain the knowledge block comparison results corresponding to each of the above mentioned problems;

[0012] According to the comparison results of each of the knowledge blocks, the hit rate and / or the average reciprocal ranking are calculated, and the recall accuracy is automatically evaluated according to the hit rate and / or the average reciprocal ranking.

[0013] In one embodiment, the step of dividing the preset knowledge base into various knowledge blocks includes:

[0014] Identify a data type in a preset knowledge base, wherein the data type is one or more;

[0015] According to the data type, the knowledge base is divided into various knowledge blocks.

[0016] In one embodiment, when it is detected that the number of the data types is one, the step of dividing the knowledge base into knowledge blocks according to the data types includes:

[0017] When it is detected that the data type is text, dividing the knowledge base into knowledge blocks according to chapters;

[0018] When it is detected that the data type is a table, each cell is taken as a knowledge block to obtain various knowledge blocks.

[0019] In one embodiment, when it is detected that the number of the data types is multiple, the step of dividing the knowledge base into knowledge blocks according to the data types includes:

[0020] According to each of the data types, the knowledge base is divided into a plurality of knowledge areas;

[0021] The division rules corresponding to each of the data types are obtained, and based on each of the division rules, each of the knowledge areas is divided into various knowledge blocks.

[0022] In one embodiment, the step of inputting each of the questions into the large language model to obtain the knowledge block tracing results corresponding to each of the questions includes:

[0023] Convert each of the knowledge blocks into a vector, and store each of the vectors in the RAG application;

[0024] Inputting each of the questions into the large language model respectively, and determining the target vector corresponding to each of the questions from the vectors stored in the RAG application through the large language model;

[0025] The knowledge blocks corresponding to the target vectors are used as the knowledge block tracing results corresponding to the problems, and the knowledge block tracing results are output.

[0026] In addition, to achieve the above purpose, the present application also proposes a recall accuracy evaluation system, the recall accuracy evaluation system comprising:

[0027] A question generation module is used to divide the preset knowledge base into various knowledge blocks, and generate questions corresponding to each of the knowledge blocks through a large language model;

[0028] A data recall module, used to input each of the questions into the large language model, and obtain the knowledge block tracing results corresponding to each of the questions, wherein the knowledge block tracing results are the knowledge blocks recalled by the large language model based on the input questions;

[0029] The automatic evaluation module is used to automatically evaluate the recall accuracy according to the knowledge blocks corresponding to each of the questions and the tracing results of the knowledge blocks corresponding to each of the questions.

[0030] In addition, to achieve the above-mentioned purpose, the present application also proposes a recall accuracy assessment device, which includes: a memory, a processor, and a computer program stored on the memory and executable on the processor, and the computer program is configured to implement the steps of the recall accuracy assessment method described above.

[0031] In addition, to achieve the above-mentioned purpose, the present application also proposes a storage medium, which is a computer-readable storage medium, and a computer program is stored on the storage medium. When the computer program is executed by a processor, the steps of the recall accuracy evaluation method described above are implemented.

[0032] In addition, to achieve the above-mentioned purpose, the present application also provides a computer program product, which includes a computer program, and when the computer program is executed by a processor, the steps of the recall accuracy evaluation method described above are implemented.

[0033] In the present application, by dividing the preset knowledge base into various knowledge blocks, and generating the corresponding questions for each knowledge block through a large language model, a correspondence between knowledge blocks and questions can be established; then each of the questions is input into the large language model respectively to obtain the knowledge block tracing results corresponding to each of the questions, wherein the knowledge block tracing results are the knowledge blocks recalled by the large language model based on the input questions, and the correspondence between the questions and the recalled knowledge blocks can be obtained based on the large language model; finally, according to the knowledge blocks corresponding to each of the questions and the corresponding knowledge block tracing results, the recall accuracy is automatically evaluated, and the recall accuracy of the large language model can be evaluated based on the correspondence between the questions and the knowledge blocks, and between the questions and the recalled knowledge blocks.

[0034] In this way, the present application can automatically evaluate the recall accuracy without manual intervention, thereby reducing the cost required to evaluate the recall accuracy. In addition, the present application also divides the knowledge base into blocks so that the correspondence between the question and the knowledge block is clear, and then compares the knowledge block with the knowledge block traceability result so that the obtained recall result is necessarily accurate or inaccurate, thereby improving the accuracy of the recall accuracy evaluation as a whole. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0036] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0037] Figure 1 A flowchart of the first embodiment of the method for evaluating the accuracy of recall of this application is provided;

[0038] Figure 2 A flowchart diagram of a specific embodiment of a method for evaluating the accuracy of recall of this application is provided;

[0039] Figure 3 A flowchart diagram of another embodiment of the recall accuracy assessment method of the present application is provided;

[0040] Figure 4 This is a schematic diagram of the module structure of the recall accuracy assessment system according to an embodiment of the present application;

[0041] Figure 5 Schematic diagram of the device structure of the hardware operating environment involved in the recall accuracy assessment method in the embodiment of the present application.

[0042] The purpose, features and advantages of this application will be further described in conjunction with the embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION

[0043] It should be understood that the specific embodiments described herein are only used to explain the technical solutions of the present application and are not used to limit the present application.

[0044] In order to better understand the technical solution of the present application, a detailed description will be given below in conjunction with the accompanying drawings and specific implementation methods.

[0045] The main solution of the embodiment of the present application is: divide the preset knowledge base into various knowledge blocks, and generate questions corresponding to each knowledge block through a large language model; input each question into the large language model respectively to obtain the knowledge block tracing results corresponding to each question, wherein the knowledge block tracing results are the knowledge blocks recalled by the large language model based on the input questions; automatically evaluate the recall accuracy according to the knowledge blocks corresponding to each question and the corresponding knowledge block tracing results.

[0046] In the following embodiments, for ease of description, the recall accuracy assessment device is used as the execution subject for explanation. Specifically, the recall accuracy assessment device can be a terminal such as a server or a personal computer. This application does not limit this.

[0047] Understandably, large language models are trained with massive amounts of data and use billions of parameters to generate raw output for tasks such as answering questions, translating languages, and completing sentences. Retrieval-augmented generation techniques are used to optimize large language model outputs, enabling them to reference knowledge bases outside of the training data source before generating responses. Building on the already powerful capabilities of large language models, retrieval-augmented generation techniques can extend them to access internal knowledge bases in specific domains or organizations. Specifically, the first step of retrieval-augmented generation techniques is to read external data, i.e., new data outside of the original training dataset of the large language model. The new data can come from multiple data sources, such as APIs, databases, or document repositories. The second step is to convert the data into a digital representation and store it in a vector database through embedded language model technology. This process creates a knowledge base that a generative AI model can understand. The third step is to perform a relevance search, converting the user query into a vector representation and matching it with the vector database. For example, a retrieval-augmented generation technology application that can answer organizational human resources questions, if an employee searches: "How much annual leave do I have?", the system will retrieve the annual leave policy document and the employee's personal past leave records. These specific documents will be recalled because they are highly relevant to what the employee entered. Next, the retrieval-augmented generation technology model enhances the user input by adding the retrieved relevant data in context. Finally, the enhanced content is fed into the large language model to generate more relevant answers to the user query.

[0048] However, when injecting data search capabilities into large language model applications through retrieval enhancement generation technology, it is also necessary to measure the accuracy of the relevant recall content searched out. Nowadays, the accuracy of recall content is usually evaluated by human labor, but human evaluation requires multiple human participation and requires high costs.

[0049] Therefore, how to reduce the cost required for evaluation when evaluating recall accuracy is a technical problem that has yet to be solved by those skilled in the art.

[0050] To solve the above problems, this application proposes a recall accuracy evaluation method.

[0051] Please refer to Figure 1 , Figure 1 This is a flow chart of the first embodiment of the recall accuracy assessment method of the present application.

[0052] In this embodiment, the recall accuracy evaluation method includes steps S10 to S30:

[0053] Step S10, dividing the preset knowledge base into various knowledge blocks, and generating questions corresponding to each of the knowledge blocks through a large language model;

[0054] It should be noted that, in this embodiment, the knowledge base is only used to evaluate the recall accuracy. Therefore, this application does not limit the content of the knowledge base. It should also be noted that the knowledge block refers to the block obtained after the knowledge base is divided.

[0055] In this embodiment, in order to establish a corresponding relationship between questions and knowledge blocks, the knowledge base can be divided into multiple blocks, that is, multiple knowledge blocks are obtained. Then, for each knowledge block, one or more questions can be generated. The questions are used to input the large language model to evaluate the recall accuracy.

[0056] It is understandable that the technical solution for the content generation problem in each knowledge block is an existing mature technology, and this application does not impose any limitation on this.

[0057] Step S20, inputting each of the questions into the large language model respectively, and obtaining knowledge block tracing results corresponding to each of the questions, wherein the knowledge block tracing results are knowledge blocks recalled by the large language model based on the input questions;

[0058] It is understandable that after the question is input into the large language model, the large language model can infer the knowledge block associated with the input question from the divided knowledge blocks and return the knowledge block. In this embodiment, the knowledge block tracing result is the returned knowledge block. Thus, based on the large language model, the knowledge block tracing result corresponding to each question can be obtained.

[0059] Step S30, automatically evaluating the recall accuracy according to the knowledge blocks corresponding to each of the questions and the tracing results of the knowledge blocks corresponding to each of the questions.

[0060] In this embodiment, after obtaining the knowledge block tracing results corresponding to each question, the knowledge block tracing results can be compared with the knowledge blocks used when the questions were generated, thereby evaluating the recall accuracy.

[0061] In a feasible implementation manner, the above step S20 further includes:

[0062] Step S201, converting each of the knowledge blocks into a vector, and storing each of the vectors in a RAG application;

[0063] It should be noted that, since the large language model cannot understand the knowledge base or knowledge block, after obtaining the knowledge block, it is necessary to convert the knowledge block into a data format that the large language model can understand. In this embodiment, the knowledge block can be converted into a vector and then the vector is stored. In addition, the RAG application is an application developed using the RAG technology. In this embodiment, the RAG application is used to store the vector obtained by converting the knowledge block.

[0064] Step S202, inputting each of the questions into the large language model respectively, and determining the target vector corresponding to each of the questions from the vectors stored in the RAG application through the large language model;

[0065] It can be understood that the target vector refers to the vector corresponding to the currently input question among the various vectors.

[0066] For example, after the input question is converted into a vector, it will be compared with the vectors stored in the RAG application, and then the vector with the highest similarity can be used as the target vector.

[0067] In a feasible implementation, the number of target vectors can be one or more. When the number of target vectors is one, the target vector has the highest similarity with the vector converted from the input question. When the number of target vectors is multiple, the vectors corresponding to the knowledge blocks can be sorted according to the similarity, so that multiple target vectors can be output for a question.

[0068] Step S203, taking the knowledge blocks corresponding to the target vectors as the knowledge block tracing results corresponding to the questions, and outputting the knowledge block tracing results.

[0069] In this embodiment, by converting knowledge blocks into vectors, it is not only easier for the large language model to understand questions and knowledge blocks, but also possible to quickly determine the target vector based on similarity, thereby reducing the time required to determine the target vector and improving the efficiency of evaluating recall accuracy.

[0070] In a feasible implementation manner, the above step S30 further includes:

[0071] Step S301, for each of the questions, comparing the knowledge blocks corresponding to the same question with the knowledge block tracing results, to obtain the knowledge block comparison results corresponding to each of the questions;

[0072] It is understandable that when the target vector is one, the knowledge block comparison result is one of "the knowledge block tracing result is the same as the knowledge block that generates the question" and "the knowledge block tracing result is not the same as the knowledge block that generates the question". When the target vector is multiple, the knowledge block comparison result is one of "the knowledge block tracing result includes the knowledge block that generates the question" and "the knowledge block tracing result does not include the knowledge block that generates the question". Among them, when the knowledge block comparison result is "the knowledge block tracing result includes the knowledge block that generates the question", the ranking of each knowledge block in the knowledge block tracing result can also be determined based on the similarity, and then the ranking of the knowledge block that generates the question in the knowledge tracing result can be obtained. It is understandable that when there are multiple target vectors, determining the ranking of the knowledge block that generates the question in the knowledge tracing result can comprehensively evaluate the recall accuracy of the large language model, and can avoid the problem of one-sided accuracy when the target vector is one.

[0073] Step S302, calculating the hit rate and / or the average reciprocal ranking according to the comparison results of each of the knowledge blocks, and automatically evaluating the recall accuracy according to the hit rate and / or the average reciprocal ranking.

[0074] It should be noted that recall accuracy is generally measured by two indicators: Hit Rate and Mean Rec i procedure Rank (MRR). Among them, the hit rate calculates the proportion of questions that contain correct answers in the first k recalled knowledge blocks, that is, how many times the first k searched items in the question and answer hit the correct answer. The mean reciprocal rank evaluates the performance of the retrieval system by the ranking of the correct retrieval result value in the retrieval results. For example, if the first relevant document is the first searched, the reciprocal rank is 1; if it is the second searched, the reciprocal rank is 1 / 2, and so on. Finally, the reciprocal rankings of different questions and answers are averaged.

[0075] In this embodiment, by calculating the hit rate and the inverse of the average ranking through the knowledge block comparison results, the recall accuracy can be intuitively displayed through data, thereby improving the user experience.

[0076] In an embodiment of the present application, by dividing a preset knowledge base into various knowledge blocks, and generating corresponding questions for each of the knowledge blocks through a large language model, a correspondence between knowledge blocks and questions can be established; then each of the questions is input into the large language model respectively to obtain a knowledge block tracing result corresponding to each of the questions, wherein the knowledge block tracing result is a knowledge block recalled by the large language model based on the input question, and the correspondence between the question and the recalled knowledge block can be obtained based on the large language model; finally, based on the knowledge blocks corresponding to each of the questions and the corresponding knowledge block tracing results, the recall accuracy is automatically evaluated, and the recall accuracy of the large language model can be evaluated based on the correspondence between the question and the knowledge block, and between the question and the recalled knowledge block.

[0077] In this way, the present application can automatically evaluate the recall accuracy without manual intervention, thereby reducing the cost required to evaluate the recall accuracy. In addition, the present application also divides the knowledge base into blocks so that the correspondence between the question and the knowledge block is clear, and then compares the knowledge block with the knowledge block traceability result so that the obtained recall result is necessarily accurate or inaccurate, thereby improving the accuracy of the recall accuracy calculation as a whole.

[0078] Furthermore, based on the above-mentioned first embodiment of the recall accuracy assessment method of the present application, a second embodiment of the recall accuracy assessment method of the present application is proposed.

[0079] In this embodiment, the above step S10 further includes:

[0080] Step S101, identifying a data type in a preset knowledge base, wherein the data type is one or more;

[0081] It is understandable that after acquiring the knowledge base, in order to evaluate the recall accuracy based on the limited content in the knowledge base, the data type of the content in the knowledge base can be identified so as to be divided according to the data type. Specifically, the data type can be text, table, image, etc.

[0082] Step S102: dividing the knowledge base into various knowledge blocks according to the data types.

[0083] It is understandable that different knowledge bases usually store different contents, and a knowledge base usually stores multiple data types. For example, a knowledge base can contain text, tables, and images at the same time. Therefore, in order to fully utilize the contents in the knowledge base, the contents belonging to the same data type can be divided into a knowledge block, so that each knowledge block can be obtained.

[0084] Specifically, the above step S102 includes:

[0085] Step S1021, dividing the knowledge base into a plurality of knowledge areas according to each of the data types;

[0086] It should be noted that in addition to directly dividing the content of the same data type into an area and then using the area as a knowledge block, you can also divide the knowledge block content belonging to the same data type into the same area according to the data type, and then divide the content in the same area to obtain knowledge blocks.

[0087] Step S1022, obtaining the division rules corresponding to each of the data types, and dividing each of the knowledge areas into knowledge blocks based on the division rules.

[0088] It is understandable that different division rules can be preset for different data types, so that the content of different data types can be fully divided.

[0089] In a feasible implementation manner, when it is detected that the number of the data types is one, the above step S102 further includes:

[0090] Step S1023, when it is detected that the data type is text, the knowledge base is divided into various knowledge blocks according to chapters;

[0091] Step S1024, when it is detected that the data type is a table, each cell is regarded as a knowledge block to obtain individual knowledge blocks.

[0092] It is worth mentioning that in the present application, for the table, the cell division method can be adopted to divide it into multiple knowledge blocks, so that the recall accuracy can be evaluated with the limited content in the table.

[0093] For example, for a table, the code can be used to traverse the cells to generate the question "What is the number in the row and column?", and then the recall accuracy can be evaluated by evaluating whether the correct value is returned.

[0094] In this embodiment, the present application divides the text content by chapters and divides the table by cells, which can increase the number of generated questions and the amount of test data used to evaluate recall accuracy, thereby more accurately evaluating recall accuracy.

[0095] Furthermore, based on the first embodiment and the second embodiment of the recall accuracy evaluation method of the present application described above, two specific embodiments of the recall accuracy evaluation method of the present application are proposed.

[0096] Please refer to Figure 2,When the knowledge base content is a table, each cell can be obtained by traversing, and then for each cell, the question of "what is the number in the nth row and nth column" can be generated, and then the question can be input into the large language model to evaluate the recall accuracy based on the answer of the large language model.

[0097] Please refer to Figure 3 When the knowledge base content is text, the text can be divided into blocks, and then the questions corresponding to each text block are generated through the large language model. At the same time, the text block is also converted into a vector and stored in the RAG application so that it can be recalled through the RAG application. Therefore, based on a knowledge base with only text, the recall accuracy can be evaluated while reducing costs.

[0098] It should also be noted that when answering questions with a large language model, data on whether the recall result is relevant to the question will be generated. Multiple questions and answers can be used to calculate various indicators of search recall evaluation. For a table, the total number of tests is the product of rows and columns; if the test unit is expanded from cells to row and column ranges, model capabilities such as averaging, finding the maximum value, and data analysis and comparison can also be evaluated. For text, you can define different block lengths and generate several questions per block to adjust the number of tests, so that this application can be suitable for different application scenarios.

[0099] In addition, since the application can generate test data without introducing additional manual and model links, it is very suitable for the cold start of the RAG application early evaluation system. Moreover, after being put into the production environment, the RAG application will face a constantly changing knowledge base. The application also supports flexible sampling in the knowledge base and convenient tracking of the parameters used in each evaluation through the code, so as to adjust the evaluation process, thereby further improving the evaluation efficiency.

[0100] It should be noted that the above examples are only used to understand the present application and do not constitute a limitation on the recall accuracy assessment method of the present application. More simple transformations based on this technical concept are all within the scope of protection of the present application.

[0101] This application also provides a recall accuracy assessment system, please refer to Figure 4 , the recall accuracy evaluation system comprises:

[0102] The question generation module 10 is used to divide the preset knowledge base into various knowledge blocks, and generate questions corresponding to each of the knowledge blocks through a large language model;

[0103] A data recall module 20 is used to input each of the questions into the large language model to obtain knowledge block tracing results corresponding to each of the questions, wherein the knowledge block tracing results are knowledge blocks recalled by the large language model based on the input questions;

[0104] The automatic evaluation module 30 is used to automatically evaluate the recall accuracy according to the knowledge blocks corresponding to each of the questions and the corresponding knowledge block tracing results.

[0105] In one embodiment, the automatic assessment module 30 is further configured to:

[0106] For each of the above mentioned problems, the knowledge blocks corresponding to the same problem and the knowledge block tracing results are compared to obtain the knowledge block comparison results corresponding to each of the above mentioned problems;

[0107] According to the comparison results of each of the knowledge blocks, the hit rate and / or the average reciprocal ranking are calculated, and the recall accuracy is automatically evaluated according to the hit rate and / or the average reciprocal ranking.

[0108] In one embodiment, the question generating module 10 is further used for:

[0109] Identify a data type in a preset knowledge base, wherein the data type is one or more;

[0110] According to the data type, the knowledge base is divided into various knowledge blocks.

[0111] In one embodiment, when it is detected that the number of the data types is one, the question generating module 10 is further configured to:

[0112] When it is detected that the data type is text, dividing the knowledge base into knowledge blocks according to chapters;

[0113] When it is detected that the data type is a table, each cell is taken as a knowledge block to obtain various knowledge blocks.

[0114] In one embodiment, when it is detected that the number of the data types is multiple, the question generating module 10 is further configured to:

[0115] According to each of the data types, the knowledge base is divided into a plurality of knowledge areas;

[0116] The division rules corresponding to each of the data types are obtained, and based on each of the division rules, each of the knowledge areas is divided into various knowledge blocks.

[0117] In one embodiment, the data recall module 20 is further used to:

[0118] Convert each of the knowledge blocks into a vector, and store each of the vectors in the RAG application;

[0119] Inputting each of the questions into the large language model respectively, and determining the target vector corresponding to each of the questions from the vectors stored in the RAG application through the large language model;

[0120] The knowledge blocks corresponding to the target vectors are used as the knowledge block tracing results corresponding to the problems, and the knowledge block tracing results are output.

[0121] The recall accuracy evaluation system provided by the present application adopts the recall accuracy evaluation method in the above embodiment, which can solve the technical problem of how to reduce the cost required for evaluation when evaluating recall accuracy. Compared with the prior art, the beneficial effects of the recall accuracy evaluation system provided by the present application are the same as the beneficial effects of the recall accuracy evaluation method provided by the above embodiment, and other technical features in the recall accuracy evaluation system are the same as the features disclosed in the above embodiment method, which will not be repeated here.

[0122] The present application provides a recall accuracy assessment device, which includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the recall accuracy assessment method in the above-mentioned embodiment one.

[0123] Reference below Figure 5 , which shows a schematic structural diagram of a recall accuracy assessment device suitable for implementing an embodiment of the present application. Figure 5 The recall accuracy assessment device shown is merely an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.

[0124] like Figure 5As shown, the recall accuracy evaluation device may include a processing device 1001 (e.g., a central processing unit, a graphics processing unit, etc.), which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM: Read On ly Memory) 1002 or a program loaded from a storage device 1003 to a random access memory (RAM: Random Access Memory) 1004. In RAM1004, various programs and data required for the operation of the recall accuracy evaluation device are also stored. The processing device 1001, ROM1002, and RAM1004 are connected to each other via a bus 1005. An input / output (I / O) interface 1006 is also connected to the bus. Typically, the following systems may be connected to the I / O interface 1006: an input device 1007 including, for example, a touch screen, a touchpad, a keyboard, a mouse, an image sensor, a microphone, an accelerometer, a gyroscope, etc.; an output device 1008 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; a storage device 1003 including, for example, a magnetic tape, a hard disk, etc.; and a communication device 1009. The communication device 1009 may allow the recall accuracy assessment device to communicate wirelessly or wired with other devices to exchange data. Although the figure shows a recall accuracy assessment device with various systems, it should be understood that it is not required to implement or have all the systems shown. More or fewer systems may be implemented or have instead.

[0125] In particular, according to the embodiments disclosed in the present application, the process described above with reference to the flowchart can be implemented as a computer software program. For example, the embodiments disclosed in the present application include a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network through a communication device, or installed from a storage device 1003, or installed from a ROM 1002. When the computer program is executed by the processing device 1001, the above-mentioned functions defined in the method of the embodiment disclosed in the present application are executed.

[0126] The recall accuracy assessment device provided by the present application adopts the recall accuracy assessment method in the above embodiment, which can solve the technical problem of how to reduce the cost required for assessment when assessing recall accuracy. Compared with the prior art, the beneficial effects of the recall accuracy assessment device provided by the present application are the same as the beneficial effects of the recall accuracy assessment method provided by the above embodiment, and the other technical features in the recall accuracy assessment device are the same as the features disclosed in the method of the previous embodiment, which will not be repeated here.

[0127] It should be understood that the various parts disclosed in this application can be implemented by hardware, software, firmware or a combination thereof. In the description of the above embodiments, specific features, structures, materials or characteristics can be combined in any one or more embodiments or examples in a suitable manner.

[0128] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art who is familiar with the present technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

[0129] The present application provides a computer-readable storage medium having computer-readable program instructions (ie, computer programs) stored thereon, and the computer-readable program instructions are used to execute the recall accuracy assessment method in the above-mentioned embodiment.

[0130] The computer-readable storage medium provided in the present application may be, for example, a USB flash drive, but is not limited to electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, systems or devices, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM: Random Access Memory), a read-only memory (ROM: Read On ly Memory), an erasable programmable read-only memory (EPROM: Erasable Programmab le Read On ly Memory or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM: CD-Read On ly Memory), an optical storage device, a magnetic storage device, or any suitable combination of the above. In this embodiment, the computer-readable storage medium may be any tangible medium containing or storing a program, which may be used by or in combination with an instruction execution system, system or device. The program code contained on the computer-readable storage medium may be transmitted using any appropriate medium, including but not limited to: wires, optical cables, RF (Radio Frequency: Radio Frequency), etc., or any suitable combination of the above.

[0131] The computer-readable storage medium may be included in the recall accuracy assessment device; or may exist independently without being assembled into the recall accuracy assessment device.

[0132] The above-mentioned computer-readable storage medium carries one or more programs. When the above-mentioned one or more programs are executed by the recall accuracy assessment device, the recall accuracy assessment device: divides the preset knowledge base into various knowledge blocks, and generates questions corresponding to each of the knowledge blocks through a large language model; inputs each of the questions into the large language model respectively, and obtains the knowledge block tracing results corresponding to each of the questions, wherein the knowledge block tracing results are the knowledge blocks recalled by the large language model based on the input questions; and automatically evaluates the recall accuracy according to the knowledge blocks corresponding to each of the questions and the corresponding knowledge block tracing results.

[0133] The computer program code for performing the operation of the present application can be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages, such as Java, Smalltalk, C++, and conventional procedural programming languages, such as "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as an independent software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer can be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or can be connected to an external computer (for example, using an Internet service provider to connect through the Internet).

[0134] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present application. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0135] The modules involved in the embodiments described in this application may be implemented by software or hardware, wherein the name of the module does not constitute a limitation on the unit itself in some cases.

[0136] The readable storage medium provided in the present application is a computer-readable storage medium, which stores computer-readable program instructions (i.e., computer programs) for executing the above-mentioned recall accuracy evaluation method, and can solve the technical problem of how to reduce the cost required for evaluation when evaluating recall accuracy. Compared with the prior art, the beneficial effects of the computer-readable storage medium provided in the present application are the same as the beneficial effects of the recall accuracy evaluation method provided in the above-mentioned embodiment, and will not be repeated here.

[0137] The present application also provides a computer program product, including a computer program, which implements the steps of the recall accuracy assessment method as described above when executed by a processor.

[0138] The computer program product provided by the present application can solve the technical problem of how to reduce the cost required for evaluation when evaluating recall accuracy. Compared with the prior art, the beneficial effects of the computer program product provided by the present application are the same as the beneficial effects of the recall accuracy evaluation method provided by the above embodiment, which will not be repeated here.

[0139] The above descriptions are only some embodiments of the present application, and are not intended to limit the patent scope of the present application. All equivalent structural changes made using the contents of the present application specification and drawings under the technical concept of the present application, or direct / indirect applications in other related technical fields are included in the patent protection scope of the present application.

Claims

1. A recall accuracy evaluation method, characterized in that: The recall accuracy evaluation method includes: Divide the preset knowledge base into various knowledge blocks, and generate questions corresponding to each of the knowledge blocks through a large language model; Input each of the questions into the large language model respectively, and obtain the knowledge block tracing results corresponding to each of the questions, wherein the knowledge block tracing results are the knowledge blocks recalled by the large language model based on the input questions; The recall accuracy is automatically evaluated based on the knowledge blocks corresponding to each of the questions and the tracing results of the knowledge blocks corresponding to each of the questions.

2. The recall accuracy evaluation method according to claim 1, characterized in that: The step of automatically evaluating the recall accuracy according to the knowledge blocks corresponding to the respective questions and the tracing results of the respective knowledge blocks includes: For each of the above mentioned problems, the knowledge blocks corresponding to the same problem and the knowledge block tracing results are compared to obtain the knowledge block comparison results corresponding to each of the above mentioned problems; According to the comparison results of each of the knowledge blocks, the hit rate and / or the average reciprocal ranking are calculated, and the recall accuracy is automatically evaluated according to the hit rate and / or the average reciprocal ranking.

3. The recall accuracy evaluation method according to claim 1, characterized in that: The step of dividing the preset knowledge base into various knowledge blocks includes: Identify a data type in a preset knowledge base, wherein the data type is one or more; According to the data type, the knowledge base is divided into various knowledge blocks.

4. The recall accuracy evaluation method according to claim 3, characterized in that: When it is detected that the number of the data types is one, the step of dividing the knowledge base into knowledge blocks according to the data types includes: When it is detected that the data type is text, dividing the knowledge base into knowledge blocks according to chapters; When it is detected that the data type is a table, each cell is taken as a knowledge block to obtain various knowledge blocks.

5. The recall accuracy evaluation method according to claim 3, characterized in that: When it is detected that the number of the data types is multiple, the step of dividing the knowledge base into knowledge blocks according to the data types includes: According to each of the data types, the knowledge base is divided into a plurality of knowledge areas; The division rules corresponding to each of the data types are obtained, and based on each of the division rules, each of the knowledge areas is divided into various knowledge blocks.

6. The recall accuracy evaluation method according to claim 1, characterized in that: The step of inputting each of the questions into the large language model to obtain the knowledge block tracing results corresponding to each of the questions includes: Convert each of the knowledge blocks into a vector, and store each of the vectors in the RAG application; Inputting each of the questions into the large language model respectively, and determining the target vector corresponding to each of the questions from the vectors stored in the RAG application through the large language model; The knowledge blocks corresponding to the target vectors are used as the knowledge block tracing results corresponding to the problems, and the knowledge block tracing results are output.

7. A recall accuracy assessment system, characterized in that: The recall accuracy evaluation system comprises: A question generation module is used to divide the preset knowledge base into various knowledge blocks, and generate questions corresponding to each of the knowledge blocks through a large language model; A data recall module, used to input each of the questions into the large language model, and obtain the knowledge block tracing results corresponding to each of the questions, wherein the knowledge block tracing results are the knowledge blocks recalled by the large language model based on the input questions; The automatic evaluation module is used to automatically evaluate the recall accuracy according to the knowledge blocks corresponding to each of the questions and the tracing results of the knowledge blocks corresponding to each of the questions.

8. A recall accuracy assessment device, characterized in that: The recall accuracy assessment device comprises: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program is configured to implement the steps of the recall accuracy assessment method according to any one of claims 1 to 6.

9. A storage medium, characterized in that: The storage medium is a computer-readable storage medium, and a computer program is stored on the storage medium. When the computer program is executed by a processor, the steps of the recall accuracy assessment method according to any one of claims 1 to 6 are implemented.

10. A computer program product, characterized in that The computer program product comprises a computer program, and when the computer program is executed by a processor, the steps of the recall accuracy assessment method according to any one of claims 1 to 6 are implemented.

Images