Knowledge Base Evaluation Method, Device and Storage Medium

Abstract

This application relates to the field of artificial intelligence technology, and discloses a knowledge base evaluation method, device, and storage medium. The knowledge base evaluation method includes: for each combination of i, j, and k, obtaining the generated answer of the i-th knowledge base for the k-th question based on the j-th reference document, and comprehensively scoring the generated answer to obtain a scoring matrix of I knowledge bases; based on the scoring matrix, determining the qualified knowledge bases with qualified learning ability from the I knowledge bases; based on each qualified knowledge base, calculating the sum of squared residuals and the standard deviation of all comprehensive scores corresponding to the qualified knowledge base, and based on the sum of squared residuals and the standard deviation, determining the evaluation result of the qualified knowledge base. The method of this application can accurately measure the accuracy, response stability, and evaluation consistency of each knowledge base by analyzing the sum of squared residuals and the standard deviation when the knowledge base processes reference documents.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and in particular to a knowledge base evaluation method, device, and storage medium. Background Art

[0002] With the rapid development of Large Language Model (LLM) technology, artificial intelligence (AI) knowledge base systems have become a crucial platform for information management and knowledge sharing. By integrating diverse enterprise data and information, AI knowledge base systems provide intelligent search, recommendation, and question-and-answer services, significantly improving employee productivity and decision-making quality. However, evaluating the accuracy of AI knowledge base systems' questions and answers, as well as other performance metrics, remains challenging. Effectively evaluating and optimizing AI knowledge base systems is crucial to ensuring their efficient operation and meeting user needs.

[0003] Existing technical solutions generally use evaluation methods based on public datasets to judge the performance of AI knowledge base systems. These evaluation methods mostly focus on quantitative analysis (such as accuracy and recall). These methods cannot fully reflect the performance of AI systems in actual applications, especially in terms of user experience and personalized needs. Summary of the Invention

[0004] The embodiments of the present application aim to provide a knowledge base evaluation method, device, and storage medium to address the problem that the existing methods for evaluating knowledge bases cannot fully reflect the performance of AI systems in actual applications.

[0005] To solve the above technical problems, the embodiments of the present application provide the following technical solutions:

[0006] According to a first aspect of the present application, a knowledge base evaluation method is provided, the method comprising:

[0007] for 、 and For each combination of The knowledge base is based on Reference Generate answers to questions , and generate answers to the Perform comprehensive scoring to obtain the scoring matrix of K questions corresponding to I knowledge base based on J references , where I, J and K are positive integers, =1, …, I, =1, …, J, =1,…,K;

[0008] Based on the scoring matrix , determine a qualified knowledge base with qualified learning ability from I knowledge bases;

[0009] Based on each qualified knowledge base, the residual sum of squares and the standard deviation of all comprehensive scores corresponding to the qualified knowledge base are calculated, and the evaluation result of the qualified knowledge base is determined based on the residual sum of squares and the standard deviation.

[0010] Optionally, the generated answer Comprehensive scoring includes:

[0011] Get the generated answer Corresponding reference answers , generate the answer Scoring is performed based on completeness, matching and relevance to obtain the generated answer Completeness score, matching score and relevance score;

[0012] Generate answers based on The completeness score, matching score and relevance score of 's comprehensive rating.

[0013] Optionally, the generated answer Scoring based on the completeness, the generated answer is obtained The completeness score includes:

[0014] Generate an answer to the Perform word segmentation processing to generate a word segmentation result set;

[0015] The reference answer Perform word segmentation processing to obtain a reference word segmentation result set;

[0016] Calculate the intersection of the generated word segmentation result set and the reference word segmentation result set, divide the size of the intersection by the size of the reference word segmentation result set, and obtain the generated answer Completeness rating.

[0017] Optionally, the generated answer Scoring is performed based on the matching degree to obtain the generated answer The compatibility score includes:

[0018] Generate an answer to the Perform word segmentation processing to generate a word segmentation result set;

[0019] The reference answer Perform word segmentation processing to obtain a reference word segmentation result set;

[0020] Calculate the intersection and union of the generated word segmentation result set and the reference word segmentation result set, and divide the size of the intersection by the size of the union to obtain the generated answer 's matching degree score.

[0021] Optionally, the scoring of the generated answer from the relevance to obtain the relevance score of the generated answer includes:

[0022] Perform an embedding vector conversion on the generated answer to obtain a generated embedding vector;

[0023] Perform an embedding vector conversion on the reference answer to obtain a reference embedding vector;

[0024] Calculate the cosine similarity between the generated embedding vector and the reference embedding vector;

[0025] Normalize the cosine similarity to obtain the relevance score of the generated answer .

[0026] Optionally, the determining of the qualified knowledge bases with qualified learning ability from I knowledge bases based on the scoring matrix includes:

[0027] Based on the scoring matrix calculate the average literature score of each knowledge base based on each reference document ;

[0028] According to the average literature score and the literature learning threshold corresponding to each reference document determine the qualified knowledge bases with qualified learning ability from I knowledge bases.

[0029] Optionally, the determining of the qualified knowledge bases with qualified learning ability from I knowledge bases according to the average literature score and the literature learning threshold corresponding to each reference document includes:

[0030] Based on each value corresponding knowledge base, if the average literature score of the knowledge base based on the reference documents corresponding to each value is greater than the literature learning threshold of the reference document , then determine that the knowledge base is a qualified knowledge base with qualified learning ability.

[0031] Optionally, determining the evaluation result of the qualified knowledge base based on the sum of squared residuals and the standard deviation includes:

[0032] If the sum of squared residuals of the qualified knowledge base is less than a preset sum-of-squared-residuals threshold and the standard deviation of the qualified knowledge base is less than a preset standard-deviation threshold, it is determined that both the accuracy and stability of the qualified knowledge base meet the expectations;

[0033] If the sum of squared residuals of the qualified knowledge base is greater than or equal to the sum-of-squared-residuals threshold, or the standard deviation of the qualified knowledge base is greater than or equal to the standard-deviation threshold, it is determined that the accuracy and / or stability of the qualified knowledge base do not meet the expectations.

[0034] According to a second aspect of the present application, there is provided an electronic device, including at least one processor and a memory communicatively connected to the at least one processor, where the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the knowledge base evaluation method described above. [[ID=eleven]]

[0035] According to a third aspect of the present application, there is provided a computer storage medium storing instructions or programs, and when the instructions or programs are executed by at least one processor, the at least one processor is caused to execute the knowledge base evaluation method described above.

[0036] The beneficial effects of the embodiments of the present application are as follows: Different from the prior art, in the embodiments of the present application, a knowledge base evaluation method is provided. First, a comprehensive score is obtained for the answers generated by each knowledge base for each question of each reference document to obtain a score matrix; then, qualified knowledge bases with qualified learning capabilities are determined based on the score matrix; finally, for each qualified knowledge base, the sum of squared residuals and the standard deviation of all the comprehensive scores corresponding to the qualified knowledge base are calculated, and the evaluation result of the qualified knowledge base is determined according to the sum of squared residuals and the standard deviation. The method of the present application obtains the comprehensive scores of the knowledge base when processing reference documents from multiple dimensions, and analyzes the sum of squared residuals and the standard deviation of all the comprehensive scores corresponding to the knowledge base, which can accurately measure the accuracy, response stability and evaluation consistency of the knowledge base. When the performance of the knowledge base does not meet the expectations, the evaluation result can also be used to guide the further optimization of the knowledge base, thereby improving the performance of the knowledge base and ensuring that the knowledge base can operate efficiently and meet user needs. Description of the Drawings

[0037] One or more embodiments are illustrated by way of example in the accompanying drawings, which do not constitute a limitation on the embodiments. Elements with the same reference numerals in the drawings are represented as similar elements, unless otherwise stated, and the drawings in the figures do not constitute a scale limitation.

[0038] Figure 1 It is a schematic flowchart of a knowledge base evaluation method provided by an embodiment of the present application;

[0039] Figure 2 It is a schematic flowchart of a process for scoring the completeness of a generated answer provided by an embodiment of the present application;

[0040] Figure 3 It is a schematic flowchart of a process for scoring the matching degree of a generated answer provided by an embodiment of the present application;

[0041] Figure 4 It is a schematic flowchart of a process for scoring the relevance of a generated answer provided by an embodiment of the present application;

[0042] Figure 5 It is a schematic diagram of the mean distribution of literature scores of three knowledge bases based on fifteen references provided by an embodiment of the present application;

[0043] Figure 6 It is a structural diagram of an electronic device provided by an embodiment of the present application. Detailed implementation manners

[0044] To make the objectives, technical solutions, and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present application. Apparently, the described embodiments are some, but not all, of the embodiments of the present application. All other embodiments obtained by those of ordinary skill in the art based on the embodiments in the present application without creative efforts shall fall within the scope of protection of the present application.

[0045] In addition, the technical features involved in the various embodiments of the present application described below can be combined with each other as long as they do not conflict with each other.

[0046] It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions, and although the logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in a different order than here.

[0047] Please refer to Figure 1 , Figure 1 It is a schematic flowchart of a knowledge base evaluation method provided by an embodiment of the present application, which specifically includes the following steps:

[0048] Step S101, for each combination of , and , obtain the th knowledge base based on the Reference Generate answers to questions , and generate answers to the Perform comprehensive scoring to obtain the scoring matrix of K questions corresponding to I knowledge base based on J references , where I, J and K are positive integers, =1, …, I, =1, …,J, =1, …, K.

[0049] Specifically, I is the number of knowledge bases, J is the number of references, and K is the number of questions corresponding to each reference. The knowledge base in this application is a system based on an artificial intelligence system that helps users query, search and obtain information in specific fields through technical means such as large language models and semantic vector models. Each knowledge base can be connected to the evaluation system through API (Application Programming Interface) and other calling methods to realize the automatic sending of questions and the reception of generated answers. The evaluation system obtains the generated answers of each knowledge base based on each question of each reference through API calls. Then, generate the answer to Scoring from multiple dimensions to generate answers 's comprehensive rating.

[0050] In one embodiment, the answer to the Scoring from multiple dimensions to generate answers The implementation method of comprehensive scoring includes: generating answers based on Corresponding reference answers , generate the answer to Scoring is done from three dimensions: completeness, matching, and relevance to get the generated answer. Completeness score , matching score and relevance score ; Then generate the answer Completeness score , matching score and relevance score Generate answers based on the average method Overall rating .

[0051] Among them, the completeness score Used to measure the generated answer Whether the reference answers are fully covered All key points, with a value range of [0, 1]. The higher the score, the more comprehensive the generated answer; matching degree score Used to measure the generated answer And the reference answer Consistency, with a value range of [0, 1]. The higher the score, the more consistent the generated answer is with the reference answer; relevance score Used to measure the generated answer Relevance to the question, with a value range of [0, 1]. The higher the score, the more closely related the generated answer is to the question. The average value method is any one of the weighted average, geometric average, harmonic average, and truncated average. Taking the weighted average method as an example, the comprehensive score The calculation formula is:

[0052]

[0053] Among them, 、 And Are the weights of the three scoring dimensions respectively, and, + + = 1. The generated answer Integrity score 、Matching degree score And relevance score Determination method, please refer to the following description about Figures 2 to 4 .

[0054] After obtaining the comprehensive score Of each generated answer , the three-dimensional scoring matrix of K questions corresponding to I knowledge bases based on J reference documents can be constructed according to the comprehensive score , specifically as follows: Specifically as follows:

[0055]

[0056] Among them, for = 1,..., I, Is the scoring sub-matrix corresponding to the th knowledge base, and its dimension is , Each row in represents the comprehensive score of the th knowledge base for each question in the th reference document.

[0057] Step S102, based on the scoring matrix , determine the qualified knowledge bases with qualified learning ability from I knowledge bases.

[0058] In one embodiment, first, based on the scoring matrix , calculate the average literature score of each knowledge base based on each reference , and the formula is as follows:

[0059]

[0060] Then, based on the average literature score and the literature learning threshold corresponding to each reference , determine the knowledge bases with qualified learning ability.

[0061] Specifically, for the knowledge base corresponding to each value, if the average literature score of the references corresponding to this knowledge base based on each value is greater than the literature learning threshold of this reference , then determine that this knowledge base is a qualified knowledge base with qualified learning ability. If the average literature score of the references corresponding to this knowledge base based on at least one value is not greater than the literature learning threshold of this reference , then this knowledge base is a knowledge base with unqualified learning ability. Among them, = 1, …, I, = 1, …, J. For the knowledge bases with unqualified learning ability, the references corresponding to the value whose average literature score is not greater than the literature learning threshold can be found, and targeted optimized learning training is carried out according to the type of this reference, so that the knowledge base can be efficiently trained and meet the user's needs. value.

[0062] Step S103: Based on each qualified knowledge base, calculate the sum of squared residuals and the standard deviation of all comprehensive scores corresponding to this qualified knowledge base, and based on the sum of squared residuals and the standard deviation, obtain the evaluation result of this qualified knowledge base.

[0063] In this application, by analyzing the sum of squared residuals and the standard deviation of the knowledge base when processing references, the accuracy and stability of each knowledge base can be accurately measured. Among them, the sum of squared residuals reflects the cumulative degree of the difference between the generated answers and the reference answers of the knowledge base for each query problem, and is used to evaluate the correctness of the knowledge base. The standard deviation reflects the degree of dispersion of the difference between the generated answers and the reference answers of the knowledge base for each query problem, and is used to evaluate the stability of the knowledge base.

[0064] In one embodiment, calculating the sum of squared residuals and the standard deviation of all comprehensive scores corresponding to the qualified knowledge base includes:

[0065] Step 1: Calculate the learning score mean of all comprehensive scores corresponding to the qualified knowledge base , and the formula is as follows:

[0066]

[0067] Where, refers to the learning score mean corresponding to the th qualified knowledge base .

[0068] Step 2: Based on this learning score mean and the literature score mean of this qualified knowledge base based on each reference , calculate the sum of squared residuals of all comprehensive scores corresponding to this qualified knowledge base , and the formula is as follows:

[0069]

[0070] Step 3: Based on this learning score mean and the literature score mean of this qualified knowledge base based on each reference , calculate the standard deviation of all comprehensive scores corresponding to this qualified knowledge base , and the formula is as follows:

[0071]

[0072] After obtaining the sum of squared residuals and the standard deviation of the qualified knowledge base, based on the sum of squared residuals and the standard deviation, the evaluation results of this qualified knowledge base specifically include: a preset sum of squared residuals threshold and a standard deviation threshold . If the sum of squared residuals of the qualified knowledge base is less than the sum of squared residuals threshold and the standard deviation of this qualified knowledge base is less than the standard deviation threshold , it is determined that the accuracy and stability of this qualified knowledge base both meet the expectations; if the sum of squared residuals of the qualified knowledge base is greater than or equal to the sum of squared residuals threshold or the standard deviation of this qualified knowledge base is greater than or equal to the standard deviation threshold , it is determined that the accuracy and / or stability of this qualified knowledge base do not meet the expectations and further optimization is required. Specifically, when the standard deviation of the qualified knowledge base is greater than or equal to the standard deviation threshold , it indicates that the response consistency of this qualified knowledge base to different queries needs to be improved, and there are situations where it performs better for some references and worse for other references. When the standard deviation of the qualified knowledge base Less than the standard deviation threshold , but the residual sum of squares of the qualified knowledge base Greater than or equal to the residual sum of squares threshold , it indicates that the qualified knowledge base has high consistency in responding to different queries, but its overall knowledge processing capability is weak and needs to be further improved.

[0073] Please refer to Figure 5 , Figure 5 : This is a schematic diagram of the mean distribution of the literature scores of the three knowledge bases provided in the embodiment of the present application based on fifteen references. From the score point of view, the overall performance of knowledge base 1 is the most stable, with the smallest fluctuation, and the best learning effect on reference 12. The mean literature score of each reference is above 0.9 points. The fluctuation of knowledge base 2 is the most intense, with significant highs and lows, and the performance is unstable. The learning effect is the best for reference 2, and the mean score of each reference is above 0.8 points. The overall performance of knowledge base 3 is relatively stable, with smaller fluctuations, but the overall learning effect is not good, and the mean score of each reference is above 0.85 points. The standard deviation and residual sum of squares of the three knowledge bases in the fifteen references are shown in Table 1 below:

[0074] Table 1

[0075]

[0076] comprehensive Figure 5 Compared to the data in Table 1, we can see that Knowledge Base 1 performed exceptionally well in the evaluation of fifteen references, demonstrating superior performance both in individual reference scores and overall stability. While Knowledge Base 3's overall stability was similar to that of Knowledge Base 1, its individual reference scores were not high, indicating that its overall knowledge processing capabilities were weak and that its knowledge comprehension capabilities needed improvement. Knowledge Base 2's individual reference scores fluctuated, with both the residual sum of squares and standard deviation significantly higher than those of Knowledge Base 1 and Knowledge Base 3, indicating that further optimization is needed in both its knowledge processing capabilities and stability to improve its performance.

[0077] Please refer to Figure 2 , Figure 2 This is a flow chart of a completeness scoring process for generating answers, provided in an embodiment of the present application, which specifically includes:

[0078] Step S201: Generate an answer Perform word segmentation processing to generate a word segmentation result set.

[0079] Specifically, a word segmenter (e.g., Jieba) can be used to generate answers Perform word segmentation processing to generate a word segmentation result set . Generate a word segmentation result set includes generating an answer all the words after word segmentation. Jieba is an open-source Chinese word segmentation tool, mainly used for segmenting Chinese texts, providing basic support for subsequent text processing, analysis, mining and other tasks.

[0080] Step S202, perform word segmentation on the generated answer corresponding reference answer to obtain a reference word segmentation result set.

[0081] Similarly, a word segmenter (e.g., Jieba) can be used to perform word segmentation on the reference answer to obtain a reference word segmentation result set . The reference word segmentation result set includes all the words after word segmentation of the reference answer .

[0082] Step S203, calculate the intersection of the generated word segmentation result set and the reference word segmentation result set.

[0083] Specifically, the generated word segmentation result set and the reference word segmentation result set intersection . The intersection includes the common words of the generated word segmentation result set and the reference word segmentation result set .

[0084] Step S204, divide the size of the intersection by the size of the reference word segmentation result set to obtain the integrity score of the generated answer .

[0085] Specifically, the integrity score of the generated answer , where represents the size of the intersection, represents the size of the reference word segmentation result set.

[0086] Please refer to Figure 3 , Figure 3 which is a schematic flowchart of a process for matching degree scoring of a generated answer provided by an embodiment of the present application, specifically including:

[0087] Step S301, perform word segmentation on the generated answer to obtain a generated word segmentation result set.

[0088] This step is the same as step S201 and will not be elaborated here.

[0089] Step S302, perform word segmentation on the generated answer Corresponding reference answer Perform word segmentation processing to obtain a reference word segmentation result set.

[0090] This step is the same as step S202 and will not be elaborated here.

[0091] Step S303, calculate the intersection and union of the generated word segmentation result set and the reference word segmentation result set.

[0092] Generated word segmentation result set and reference word segmentation result set of the intersection The calculation method is as described in step S203. The generated word segmentation result set and reference word segmentation result set of the union . The union contains all the words in the generated word segmentation result set and reference word segmentation result set .

[0093] Step S304, divide the size of the intersection by the size of the union to obtain the matching degree score of the generated answer .

[0094] Specifically, the matching degree score of the generated answer where , represents the size of the intersection, represents the size of the union.

[0095] Please refer to Figure 4 , Figure 4 which is a schematic flowchart of a process for evaluating the relevance of a generated answer provided by an embodiment of the present application, specifically including:

[0096] Step S401, perform embedding vector conversion on the generated answer to obtain a generated embedding vector.

[0097] Specifically, use a natural language processing model (e.g., BERT) to convert the generated answer into a generated embedding vector . BERT (Bidirectional Encoder Representations from Transformers, bidirectional encoder representations from Transformers) is a natural language processing pre-training model developed by Google based on the Transformer architecture.

[0098] Step S402, for the reference answer corresponding to the generated answer ​Perform embedding vector transformation to obtain the reference embedding vector.

[0099] Similarly, using natural language processing models (e.g., BERT) to translate reference answers Convert to reference embedding vector .

[0100] Step S403: Calculate the cosine similarity between the generated embedding vector and the reference embedding vector.

[0101] Specifically, first calculate the generated embedding vector and the reference embedding vector The dot product score is as follows:

[0102]

[0103] in, is the dimension of the embedding vector.

[0104] Secondly, the embedding vector is calculated Embedding norm (i.e., magnitude 1) and the reference embedding vector Embedding norm (i.e., magnitude 2), the formula is as follows:

[0105]

[0106]

[0107] Finally, the cosine similarity between the generated embedding vector and the reference embedding vector is calculated as follows:

[0108]

[0109] Step S404: normalize the cosine similarity to generate the answer 's relevance score.

[0110] The value range of cosine similarity is [-1, 1]. The cosine similarity is normalized to make its value range [0, 1], and the normalized value is used as the generated answer. Relevance score In one embodiment, the relevance score Calculated based on the following formula:

[0111]

[0112] The knowledge base evaluation method provided by this application first comprehensively scores the answers generated by each knowledge base for each question of each reference document to obtain a scoring matrix; then determines the qualified knowledge bases with qualified learning capabilities based on the scoring matrix; finally, based on each qualified knowledge base, calculates the sum of squared residuals and the standard deviation of all the comprehensive scores corresponding to the qualified knowledge base, and determines the evaluation result of the qualified knowledge base according to the sum of squared residuals and the standard deviation. The method of this application obtains the comprehensive scores of the knowledge base when processing reference documents from multiple dimensions, and analyzes the sum of squared residuals and the standard deviation of all the comprehensive scores corresponding to the knowledge base, which can accurately measure the accuracy, response stability and evaluation consistency of the knowledge base. When the performance of the knowledge base does not meet the expectations, the evaluation result can also be used to guide the further optimization of the knowledge base, thereby improving the performance of the knowledge base and ensuring that the knowledge base can operate efficiently and meet user requirements.

[0113] According to an embodiment of the present application, there is provided an electronic device, such as Figure 6 , which is a schematic structural diagram of an electronic device provided by an embodiment of the present application. The electronic device may include a processor 601, a communication interface 602, a memory 603, and a communication bus 604. Among them, the processor 601, the communication interface 602, and the memory 603 complete mutual communication through the communication bus 604. The processor 601 may call the logic instructions in the memory 603 to execute the above-mentioned knowledge base evaluation method.

[0114] In addition, when the logic instructions in the above-mentioned memory 603 are implemented in the form of software functional units and sold or used as an independent product, they may be stored in several computer-readable storage media. Based on such an understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a part of this technical solution, may be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions for causing a computer device (which may be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the above-mentioned knowledge base evaluation method of this application. The foregoing storage media include: USB flash drives, mobile hard disks, read-only memories (ROM, Read-Only Memory), random access memories (RAM, Random Access Memory), magnetic disks, or optical disks, etc., which can store program codes.

[0115] According to an embodiment of the present application, there is provided a computer-readable storage medium, the type of which is as described above. The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the processor executes the steps of the above-mentioned knowledge base evaluation method.

[0116] Through the description of the above embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus a general hardware platform, and of course, it can also be implemented by hardware. Based on this understanding, the essence of the above technical solution or the part that contributes to the related technology can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to enable a computer device (which can be a personal computer, server, or network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0117] The above is only the specific embodiment of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art can easily think of changes or substitutions within the technical scope disclosed by the present application. Therefore, any modification, equivalent substitution, improvement, etc. made within the spirit and principle of the present application shall be included in the protection scope of the present application.

Claims

1. A knowledge base evaluation method, characterized in that: The method comprises: for 、 and For each combination of The knowledge base is based on Reference Generate answers to questions , and generate answers to the Perform comprehensive scoring to obtain the scoring matrix of K questions corresponding to I knowledge base based on J references , where I, J and K are positive integers, =1, …, I, =1, …, J, =1, …,K; Based on the scoring matrix , determine a qualified knowledge base with qualified learning ability from I knowledge bases; Based on each qualified knowledge base, calculating the residual sum of squares and the standard deviation of all comprehensive scores corresponding to the qualified knowledge base, and determining an evaluation result of the qualified knowledge base based on the residual sum of squares and the standard deviation; Based on the scoring matrix , from I knowledge bases, qualified knowledge bases with qualified learning ability are determined to include: Based on the scoring matrix , calculate the average literature score of each knowledge base based on each reference ; Based on each The knowledge base corresponding to the value, if the knowledge base is based on each The mean score of the references corresponding to the value are greater than the literature learning threshold of the reference , then the knowledge base is determined to be a qualified knowledge base with qualified learning ability; The calculation of the residual sum of squares and standard deviation of all comprehensive scores corresponding to the qualified knowledge base includes: Step 1: Calculate the mean learning score of all comprehensive scores corresponding to the qualified knowledge base , the formula is as follows: Step 2: Based on the mean learning score and the mean literature score of the qualified knowledge base based on each reference , calculate the residual sum of squares of all comprehensive scores corresponding to the qualified knowledge base , the formula is as follows: Step 3: Based on the mean learning score and the mean literature score of the qualified knowledge base based on each reference , calculate the standard deviation of all comprehensive scores corresponding to the qualified knowledge base , the formula is as follows: 。 2. The method according to claim 1, characterized in that The generated answer Comprehensive scoring includes: Get the generated answer Corresponding reference answers , generate the answer Scoring is performed based on completeness, matching and relevance to obtain the generated answer Completeness score, matching score and relevance score; Generate answers based on The completeness score, matching score and relevance score of 's comprehensive rating.

3. The method according to claim 2, characterized in that The generated answer Scoring based on the completeness, the generated answer is obtained The completeness score includes: Generate an answer to the Perform word segmentation processing to generate a word segmentation result set; The reference answer Perform word segmentation processing to obtain a reference word segmentation result set; Calculate the intersection of the generated word segmentation result set and the reference word segmentation result set, divide the size of the intersection by the size of the reference word segmentation result set, and obtain the generated answer Completeness rating.

4. The method according to claim 2, characterized in that The generated answer Scoring is performed based on the matching degree to obtain the generated answer The compatibility score includes: Generate an answer to the Perform word segmentation processing to generate a word segmentation result set; The reference answer Perform word segmentation processing to obtain a reference word segmentation result set; Calculate the intersection and union of the generated word segmentation result set and the reference word segmentation result set, and divide the size of the intersection by the size of the union to obtain the generated answer The matching score of .

5. The method according to claim 2, characterized in that The generated answer Scoring is performed based on the relevance to obtain the generated answer The relevance scores include: Generate an answer to the Perform embedding vector conversion to generate embedding vector; The reference answer Perform embedding vector transformation to obtain reference embedding vector; Calculating the cosine similarity between the generated embedding vector and the reference embedding vector; Normalize the cosine similarity to obtain the generated answer 's relevance score.

6. The method according to any one of claims 1 to 5, characterized in that Determining the evaluation result of the qualified knowledge base based on the residual sum of squares and the standard deviation includes: If the residual sum of squares of the qualified knowledge base is less than a preset residual sum of squares threshold and the standard deviation of the qualified knowledge base is less than a preset standard deviation threshold, it is determined that the accuracy and stability of the qualified knowledge base meet expectations; If the residual sum of squares of the qualified knowledge base is greater than or equal to the residual sum of squares threshold, or the standard deviation of the qualified knowledge base is greater than or equal to the standard deviation threshold, it is determined that the accuracy and / or stability of the qualified knowledge base does not meet expectations.

7. An electronic device, characterized in that: The method comprises at least one processor and a memory communicatively connected to the at least one processor, wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so as to enable the at least one processor to perform the knowledge base evaluation method according to any one of claims 1 to 6.

8. A computer storage medium, characterized in that The computer storage medium stores instructions or programs, and when the instructions or programs are executed by at least one processor, the at least one processor is caused to perform the knowledge base evaluation method according to any one of claims 1 to 6.

Images