Knowledge base evaluation method and device and storage medium

Abstract

The invention relates to the technical field of artificial intelligence, and discloses a knowledge base evaluation method and device and a storage medium. The knowledge base evaluation method comprises the following steps: for each combination of # imgabs0 #, # imgabs1 # and # imgabs2 #, obtaining a generation answer # imgabs6 # of a # imgabs5 # question of a # imgabs3 # knowledge base based on a # imgabs4 # reference, and carrying out comprehensive scoring on the generation answer # imgabs7 # to obtain a scoring matrix # imgabs8 # of I knowledge bases; based on the scoring matrix # imgabs9 #, determining a qualified knowledge base with qualified learning ability from the I knowledge bases; and based on each qualified knowledge base, calculating a residual sum of squares and a standard deviation of all the comprehensive scores corresponding to the qualified knowledge base, and based on the residual sum of squares and the standard deviation, determining an evaluation result of the qualified knowledge base. According to the method provided by the invention, the accuracy, the reaction stability and the evaluation consistency of each knowledge base can be accurately measured by analyzing the residual sum of squares and the standard deviation when the knowledge base processes the reference.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and in particular to a knowledge base evaluation method, device and storage medium. Background Art

[0002] With the rapid development of Large Language Model (LLM) technology, Artificial Intelligence (AI) knowledge base systems have become an important platform for information management and knowledge sharing. AI knowledge base systems integrate various types of data and information within the enterprise to provide intelligent search, recommendation and question-answering services, greatly improving employee work efficiency and decision-making quality. However, it is difficult to define the evaluation of AI knowledge base system question-answering accuracy and other performance indicators. How to effectively evaluate and optimize the AI ​​knowledge base system has become the key to ensuring its efficient operation and meeting user needs.

[0003] Existing technical solutions generally use evaluation methods based on public data sets to judge the performance of AI knowledge base systems. Evaluation methods mostly focus on quantitative analysis (such as accuracy and recall). These methods cannot fully reflect the performance of AI systems in practical applications, especially in terms of user experience and personalized needs. Summary of the invention

[0004] The embodiments of the present application aim to provide a knowledge base evaluation method, device and storage medium to solve the problem that the method for evaluating the knowledge base in the prior art cannot fully reflect the performance of the AI ​​system in practical applications.

[0005] To solve the above technical problems, the present application provides the following technical solutions: According to a first aspect of the present application, a knowledge base evaluation method is provided, the method comprising: for , and For each combination of The knowledge base is based on Reference Generate answers to questions , and generate answers to the Perform comprehensive scoring to obtain the scoring matrix of K questions corresponding to I knowledge base based on J references , where I, J and K are positive integers, =1, …, I, =1, …, J, =1,…,K; Based on the scoring matrix , determine a qualified knowledge base with qualified learning ability from I knowledge bases; Based on each qualified knowledge base, the residual sum of squares and the standard deviation of all comprehensive scores corresponding to the qualified knowledge base are calculated, and based on the residual sum of squares and the standard deviation, the evaluation result of the qualified knowledge base is determined.

[0006] Optionally, the generating an answer to Comprehensive scoring includes: Get the generated answer Corresponding reference answer , generate the answer Scoring the completeness, matching and relevance respectively to obtain the generated answer Completeness score, matching score and relevance score; Generate answers based on The completeness score, matching score and relevance score of The comprehensive rating of .

[0007] Optionally, the generating an answer to Scoring based on completeness, the generated answer is obtained The completeness score includes: Generate an answer to the Perform word segmentation processing to generate a word segmentation result set; The reference answer Perform word segmentation processing to obtain a reference word segmentation result set; Calculate the intersection of the generated word segmentation result set and the reference word segmentation result set, divide the size of the intersection by the size of the reference word segmentation result set, and obtain the generated answer Completeness rating.

[0008] Optionally, the generating an answer to Scoring the matching degree to obtain the generated answer The match scores include: Generate an answer to the Perform word segmentation processing to generate a word segmentation result set; The reference answer Perform word segmentation processing to obtain a reference word segmentation result set; Calculate the intersection and union of the generated word segmentation result set and the reference word segmentation result set, divide the size of the intersection by the size of the union, and obtain the generated answer The matching score of .

[0009] Optionally, the generating an answer to Scoring the relevance to obtain the generated answer The relevance scores include: Generate an answer to the Perform embedding vector conversion to generate embedding vector; The reference answer Perform embedding vector transformation to obtain reference embedding vector; Calculating the cosine similarity between the generated embedding vector and the reference embedding vector; The cosine similarity is normalized to obtain the generated answer 's relevance score.

[0010] Optionally, the scoring matrix , the qualified knowledge bases with qualified learning ability are determined from I knowledge bases, including: Based on the scoring matrix , calculate the average literature score of each knowledge base based on each reference ; According to the mean score of the literature The literature learning threshold corresponding to each reference , determine a qualified knowledge base with qualified learning ability from I knowledge bases.

[0011] Optionally, the mean score of the literature score The literature learning threshold corresponding to each reference , the qualified knowledge bases with qualified learning ability are determined from I knowledge bases, including: Based on each The knowledge base corresponding to the value, if the knowledge base is based on each The mean score of the references corresponding to the value are greater than the literature learning threshold of the reference , then it is determined that the knowledge base is a qualified knowledge base with qualified learning ability.

[0012] Optionally, determining the evaluation result of the qualified knowledge base based on the residual sum of squares and the standard deviation includes: If the residual sum of squares of the qualified knowledge base is less than a preset residual sum of squares threshold and the standard deviation of the qualified knowledge base is less than a preset standard deviation threshold, it is determined that the accuracy and stability of the qualified knowledge base are both as expected; If the residual sum of squares of the qualified knowledge base is greater than or equal to the residual sum of squares threshold, or the standard deviation of the qualified knowledge base is greater than or equal to the standard deviation threshold, it is determined that the accuracy and / or stability of the qualified knowledge base does not meet expectations.

[0013] According to the second aspect of the present application, an electronic device is provided, comprising at least one processor and a memory communicatively connected to the at least one processor, the memory storing instructions executable by the at least one processor, the instructions being executed by the at least one processor so that the at least one processor can execute the knowledge base evaluation method described above.

[0014] According to a third aspect of the present application, a computer storage medium is provided, wherein the computer storage medium stores instructions or programs, and when the instructions or programs are executed by at least one processor, the at least one processor executes the knowledge base evaluation method described above.

[0015] The beneficial effects of the embodiments of the present application are as follows: Different from the prior art, in the embodiments of the present application, a knowledge base evaluation method is provided, firstly, based on each knowledge base, a comprehensive score is given to the generated answers of each question of each reference to obtain a scoring matrix; then, based on the scoring matrix, a qualified knowledge base with qualified learning ability is determined; finally, based on each qualified knowledge base, the residual sum of squares and standard deviation of all comprehensive scores corresponding to the qualified knowledge base are calculated, and the evaluation result of the qualified knowledge base is determined according to the residual sum of squares and standard deviation. The method of the present application obtains the comprehensive score of the knowledge base when processing references from multiple dimensions, and analyzes the residual sum of squares and standard deviation of all comprehensive scores corresponding to the knowledge base, which can accurately measure the accuracy, response stability and evaluation consistency of the knowledge base. When the performance of the knowledge base does not meet expectations, the evaluation results can also be used to guide the further optimization of the knowledge base, thereby improving the performance of the knowledge base and ensuring that the knowledge base can operate efficiently and meet user needs. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] One or more embodiments are exemplarily described by pictures in the corresponding drawings, and these exemplified descriptions do not constitute limitations on the embodiments. Elements with the same reference numerals in the drawings represent similar elements, and unless otherwise stated, the figures in the drawings do not constitute proportional limitations.

[0017] Figure 1 It is a flowchart of a knowledge base evaluation method provided in an embodiment of the present application; Figure 2 It is a schematic diagram of a process for scoring the completeness of a generated answer provided in an embodiment of the present application; Figure 3 It is a schematic diagram of a flow chart of matching score of generated answers provided in an embodiment of the present application; Figure 4 It is a schematic diagram of a process for scoring the relevance of generated answers provided in an embodiment of the present application; Figure 5It is a schematic diagram of the mean distribution of the literature scores of the three knowledge bases provided in the embodiment of the present application based on fifteen references; Figure 6 It is a structural diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0018] In order to make the purpose, technical solution and advantages of the embodiments of the present application clearer, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0019] In addition, the technical features involved in each embodiment of the present application described below can be combined with each other as long as they do not conflict with each other.

[0020] It should be noted that the steps shown in the flowcharts of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and that, although a logical order is shown in the flowcharts, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0021] Please refer to Figure 1 , Figure 1 This is a flow chart of a knowledge base evaluation method provided in an embodiment of the present application, which specifically includes the following steps: Step S101, for , and For each combination of The knowledge base is based on Reference Generate answers to questions , and generate answers to the Perform comprehensive scoring to obtain the scoring matrix of K questions corresponding to I knowledge base based on J references , where I, J and K are positive integers, =1, …, I, =1, …,J, =1, …, K.

[0022] Specifically, I is the number of knowledge bases, J is the number of references, and K is the number of questions corresponding to each reference. The knowledge base in this application is a system based on an artificial intelligence system that helps users query, search and obtain information in specific fields through technical means such as large language models and semantic vector models. Each knowledge base can be connected to the evaluation system through API (Application Programming Interface) and other calling methods to realize the automatic sending of questions and the reception of generated answers. The evaluation system obtains the generated answers of each knowledge base based on each question of each reference through API calls. Then, generate the answer to Scoring from multiple dimensions to generate answers The comprehensive rating of .

[0023] In one embodiment, to generate an answer Scoring from multiple dimensions to generate answers The implementation method of comprehensive scoring includes: generating answers based on Corresponding reference answer , which generates the answer The generated answer is obtained by scoring from three dimensions: completeness, matching and relevance. Completeness Rating , Matching score and relevance score ; Then generate the answer Completeness Rating , Matching score and relevance score Generate the answer based on the average method Overall rating .

[0024] Among them, the completeness score Used to measure the generated answer Whether the reference answers are fully covered The value range is [0,1]. The higher the score, the more comprehensive the generated answer. Used to measure the generated answer With reference answer The consistency of the answer is in the range of [0,1]. The higher the score, the more consistent the generated answer is with the reference answer. Used to measure the generated answer The degree of relevance to the question, with a value range of [0,1]. The higher the score, the more closely the generated answer is related to the question. The average method can be any of the weighted average, geometric mean, harmonic mean, and truncated mean. Taking the weighted average method as an example, the comprehensive score The calculation formula is:

[0025] in, , and are the weights of the three scoring dimensions respectively, and, + + = 1. Generate answer Completeness Rating , Matching score and relevance score Please refer to the following for the method of determining Figure 2~Figure 4 Description.

[0026] Get the generated answers Overall rating After that, according to the comprehensive score Construct a three-dimensional scoring matrix of K questions corresponding to I knowledge bases based on J references , as follows:

[0027] Among them, for =1, …, I, For the The scoring sub-matrix corresponding to the knowledge base has the dimension , Each row in the The knowledge base The comprehensive score of each question in the reference.

[0028] Step S102, based on the scoring matrix , determine a qualified knowledge base with qualified learning ability from I knowledge bases.

[0029] In one embodiment, based on the scoring matrix , calculate the average literature score of each knowledge base based on each reference , the formula is as follows:

[0030] Then, according to the mean score of the literature The literature learning threshold corresponding to each reference , determine the knowledge base with qualified learning ability.

[0031] Specifically, based on each The knowledge base corresponding to the value, if the knowledge base is based on each The mean score of the references corresponding to the value are greater than the reference learning threshold of the reference , then the knowledge base is determined to be a qualified knowledge base with qualified learning ability if the knowledge base is based on at least one of The mean score of the references corresponding to the value Not greater than the reference learning threshold of the reference , then the knowledge base is a knowledge base with unqualified learning ability, where =1,…, I, =1, …, J. For knowledge bases with poor learning ability, the mean of the literature ratings can be found Not greater than the literature learning threshold of The reference corresponding to the value is optimized and trained according to the type of the reference so that the knowledge base can be trained efficiently and meet user needs.

[0032] Step S103, based on each qualified knowledge base, calculate the residual sum of squares and standard deviation of all comprehensive scores corresponding to the qualified knowledge base, and obtain the evaluation result of the qualified knowledge base based on the residual sum of squares and standard deviation.

[0033] In this application, by analyzing the residual sum of squares and standard deviation of the knowledge base when processing references, the accuracy and stability of each knowledge base can be accurately measured. Among them, the residual sum of squares reflects the cumulative degree of difference between the generated answers of the knowledge base for each query question and the reference answer, which is used to evaluate the correctness of the knowledge base. The standard deviation reflects the degree of discreteness of the difference between the generated answers of the knowledge base for each query question and the reference answer, which is used to evaluate the stability of the knowledge base.

[0034] In one embodiment, calculating the residual sum of squares and standard deviation of all comprehensive scores corresponding to the qualified knowledge base includes: Step 1: Calculate the mean learning score of all comprehensive scores corresponding to the qualified knowledge base , the formula is as follows:

[0035] in, It refers to The average learning score corresponding to qualified knowledge bases .

[0036] Step 2: Based on the mean learning score and the mean literature score of the qualified knowledge base based on each reference , calculate the residual sum of squares of all comprehensive scores corresponding to the qualified knowledge base , the formula is as follows:

[0037] Step 3: Based on the mean learning score and the mean literature score of the qualified knowledge base based on each reference , calculate the standard deviation of all comprehensive scores corresponding to the qualified knowledge base , the formula is as follows:

[0038] After obtaining the residual sum of squares and standard deviation of the qualified knowledge base, the evaluation result of the qualified knowledge base is obtained based on the residual sum of squares and standard deviation, which specifically includes: presetting the residual sum of squares threshold and standard deviation threshold , if the residual sum of squares of the qualified knowledge base Less than the residual sum of squares threshold And the standard deviation of the qualified knowledge base Less than standard deviation threshold , it is determined that the accuracy and stability of the qualified knowledge base meet expectations; if the residual sum of squares of the qualified knowledge base Greater than or equal to the residual sum of squares threshold Or the standard deviation of the qualified knowledge base Greater than or equal to standard deviation threshold , it is determined that the accuracy and / or stability of the qualified knowledge base does not meet expectations and needs further optimization. Specifically, when the standard deviation of the qualified knowledge base Greater than or equal to standard deviation threshold When the standard deviation of the qualified knowledge base is , it means that the consistency of the response of the qualified knowledge base to different queries needs to be improved, and there is a situation where the performance is better for some references and worse for other references. Less than standard deviation threshold , but the residual sum of squares of the qualified knowledge base Greater than or equal to the residual sum of squares threshold , it indicates that the qualified knowledge base has high consistency in responding to different queries, but its overall knowledge processing capability is weak, and the overall knowledge processing capability needs to be further improved.

[0039] Please refer to Figure 5 , Figure 5: This is a schematic diagram of the mean distribution of the literature scores of the three knowledge bases provided in the embodiment of the present application based on fifteen references. From the score point of view, the overall performance of knowledge base 1 is the most stable, with the smallest fluctuation, and the best learning effect on reference 12. The mean literature score of each reference is above 0.9 points. Knowledge base 2 has the most violent fluctuations, with significant highs and lows, and unstable performance. It has the best learning effect on reference 2, and the mean score of each reference is above 0.8 points. The overall performance of knowledge base 3 is relatively stable, with smaller fluctuations, but the overall learning effect is not good, and the mean score of each reference is above 0.85 points. The standard deviation and residual sum of squares of the three knowledge bases in the fifteen references are shown in Table 1 below: Table 1

[0040] comprehensive Figure 5 From the data in Table 1, we can see that: Knowledge Base 1 performed well in the evaluation of fifteen references, showing excellent performance in both the score of a single reference and overall stability. Although the overall stability of Knowledge Base 3 is similar to that of Knowledge Base 1, its score of a single reference is not high, indicating that the knowledge base's overall knowledge processing ability is weak and the knowledge understanding ability of the knowledge base needs to be improved. The score of Knowledge Base 2 in a single reference fluctuates, and the residual sum of squares and standard deviation are much higher than those of Knowledge Base 1 and Knowledge Base 3, indicating that the knowledge base needs more optimization in terms of knowledge processing ability and stability to improve its performance.

[0041] Please refer to Figure 2 , Figure 2 This is a flow chart of a completeness scoring process for generating answers provided by an embodiment of the present application, which specifically includes: Step S201, generate an answer Perform word segmentation processing to generate a word segmentation result set.

[0042] Specifically, a word segmenter (e.g., Jieba) can be used to generate answers Perform word segmentation processing to generate a word segmentation result set . Generate word segmentation result set The generated answer is included All words after word segmentation. Jieba is an open source Chinese word segmentation tool, which is mainly used to segment Chinese text and provide basic support for subsequent text processing, analysis, mining and other tasks.

[0043] Step S202, generate an answer Corresponding reference answer Perform word segmentation processing to obtain a reference word segmentation result set.

[0044] Similarly, a word segmenter (e.g., Jieba) can be used to segment the reference answers. Perform word segmentation processing to obtain a reference word segmentation result set . Reference word segmentation result set Contains reference answers All words after word segmentation.

[0045] Step S203: Calculate the intersection of the generated word segmentation result set and the reference word segmentation result set.

[0046] Specifically, generate a word segmentation result set And the reference word segmentation result set The intersection of . Intersection Contains the generated word segmentation result set And the reference word segmentation result set common vocabulary.

[0047] Step S204: Divide the size of the intersection by the size of the reference word segmentation result set to obtain the generated answer Completeness rating.

[0048] Specifically, generate the answer Completeness Rating ,in, represents the size of the intersection, Indicates the size of the reference word segmentation result set.

[0049] Please refer to Figure 3 , Figure 3 This is a flow chart of a matching score for a generated answer provided in an embodiment of the present application, which specifically includes: Step S301, generate an answer Perform word segmentation processing to generate a word segmentation result set.

[0050] This step is the same as step S201 and will not be described again here.

[0051] Step S302, generate an answer Corresponding reference answer Perform word segmentation processing to obtain a reference word segmentation result set.

[0052] This step is the same as step S202 and will not be repeated here.

[0053] Step S303: Calculate the intersection and union of the generated word segmentation result set and the reference word segmentation result set.

[0054] Generate word segmentation result set And the reference word segmentation result set The intersection of The calculation method of is as described in step S203. Generate word segmentation result set And the reference word segmentation result set The union of . Union Contains the generated word segmentation result set And the reference word segmentation result set All vocabulary.

[0055] Step S304, divide the size of the intersection by the size of the union to get the generated answer The matching score of .

[0056] Specifically, generate the answer Match score ,in, represents the size of the intersection, Indicates the size of the union.

[0057] Please refer to Figure 4 , Figure 4 This is a flow chart of a method for scoring the relevance of generated answers provided in an embodiment of the present application, which specifically includes: Step S401, generate an answer Perform embedding vector conversion to generate embedding vector.

[0058] Specifically, a natural language processing model (e.g., BERT) is used to generate answers Convert to generate embedding vector BERT (Bidirectional Encoder Representations from Transformers) is a natural language processing pre-training model developed by Google based on the Transformer architecture.

[0059] Step S402, generate an answer Corresponding reference answer Perform embedding vector transformation to obtain reference embedding vector.

[0060] Similarly, using natural language processing models (e.g., BERT) to translate reference answers Convert to reference embedding vector .

[0061] Step S403, calculating the cosine similarity between the generated embedding vector and the reference embedding vector.

[0062] Specifically, we first calculate the generated embedding vector and the reference embedding vector The dot product score is as follows:

[0063] in, is the dimension of the embedding vector.

[0064] Secondly, the embedding vector is calculated Embedded norm (i.e., magnitude 1) and the reference embedding vector Embedded norm (i.e., magnitude 2), the formula is as follows:

[0065]

[0066] Finally, the cosine similarity between the generated embedding vector and the reference embedding vector is calculated as follows:

[0067] Step S404, normalize the cosine similarity to obtain the generated answer 's relevance score.

[0068] The value range of cosine similarity is [-1, 1]. The cosine similarity is normalized to make its value range [0, 1], and the normalized value is used as the generated answer. Relevance score In one embodiment, the relevance score Calculated based on the following formula:

[0069] The knowledge base evaluation method provided in this application first comprehensively scores the generated answers of each question of each reference based on each knowledge base to obtain a scoring matrix; then, based on the scoring matrix, a qualified knowledge base with qualified learning ability is determined; finally, based on each qualified knowledge base, the residual sum of squares and standard deviation of all comprehensive scores corresponding to the qualified knowledge base are calculated, and the evaluation result of the qualified knowledge base is determined based on the residual sum of squares and standard deviation. The method of this application obtains the comprehensive score of the knowledge base when processing references from multiple dimensions, and analyzes the residual sum of squares and standard deviation of all comprehensive scores corresponding to the knowledge base, which can accurately measure the accuracy, response stability and evaluation consistency of the knowledge base. When the performance of the knowledge base does not meet expectations, the evaluation results can also be used to guide the further optimization of the knowledge base, thereby improving the performance of the knowledge base and ensuring that the knowledge base can operate efficiently and meet user needs.

[0070] According to an embodiment of the present application, an electronic device is provided, such as Figure 6, is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application, the electronic device may include a processor 601, a communication interface 602, a memory 603 and a communication bus 604, wherein the processor 601, the communication interface 602, and the memory 603 communicate with each other through the communication bus 604. The processor 601 may call the logic instructions in the memory 603 to execute the above-mentioned knowledge base evaluation method.

[0071] In addition, the logic instructions in the above-mentioned memory 603 can be implemented in the form of software functional units and can be stored in several computer-readable storage media when sold or used as independent products. Based on this understanding, the technical solution of the present application can be essentially or partly embodied in the form of a software product that contributes to the prior art. The computer software product is stored in a storage medium, including several instructions to enable a computer device (which can be a personal computer, server or network device, etc.) to perform all or part of the steps of the aforementioned knowledge base evaluation method of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program codes.

[0072] According to an embodiment of the present application, a computer-readable storage medium is provided, the type of which is as described above, the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the processor performs the steps of the knowledge base evaluation method described above.

[0073] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a general hardware platform, and of course, by hardware. Based on this understanding, the above technical solution, in essence or in other words, the part that contributes to the relevant technology, can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiment.

[0074] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any technician familiar with the technical field can easily think of changes or substitutions within the technical scope disclosed in the present application. Therefore, any modification, equivalent substitution, improvement, etc. made within the spirit and principle of the present application should be included in the protection scope of the present application.

Claims

1. A knowledge base evaluation method, characterized in that: The method comprises: for , and For each combination of The knowledge base is based on Reference Generate answers to questions , and generate answers to the Perform comprehensive scoring to obtain the scoring matrix of K questions corresponding to I knowledge base based on J references , where I, J and K are positive integers, =1, …, I, =1, …, J, =1, …,K; Based on the scoring matrix , determine a qualified knowledge base with qualified learning ability from I knowledge bases; Based on each qualified knowledge base, the residual sum of squares and the standard deviation of all comprehensive scores corresponding to the qualified knowledge base are calculated, and based on the residual sum of squares and the standard deviation, the evaluation result of the qualified knowledge base is determined.

2. The method according to claim 1, characterized in that The generated answer to Comprehensive scoring includes: Get the generated answer Corresponding reference answer , generate the answer Scoring the completeness, matching and relevance respectively to obtain the generated answer Completeness score, matching score and relevance score; Generate answers based on The completeness score, matching score and relevance score of The comprehensive rating of .

3. The method according to claim 2, characterized in that The generated answer to Scoring based on completeness, the generated answer is obtained The completeness score includes: Generate an answer to the Perform word segmentation processing to generate a word segmentation result set; The reference answer Perform word segmentation processing to obtain a reference word segmentation result set; Calculate the intersection of the generated word segmentation result set and the reference word segmentation result set, divide the size of the intersection by the size of the reference word segmentation result set, and obtain the generated answer Completeness rating.

4. The method according to claim 2, characterized in that: The generated answer to Scoring the matching degree to obtain the generated answer The match scores include: Generate an answer to the Perform word segmentation processing to generate a word segmentation result set; The reference answer Perform word segmentation processing to obtain a reference word segmentation result set; Calculate the intersection and union of the generated word segmentation result set and the reference word segmentation result set, divide the size of the intersection by the size of the union, and obtain the generated answer The matching score of .

5. The method according to claim 2, characterized in that: The generated answer to Scoring the relevance to obtain the generated answer The relevance scores include: Generate an answer to the Perform embedding vector conversion to generate embedding vector; The reference answer Perform embedding vector transformation to obtain reference embedding vector; Calculating the cosine similarity between the generated embedding vector and the reference embedding vector; The cosine similarity is normalized to obtain the generated answer 's relevance score.

6. The method according to claim 1, characterized in that Based on the scoring matrix , the qualified knowledge bases with qualified learning ability are determined from I knowledge bases, including: Based on the scoring matrix , calculate the average literature score of each knowledge base based on each reference ; According to the mean score of the literature The literature learning threshold corresponding to each reference , determine a qualified knowledge base with qualified learning ability from I knowledge bases.

7. The method according to claim 6, characterized in that The mean score according to the literature The literature learning threshold corresponding to each reference , the qualified knowledge bases with qualified learning ability are determined from I knowledge bases, including: Based on each The knowledge base corresponding to the value, if the knowledge base is based on each The mean score of the references corresponding to the value are greater than the literature learning threshold of the reference , then it is determined that the knowledge base is a qualified knowledge base with qualified learning ability.

8. The method according to any one of claims 1 to 7, characterized in that: The step of determining the evaluation result of the qualified knowledge base based on the residual sum of squares and the standard deviation includes: If the residual sum of squares of the qualified knowledge base is less than a preset residual sum of squares threshold and the standard deviation of the qualified knowledge base is less than a preset standard deviation threshold, it is determined that the accuracy and stability of the qualified knowledge base are both as expected; If the residual sum of squares of the qualified knowledge base is greater than or equal to the residual sum of squares threshold, or the standard deviation of the qualified knowledge base is greater than or equal to the standard deviation threshold, it is determined that the accuracy and / or stability of the qualified knowledge base does not meet expectations.

9. An electronic device, characterized in that: The method comprises at least one processor and a memory communicatively connected to the at least one processor, wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the knowledge base evaluation method according to any one of claims 1 to 8.

10. A computer storage medium, characterized in that: The computer storage medium stores instructions or programs, and when the instructions or programs are executed by at least one processor, the at least one processor is caused to execute the knowledge base evaluation method according to any one of claims 1 to 8.

Images