Model performance evaluation method and device, electronic equipment and storage medium
By extracting atomic facts from text data and calculating the similarity of the embedded vectors of the model in these facts, the problem of poor performance evaluation in the professional field of large language models is solved, and more accurate model performance evaluation and application results are achieved.
Patent Information
- Application Number
- CN202411141999.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-20
- Publication Date
- 2024-11-26
AI Technical Summary
In the application of large language models in professional fields, due to the lack of systematic and standardized training and evaluation, the model performance is poor, and inaccurate or incorrect information is output.
By extracting atomic facts from the text data to be used, analyzing and answering the questions to be answered based on the model to be evaluated, the answers to be evaluated are generated, and the fact embedding vector and the vector to be compared are determined using the pre-constructed target model, and the vector similarity is calculated to Evaluate model performance.
A more objective and accurate evaluation of the performance of large language models is achieved, ensuring that the model provides accurate information in the application and reducing the occurrence of error information.
Smart Images

Figure CN119025877A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of natural language processing technology, and in particular to a model performance evaluation method, device, electronic device and storage medium. Background Art
[0002] With the rapid development of large language models, large language models are being used more and more widely in various fields.
[0003] However, as big language models are increasingly used in professional fields, factual problems based on big language models are gradually becoming prominent. That is to say, due to the lack of systematic and standardized training and evaluation of big language models on professional field data, the model performance of big language models may be poor, which in turn leads to inaccurate or even wrong information in the output results of big language models during the application stage.
[0004] In order to solve the above problems, it is necessary to improve the evaluation method of model performance. Summary of the invention
[0005] The present invention provides a model performance evaluation method, device, electronic device and storage medium to solve the problem that the model performance of a large language model in the prior art lacks an objective and effective evaluation method, resulting in inaccurate model performance evaluation results of the large language model.
[0006] In a first aspect, an embodiment of the present invention provides a model performance evaluation method, comprising:
[0007] Extracting at least one atomic fact from the text data to be used; wherein the atomic fact refers to a factual description of the smallest granularity, and the atomic fact includes a question to be answered and a standard answer corresponding to the question to be answered;
[0008] For each fact atom, based on the model to be evaluated, the question to be answered in the current fact atom is analyzed and answered to obtain an answer to be evaluated corresponding to the question to be answered;
[0009] Determine, based on a pre-built target model, a fact embedding vector corresponding to the standard answer and a vector to be compared corresponding to the answer to be evaluated;
[0010] Determine, based on the fact embedding vector and the vector to be compared, a fact support attribute of the answer to be evaluated for the question to be answered;
[0011] The model performance evaluation result of the model to be evaluated is determined according to the fact support attributes of all the answers to be evaluated for the corresponding questions to be answered.
[0012] In a second aspect, an embodiment of the present invention further provides a model performance evaluation device, comprising:
[0013] An atomic fact extraction module, used to extract at least one atomic fact from the text data to be used; wherein the atomic fact refers to a factual description of the smallest granularity, and the atomic fact includes a question to be answered and a standard answer corresponding to the question to be answered;
[0014] The module for determining the answer to be evaluated is used to analyze and answer the question to be answered in the current fact atom based on the model to be evaluated for each fact atom, and obtain the answer to be evaluated corresponding to the question to be answered;
[0015] A vector determination module, used to determine the fact embedding vector corresponding to the standard answer and the vector to be compared corresponding to the answer to be evaluated based on a pre-built target model;
[0016] An attribute determination module, configured to determine a fact support attribute of the answer to be evaluated for the question to be answered based on the fact embedding vector and the vector to be compared;
[0017] The evaluation result determination module is used to determine the model performance evaluation result of the model to be evaluated based on the fact support attributes of all the answers to be evaluated for the corresponding questions to be answered.
[0018] In a third aspect, an embodiment of the present invention further provides an electronic device, including:
[0019] at least one processor; and
[0020] a memory communicatively connected to the at least one processor; wherein,
[0021] The memory stores a computer program that can be executed by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can execute the model performance evaluation method described in any embodiment of the present invention.
[0022] In a fourth aspect, an embodiment of the present invention further provides a computer-readable storage medium, wherein the computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to implement the model performance evaluation method described in any embodiment of the present invention when executed.
[0023] In a fifth aspect, an embodiment of the present invention further provides a computer program product, including a computer program, which, when executed by a processor, implements a model performance evaluation method as described in any one of the embodiments of the present invention.
[0024] The technical solution of the embodiment of the present invention is to extract at least one atomic fact from the text data to be used; for each fact atom, analyze and answer the question to be answered in the current fact atom based on the model to be evaluated, and obtain the answer to be evaluated corresponding to the question to be answered; determine the fact embedding vector corresponding to the standard answer and the vector to be compared corresponding to the answer to be evaluated based on the pre-built target model; determine the fact support attribute of the answer to be evaluated for the question to be answered based on the fact embedding vector and the vector to be compared; determine the model performance evaluation result of the model to be evaluated according to the fact support attributes of all the answers to be evaluated for the corresponding question to be answered. In this technical solution, in order to evaluate the model performance of the model to be evaluated, atomic facts can be extracted from the text data to be used, and the obtained atomic facts can be vectorized based on the pre-built target model, and the fact embedding vector generated by the atomic facts corresponding to the question to be answered and the standard answer is calculated with the comparison vector generated by the atomic facts corresponding to the question to be answered and the answer to be evaluated output by the model to be evaluated to obtain the vector similarity. It should be noted that the vector similarity between the fact embedding vector and the vector to be compared can be used to measure the matching degree between the answer to be evaluated and the question to be answered. The higher the vector similarity, the higher the matching degree between the answer to be evaluated and the question to be answered. Correspondingly, it indicates that the analysis and processing performance of the model to be evaluated for the question to be answered is better. Therefore, according to the vector similarity, the fact support attribute of the answer to be evaluated output by the model to be evaluated for the question to be answered can be determined, and then the model performance of the model to be evaluated can be evaluated according to the fact support attribute. This solves the problem of the lack of objective and effective evaluation method for the model performance of large language models in the prior art, resulting in inaccurate model performance evaluation results of large language models, and achieves a more objective and accurate effect of evaluating the model performance of large language models.
[0025] It should be understood that the contents described in this section are not intended to identify the key or important features of the embodiments of the present invention, nor are they intended to limit the scope of the present invention. Other features of the present invention will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS
[0026] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.
[0027] Figure 1 is a flow chart of a model performance evaluation method provided according to Embodiment 1 of the present invention;
[0028] Figure 2is a flow chart of a model performance evaluation method provided according to Embodiment 2 of the present invention;
[0029] Figure 3 is a structural schematic diagram of a model performance evaluation device provided according to Embodiment 3 of the present invention;
[0030] Figure 4 It is a schematic diagram of the structure of an electronic device for implementing the model performance evaluation method of an embodiment of the present invention. DETAILED DESCRIPTION
[0031] In order to enable those skilled in the art to better understand the scheme of the present invention, the technical scheme in the embodiment of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiment of the present invention. Obviously, the described embodiment is only an embodiment of a part of the present invention, not all of the embodiments. Based on the embodiments in the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention. The acquisition, transmission, storage, use, and processing of data in the technical scheme of this application are in compliance with the relevant provisions of national laws and regulations. It should be noted that in the embodiments of this application, certain software, components, or models that already exist in the industry may be mentioned. They should be considered as exemplary. Their purpose is only to illustrate the feasibility of the implementation of the technical scheme of this application, but it does not mean that the applicant has or will necessarily use the scheme.
[0032] It should be noted that the terms "first", "second", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the numbers used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein.
[0033] Embodiment 1
[0034] Figure 1 A flowchart of a model performance evaluation method is provided for the first embodiment of the present invention. This embodiment can be applied to obtaining at least one set of questions to be answered and standard answers corresponding to the questions to be answered based on text data, and outputting answers to be evaluated corresponding to the questions to be answered based on the model to be evaluated. After vectorizing the atomic facts corresponding to the standard answers and the atomic facts corresponding to the answers to be evaluated, the model performance of the model to be evaluated is evaluated according to the similarity between the fact embedding vector corresponding to the standard answer and the comparison vector corresponding to the answer to be evaluated. The method can be executed by a model performance evaluation device, which can be implemented in the form of hardware and / or software, and can be configured in a computing device that can execute the model performance evaluation method.
[0035] like Figure 1 As shown, the method includes:
[0036] S110: Extract at least one atomic fact from the text data to be used.
[0037] Among them, the text data to be used refers to the text data used to evaluate the model. For example, the text data to be used can be text data of professional knowledge in a certain field, or it can be text data in a specified application scenario, such as a transaction business scenario, a government scenario, or a medical scenario. In other words, the specific data content of the text data to be used is not limited in this technical solution, and it can be obtained according to actual needs. Atomic facts refer to the smallest granularity factual description, which includes the questions to be answered and the standard answers to the questions to be answered.
[0038] In actual applications, large language models are increasingly involved in professional applications, and in the application process, the factuality of large language models has gradually become prominent. For example, if a large language model provides incorrect or misleading content, it will cause users to misunderstand or spread wrong information during use, especially for fields that require high factual accuracy, which may cause huge losses. For example, in government scenarios, if the factuality of large language models cannot be guaranteed, it may cause problems such as errors in government affairs processes or incomplete materials in the application of large language models; in medical scenarios, medical users who rely on large language models may inadvertently endanger the health of patients due to the wrong information of large language models; in business decision-making scenarios, the analysis and information provided by large language models may cause companies to make unwise time decisions, etc. Therefore, in the actual use of large language models, it is necessary to ensure the factuality of the model to maximize the accuracy of the model in providing advice or reference information in various application scenarios.
[0039] Based on this, in order to evaluate the model performance of the large language model to be evaluated in the present technical solution, the text data to be used in the corresponding application scenario can be obtained according to actual needs, and then atomic facts can be extracted from the text data to be used, and at least one atomic fact can be input into the large language model to be evaluated, so as to perform factual verification based on the output results of the large language model, and determine whether the model performance of the large language model to be evaluated is good based on the verification results.
[0040] Optionally, extracting at least one atomic fact from the text data to be used includes: acquiring the text data to be used, and performing data preprocessing on the text data to be used to obtain target text data; and performing atomic fact splitting on the target text data to obtain at least one atomic fact.
[0041] The data preprocessing includes regular expression algorithm processing and / or text matching algorithm processing.
[0042] Specifically, taking the government affairs scenario as an example, the text data to be used in the government affairs scenario is reasonably and legally obtained from the authorized channels, and the text data to be used is cleaned to remove irrelevant content, duplicate data, and erroneous format information in the text data to be used to obtain the target text data, thereby ensuring the data quality of the target text data. For example, when cleaning the text data to be used, regular expressions and / or text matching algorithms can be used for data preprocessing to obtain the target text data.
[0043] Furthermore, the target text data is input into the model to be evaluated to perform atomic fact extraction on the target text data to obtain at least one atomic fact, and the obtained atomic fact is checked and modified by a professional user to ensure the accuracy and completeness of the atomic fact.
[0044] S120: For each fact atom, analyze and answer the question to be answered in the current fact atom based on the model to be evaluated, and obtain an answer to be evaluated corresponding to the question to be answered.
[0045] The model to be evaluated can be understood as a large language model that needs to be evaluated and verified for its factuality. For example, the model to be evaluated can be a deep learning model trained with a large amount of text data. In this technical solution, the questions generated based on fact atoms are called questions to be answered, the answer information generated based on the questions to be answered corresponding to the fact atoms is used as the standard answer, and the answer information obtained after analyzing and processing the questions to be answered based on the model to be evaluated is used as the answer to be evaluated.
[0046] It should be noted that atomic facts are usually factual descriptions of the smallest granularity. Corresponding questions and answers can be generated based on atomic facts. For example, the atomic fact is "Company A's office hours are from 8:30 to 17:30", and questions such as "What are Company A's office hours?" and "When does Company A start working?" can be generated based on atomic facts. Correspondingly, the questions generated based on atomic facts can also get corresponding answers, that is, "Company A's office hours are from 8:30 to 17:30" or "Company A starts working at 8:30 and ends working at 17:30". In order to verify the accuracy of the model to be evaluated in processing the questions to be answered, while generating the questions to be answered based on atomic facts, it is also necessary to generate standard answers corresponding to the questions to be answered. Furthermore, the questions to be answered are input into the model to be evaluated, so that after the model to be evaluated performs question analysis on the questions to be answered, the response information output by the model to be evaluated is used as the answer to be evaluated corresponding to the question to be answered, and then the model accuracy of the model to be evaluated is determined based on whether the answer to be evaluated is accurate.
[0047] S130: Determine, based on the pre-built target model, a fact embedding vector corresponding to the standard answer and a vector to be compared corresponding to the answer to be evaluated.
[0048] In the present technical solution, the target model refers to a neural network model used to vectorize the standard answers and the answers to be evaluated corresponding to the questions to be answered. For example, the target model can be a neural network model that combines a pre-trained language representation model (Bidirectional Encoder Representations from Transformers, BERT) with a downstream network training embedding model. The fact embedding vector can be understood as a vector obtained by vectorizing the standard answers based on the target model. The vector to be compared refers to a vector obtained by vectorizing the answers to be evaluated based on the target model.
[0049] It is understandable that the target model needs to be trained before actually using it to ensure the accuracy of the target model when vectorizing the standard answer or the answer to be evaluated.
[0050] It should be noted that in this technical solution, in order to facilitate the processing of atomic facts after obtaining the atomic facts, the atomic facts can be subjected to entity extraction to obtain a fact chain corresponding to the atomic facts, and then vectorization processing is performed based on the target model on the basis of the fact chain to obtain a fact embedding vector corresponding to the standard answer, or a vector to be compared corresponding to the answer to be evaluated. For example, the atomic fact is "Company A's office hours are 8:30 to 17:30", and the fact chain obtained after entity extraction is "Company A - Office Hours - 8:30 to 17:30".
[0051] Optionally, based on a pre-built target model, a fact embedding vector corresponding to the standard answer and a vector to be compared corresponding to the answer to be evaluated are determined, including: calling a pre-trained target model; performing entity extraction on the standard answer to obtain a first entity triplet, and inputting the first entity triplet into the target model to obtain the fact embedding vector corresponding to the standard answer; performing entity extraction on the answer to be evaluated to obtain a second entity triplet, and inputting the second entity triplet into the target model to obtain the vector to be compared corresponding to the answer to be evaluated.
[0052] In this technical solution, the first entity triple can be obtained by performing entity extraction on the atomic facts including the question to be answered and the standard answer, and the second entity triple can be obtained by extracting the atomic facts including the question to be answered and the answer to be evaluated. Furthermore, if the first entity triple is input into the target model, a fact embedding vector can be obtained, and if the second entity triple is input into the target model, a vector to be compared can be obtained.
[0053] It should be noted that before actually using the target model to vectorize the standard answers and answers to be evaluated corresponding to the questions to be answered, it is necessary to pre-build the target model. The target model includes a pre-trained language model (i.e., BERT model) and a pre-set embedding model, which is a neural network model including an attention layer and a fully connected layer.
[0054] Specifically, a downstream network training embedding model is added on the basis of the BERT model, wherein the BERT model has powerful context understanding and word vector generation capabilities, and is suitable for fact embedding. After extracting at least one atomic fact from the text data to be used, a first entity triplet can be obtained by performing entity extraction on the atomic fact. Correspondingly, after the atomic fact is input into the model to be evaluated, a second entity triplet can be obtained based on the atomic facts corresponding to the answer to be evaluated and the question to be answered output by the model to be evaluated. The first entity triplet or the second entity triplet is input into the BERT model as a token, and a special token "[CLS]" is added in front. The "[CLS]" vector output by the BERT model is used as the fact embedding vector corresponding to the first entity triplet, or the vector to be compared corresponding to the second entity triplet.
[0055] It is understandable that in the process of training the target model, data from different application scenarios can be used for model training, where the fact chain data in the same scenario is used as positive samples, and the data from different scenarios is used as negative samples for comparative learning. Through comparative learning, the discrimination between positive and negative samples is used to improve the model's ability to distinguish different fact chains. The loss function of the training process is defined as follows.
[0056]
[0057] Where D is the training data set, t is each item in the data set, and t + is a positive sample, t - is a negative sample.
[0058] S140: Determine the fact support attribute of the answer to be evaluated for the question to be answered based on the fact embedding vector and the vector to be compared.
[0059] Among them, the fact support attribute can be used to characterize the similarity between the vector to be compared and the real-time embedding vector. The higher the fact support attribute, the higher the similarity between the vector to be compared and the fact embedding vector. Conversely, the lower the fact support attribute, the lower the similarity between the vector to be compared and the fact embedding vector. Correspondingly, since the fact embedding vector is a vector generated based on the question to be answered and the corresponding standard answer, and the vector to be compared is a vector generated based on the question to be answered and the answer to be evaluated, the similarity between the answer to be evaluated and the standard answer of the question to be answered can also be determined based on the fact support attribute.
[0060] When the model performance of the model to be evaluated is actually evaluated, at least one question to be answered can be input into the model to be evaluated, and the answer to be evaluated corresponding to each question to be answered can be output based on the model to be evaluated. Therefore, the model performance of the model to be evaluated can be evaluated according to the accuracy of the answer to be evaluated. It should be noted that in order to facilitate the evaluation of the accuracy of the answer to be evaluated, the technical solution is to obtain the fact embedding vector by vectorizing the question to be answered and the corresponding standard answer, and to obtain the vector to be compared after vectorizing the question to be answered and the corresponding answer to be evaluated, and then determine the accuracy of the answer to be evaluated according to the vector similarity between the comparison vector and the fact embedding vector (i.e., the fact support attribute).
[0061] Optionally, determining the fact support attribute of the answer to be evaluated for the question to be answered based on the fact embedding vector and the vector to be compared includes: determining the vector similarity corresponding to the fact embedding vector and the vector to be compared based on a similarity algorithm, and comparing the fact support attribute of the answer to be evaluated for the question to be answered.
[0062] The similarity algorithm is used to calculate the similarity between the fact embedding vector and the vector to be compared. For example, the similarity algorithm can be a cosine similarity algorithm, a Euclidean distance algorithm, or a Manhattan distance similarity algorithm.
[0063] Exemplarily, taking the cosine similarity algorithm as an example, the similarity between the fact embedding vector and the vector to be compared is calculated based on the cosine similarity calculation function to obtain the vector similarity, and the fact support attribute of the answer to be evaluated for the question to be answered is characterized based on the vector similarity. The closer the fact support attribute is to 1, the higher the similarity between the vector to be compared and the fact embedding vector, and accordingly, the higher the fact support attribute of the answer to be evaluated for the question to be answered.
[0064] S150: Determine a model performance evaluation result of the model to be evaluated based on the fact support attributes of all the answers to be evaluated for the corresponding questions to be answered.
[0065] The model performance evaluation result may be good model performance or poor model performance.
[0066] Based on the above example, a fact support attribute threshold can also be pre-set. If the fact support attribute is greater than the fact support attribute threshold, it means that the answer to be evaluated has a high degree of match with the question to be answered, indicating that the answer information output by the model to be evaluated can meet the answer requirements of the question to be answered. Correspondingly, the model evaluation result corresponding to the model to be evaluated is. On the contrary, if the fact support attribute is less than the fact support attribute threshold, it means that the answer to be evaluated has a low degree of match with the question to be answered, indicating that the answer information output by the model to be evaluated cannot meet the answer requirements of the question to be answered. Correspondingly, the model evaluation result corresponding to the model to be evaluated is poor model performance.
[0067] The technical solution of the embodiment of the present invention is to extract at least one atomic fact from the text data to be used; for each fact atom, analyze and answer the question to be answered in the current fact atom based on the model to be evaluated, and obtain the answer to be evaluated corresponding to the question to be answered; determine the fact embedding vector corresponding to the standard answer and the vector to be compared corresponding to the answer to be evaluated based on the pre-built target model; determine the fact support attribute of the answer to be evaluated for the question to be answered based on the fact embedding vector and the vector to be compared; determine the model performance evaluation result of the model to be evaluated according to the fact support attributes of all the answers to be evaluated for the corresponding question to be answered. In this technical solution, in order to evaluate the model performance of the model to be evaluated, atomic facts can be extracted from the text data to be used, and the obtained atomic facts can be vectorized based on the pre-built target model, and the fact embedding vector generated by the atomic facts corresponding to the question to be answered and the standard answer is calculated with the comparison vector generated by the atomic facts corresponding to the question to be answered and the answer to be evaluated output by the model to be evaluated to obtain the vector similarity. It should be noted that the vector similarity between the fact embedding vector and the vector to be compared can be used to measure the matching degree between the answer to be evaluated and the question to be answered. The higher the vector similarity, the higher the matching degree between the answer to be evaluated and the question to be answered. Correspondingly, it indicates that the analysis and processing performance of the model to be evaluated for the question to be answered is better. Therefore, according to the vector similarity, the fact support attribute of the answer to be evaluated output by the model to be evaluated for the question to be answered can be determined, and then the model performance of the model to be evaluated can be evaluated according to the fact support attribute. This solves the problem of the lack of objective and effective evaluation method for the model performance of large language models in the prior art, resulting in inaccurate model performance evaluation results of large language models, and achieves a more objective and accurate effect of evaluating the model performance of large language models.
[0068] Embodiment 2
[0069] Figure 2A flowchart of a model performance evaluation method provided in Embodiment 2 of the present invention, optionally, the model performance evaluation result of the model to be evaluated is determined to be refined based on the factual support attributes of all answers to be evaluated for the corresponding questions to be answered.
[0070] like Figure 2 As shown, the method includes:
[0071] S210: Extract at least one atomic fact from the text data to be used.
[0072] S220: For each fact atom, analyze and answer the question to be answered in the current fact atom based on the model to be evaluated, and obtain an answer to be evaluated corresponding to the question to be answered.
[0073] S230: Determine, based on the pre-built target model, a fact embedding vector corresponding to the standard answer and a vector to be compared corresponding to the answer to be evaluated.
[0074] S240: Determine the fact support attribute of the answer to be evaluated for the question to be answered based on the fact embedding vector and the vector to be compared.
[0075] S250: For each question to be answered, if the corresponding fact support attribute of the answer to be evaluated is greater than the preset fact support attribute, then the answer result of the model to be evaluated to the question to be answered is determined to be a correct result.
[0076] The preset fact support attribute refers to a preset fact support attribute threshold, which is used to measure the quality of the response of the model to be evaluated to the question to be answered based on whether the answer to be evaluated is greater than the preset fact support attribute.
[0077] In practical applications, the model to be evaluated obtains the answer to be evaluated after analyzing the question to be answered. Therefore, the model performance of the model to be evaluated can be evaluated based on whether the answer to be evaluated corresponds to the standard answer to the question to be answered.
[0078] Specifically, the model performance of the model to be evaluated can be evaluated by comparing the fact support attribute between the comparison vector corresponding to the answer to be evaluated and the fact embedding vector corresponding to the standard answer. If the fact support attribute of the answer to be evaluated to the question to be answered is greater than the preset fact support attribute, it means that the answer quality of the answer to be evaluated output by the model to be evaluated is good, and the answer result of the model to be evaluated to the question to be answered is determined to be the correct result.
[0079] S260: If the corresponding fact support attribute of the answer to be evaluated is less than the preset fact support attribute, it is determined that the answer result of the model to be evaluated to the question to be answered is an incorrect result.
[0080] Based on the above example, if the fact support attribute corresponding to the answer to be evaluated is less than the preset fact support attribute, it means that the response quality of the answer to be evaluated output by the model to be evaluated is poor, and it is determined that the answer result of the model to be evaluated to the question to be answered is an incorrect result.
[0081] S270: Determine the model performance evaluation result corresponding to the model to be evaluated based on the answer results of the model to be evaluated to all the questions to be answered.
[0082] Among them, the model performance evaluation result is good model performance or poor model performance.
[0083] Specifically, when actually performing model performance evaluation on the model to be evaluated, it is usually necessary to obtain corresponding answers to be evaluated after analyzing and processing multiple questions to be answered based on the model to be evaluated, and determine the model performance evaluation result of the model to be evaluated based on the factual support attributes of the multiple answers to be evaluated for the corresponding questions to be answered.
[0084] Optionally, the model performance evaluation result corresponding to the model to be evaluated is determined based on the answer results of the model to be evaluated to all the questions to be answered, including: obtaining the total result frequency based on the sum of the occurrence frequencies of correct results and incorrect results output by the model to be evaluated; obtaining the incorrect result frequency by determining the occurrence batches of incorrect results output by the model to be evaluated; and determining the model performance evaluation result of the model to be evaluated based on the incorrect result frequency and the total result frequency.
[0085] For example, 100 questions to be answered are analyzed and processed based on the model to be evaluated to obtain the answers to be evaluated corresponding to the 100 questions to be answered, and each group of questions to be answered and the corresponding answers to be evaluated are vectorized based on the pre-built target model to obtain the vectors to be compared corresponding to each answer to be evaluated. At the same time, each group of questions to be answered and the corresponding standard answers are vectorized based on the target model to obtain the corresponding fact embedding vectors.
[0086] Furthermore, the fact support attribute between the vector to be compared and the fact embedding vector corresponding to each question to be answered is calculated based on the similarity algorithm. If the fact support attribute is greater than the preset fact support attribute, the answer result of the model to be evaluated to the corresponding question to be answered is determined to be a correct result; conversely, if the fact support attribute is less than the preset fact support attribute, the answer result of the model to be evaluated to the corresponding question to be answered is determined to be an incorrect result. It can be seen that 100 answers to be evaluated can be obtained by answering 100 questions to be answered based on the model to be evaluated, that is, the total result frequency is 100, among which the number of correct results is 98 times and the number of incorrect results is 2 times, then the correct result frequency is 98, and the incorrect result frequency is 2, so as to determine the model performance evaluation result of the model to be evaluated according to the incorrect result frequency and the total result frequency.
[0087] Based on the above example, the model performance evaluation result of the model to be evaluated is determined according to the frequency of error results and the total result frequency: including: according to the ratio of the error result frequency to the total result frequency, the error result proportion is obtained. If the error result proportion is less than the preset error result proportion, the model performance evaluation result corresponding to the model to be evaluated is determined to be good model performance; if the error result proportion is greater than the preset error result proportion, the model performance evaluation result corresponding to the model to be evaluated is determined to be poor model performance.
[0088] The preset error result ratio can be understood as the maximum error tolerance rate of the error results of the answer to be evaluated output by the model to be evaluated.
[0089] Specifically, according to the ratio of the frequency of wrong results to the total frequency of results, the proportion of wrong results can be obtained as 2%. Assuming that the preset proportion of wrong results is 5%, the proportion of wrong results is less than the preset proportion of wrong results at this time, and accordingly, the model performance evaluation result of the model to be evaluated can be determined as good model performance. On the contrary, if the proportion of wrong results is greater than the preset proportion of wrong results, it indicates that the number of wrong results of the answers to be evaluated output by the model to be evaluated is large, and the model evaluation result of the model to be evaluated is determined as poor model performance.
[0090] The technical solution of the embodiment of the present invention is to extract at least one atomic fact from the text data to be used; for each fact atom, analyze and answer the question to be answered in the current fact atom based on the model to be evaluated, and obtain the answer to be evaluated corresponding to the question to be answered; determine the fact embedding vector corresponding to the standard answer and the vector to be compared corresponding to the answer to be evaluated based on the pre-built target model; determine the fact support attribute of the answer to be evaluated for the question to be answered based on the fact embedding vector and the vector to be compared; determine the model performance evaluation result of the model to be evaluated according to the fact support attributes of all the answers to be evaluated for the corresponding question to be answered. In this technical solution, in order to evaluate the model performance of the model to be evaluated, atomic facts can be extracted from the text data to be used, and the obtained atomic facts can be vectorized based on the pre-built target model, and the fact embedding vector generated by the atomic facts corresponding to the question to be answered and the standard answer is calculated with the comparison vector generated by the atomic facts corresponding to the question to be answered and the answer to be evaluated output by the model to be evaluated to obtain the vector similarity. It should be noted that the vector similarity between the fact embedding vector and the vector to be compared can be used to measure the matching degree between the answer to be evaluated and the question to be answered. The higher the vector similarity, the higher the matching degree between the answer to be evaluated and the question to be answered. Correspondingly, it indicates that the analysis and processing performance of the model to be evaluated for the question to be answered is better. Therefore, according to the vector similarity, the fact support attribute of the answer to be evaluated output by the model to be evaluated for the question to be answered can be determined, and then the model performance of the model to be evaluated can be evaluated according to the fact support attribute. This solves the problem of the lack of objective and effective evaluation method for the model performance of large language models in the prior art, resulting in inaccurate model performance evaluation results of large language models, and achieves a more objective and accurate effect of evaluating the model performance of large language models.
[0091] Embodiment 3
[0092] Figure 3 This is a schematic diagram of the structure of a model performance evaluation device provided in Embodiment 3 of the present invention. Figure 3 As shown, the apparatus includes: an atomic fact extraction module 310 , a to-be-evaluated answer determination module 320 , a vector determination module 330 , an attribute determination module 340 and an evaluation result determination module 350 .
[0093] The atomic fact extraction module 310 is used to extract at least one atomic fact from the text data to be used; wherein the atomic fact refers to the smallest granularity factual description, and the atomic fact includes the question to be answered and the standard answer corresponding to the question to be answered;
[0094] The module 320 for determining the answer to be evaluated is used to analyze and answer the question to be answered in the current fact atom based on the model to be evaluated for each fact atom, and obtain the answer to be evaluated corresponding to the question to be answered;
[0095] A vector determination module 330, for determining a fact embedding vector corresponding to the standard answer and a vector to be compared corresponding to the answer to be evaluated based on a pre-built target model;
[0096] An attribute determination module 340 is used to determine the fact support attribute of the answer to be evaluated for the question to be answered based on the fact embedding vector and the vector to be compared;
[0097] The evaluation result determination module 350 is used to determine the model performance evaluation result of the model to be evaluated based on the fact support attributes of all the answers to be evaluated for the corresponding questions to be answered.
[0098] The technical solution of the embodiment of the present invention is to extract at least one atomic fact from the text data to be used; for each fact atom, analyze and answer the question to be answered in the current fact atom based on the model to be evaluated, and obtain the answer to be evaluated corresponding to the question to be answered; determine the fact embedding vector corresponding to the standard answer and the vector to be compared corresponding to the answer to be evaluated based on the pre-built target model; determine the fact support attribute of the answer to be evaluated for the question to be answered based on the fact embedding vector and the vector to be compared; determine the model performance evaluation result of the model to be evaluated according to the fact support attributes of all the answers to be evaluated for the corresponding question to be answered. In this technical solution, in order to evaluate the model performance of the model to be evaluated, atomic facts can be extracted from the text data to be used, and the obtained atomic facts can be vectorized based on the pre-built target model, and the fact embedding vector generated by the atomic facts corresponding to the question to be answered and the standard answer is calculated with the comparison vector generated by the atomic facts corresponding to the question to be answered and the answer to be evaluated output by the model to be evaluated to obtain the vector similarity. It should be noted that the vector similarity between the fact embedding vector and the vector to be compared can be used to measure the matching degree between the answer to be evaluated and the question to be answered. The higher the vector similarity, the higher the matching degree between the answer to be evaluated and the question to be answered. Correspondingly, it indicates that the analysis and processing performance of the model to be evaluated for the question to be answered is better. Therefore, according to the vector similarity, the fact support attribute of the answer to be evaluated output by the model to be evaluated for the question to be answered can be determined, and then the model performance of the model to be evaluated can be evaluated according to the fact support attribute. This solves the problem of the lack of objective and effective evaluation method for the model performance of large language models in the prior art, resulting in inaccurate model performance evaluation results of large language models, and achieves a more objective and accurate effect of evaluating the model performance of large language models.
[0099] Optionally, the atomic fact extraction module includes: a text data acquisition submodule, which is used to acquire the text data to be used, and perform data preprocessing on the text data to be used to obtain target text data; wherein the data preprocessing includes regular expression algorithm processing and / or text matching algorithm processing;
[0100] The atomic fact extraction submodule is used to split the target text data into atomic facts to obtain at least one atomic fact.
[0101] Optionally, the vector determination module includes: a model retrieval submodule, used to retrieve a pre-trained target model; wherein the target model includes a pre-trained language model and a pre-set embedding model, and the embedding model is a neural network model including an attention layer and a fully connected layer;
[0102] A fact embedding vector determination submodule is used to extract entities from the standard answer to obtain a first entity triplet, and input the first entity triplet into the target model to obtain a fact embedding vector corresponding to the standard answer;
[0103] The submodule for determining the vector to be compared is used to extract entities from the answer to be evaluated to obtain a second entity triplet, and input the second entity triplet into the target model to obtain a vector to be compared corresponding to the answer to be evaluated.
[0104] Optionally, an attribute determination module is used to determine the vector similarity corresponding to the fact embedding vector and the vector to be compared based on a similarity algorithm, and to assign the fact support attribute of the answer to be evaluated to the question to be answered.
[0105] Optionally, the evaluation result determination module includes: a correct result determination submodule, for determining, for each question to be answered, if the corresponding fact support attribute of the answer to be evaluated is greater than a preset fact support attribute, that the answer result of the model to be evaluated to the question to be answered is a correct result;
[0106] The error result determination submodule is used to determine that the answer result of the model to be evaluated to the question to be answered is an error result if the corresponding fact support attribute of the answer to be evaluated is less than the preset fact support attribute;
[0107] The evaluation result determination submodule is used to determine the model performance evaluation result corresponding to the model to be evaluated based on the answer results of the model to be evaluated to all the questions to be answered.
[0108] Optionally, the evaluation result determination submodule includes: a total result frequency determination unit, which is used to obtain the total result frequency according to the sum of the occurrence frequencies of correct results and incorrect results output by the model to be evaluated;
[0109] An error result frequency determination unit, used to determine the frequency of error results obtained in the batch of error results output by the model to be evaluated;
[0110] The evaluation result determination unit is used to determine the model performance evaluation result of the model to be evaluated based on the error result frequency and the total result frequency.
[0111] Optionally, the evaluation result determination unit includes: a proportion determination subunit, which is used to obtain the proportion of incorrect results according to the ratio of the frequency of incorrect results to the total frequency of results,
[0112] A first evaluation result determination subunit is used to determine that the model performance evaluation result corresponding to the model to be evaluated is good model performance if the error result ratio is less than a preset error result ratio;
[0113] The second evaluation result determination subunit is used to determine that the model performance evaluation result corresponding to the model to be evaluated is poor model performance if the proportion of error results is greater than a preset proportion of error results.
[0114] The model performance evaluation device provided in the embodiment of the present invention can execute the model performance evaluation method provided in any embodiment of the present invention, and has the corresponding functional modules and beneficial effects of the execution method.
[0115] Embodiment 4
[0116] Figure 4 A schematic diagram of the structure of an electronic device 10 of an embodiment of the present invention is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices (such as helmets, glasses, watches, etc.) and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present invention described and / or required herein.
[0117] like Figure 4 As shown, the electronic device 10 includes at least one processor 11, and a memory connected to the at least one processor 11, such as a read-only memory (ROM) 12, a random access memory (RAM) 13, etc., wherein the memory stores a computer program that can be executed by at least one processor, and the processor 11 can perform various appropriate actions and processes according to the computer program stored in the read-only memory (ROM) 12 or the computer program loaded from the storage unit 18 to the random access memory (RAM) 13. In the RAM 13, various programs and data required for the operation of the electronic device 10 can also be stored. The processor 11, the ROM 12, and the RAM 13 are connected to each other through a bus 14. An input / output (I / O) interface 15 is also connected to the bus 14.
[0118] A number of components in the electronic device 10 are connected to the I / O interface 15, including: an input unit 16, such as a keyboard, a mouse, etc.; an output unit 17, such as various types of displays, speakers, etc.; a storage unit 18, such as a disk, an optical disk, etc.; and a communication unit 19, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 19 allows the electronic device 10 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.
[0119] The processor 11 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the processor 11 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The processor 11 executes the various methods and processes described above, such as the model performance evaluation method.
[0120] In some embodiments, the model performance evaluation method may be implemented as a computer program, which is tangibly contained in a computer-readable storage medium, such as a storage unit 18. In some embodiments, part or all of the computer program may be loaded and / or installed on the electronic device 10 via the ROM 12 and / or the communication unit 19. When the computer program is loaded into the RAM 13 and executed by the processor 11, one or more steps of the model performance evaluation method described above may be performed. Alternatively, in other embodiments, the processor 11 may be configured to execute the model performance evaluation method in any other appropriate manner (e.g., by means of firmware).
[0121] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.
[0122] The computer programs for implementing the model performance evaluation method of the present invention can be written in any combination of one or more programming languages. These computer programs can be provided to a processor of a general-purpose computer, a special-purpose computer or other programmable data processing device, so that when the computer program is executed by the processor, the functions / operations specified in the flow chart and / or block diagram are implemented. The computer program can be executed entirely on the machine, partially on the machine, partially on the machine as a stand-alone software package and partially on a remote machine, or entirely on a remote machine or server.
[0123] In the context of the present invention, a computer-readable storage medium may be a tangible medium that may contain or store a computer program for use by or in combination with an instruction execution system, device or equipment. A computer-readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices or equipment, or any suitable combination of the foregoing. Alternatively, a computer-readable storage medium may be a machine-readable signal medium. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.
[0124] To provide interaction with a user, the systems and techniques described herein may be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and a pointing device (e.g., a mouse or trackball) through which the user can provide input to the electronic device. Other types of devices may also be used to provide interaction with the user; for example, the feedback provided to the user may be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user may be received in any form (including acoustic input, voice input, or tactile input).
[0125] The systems and techniques described herein may be implemented in a computing system that includes backend components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes frontend components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such backend components, middleware components, or frontend components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), a blockchain network, and the Internet.
[0126] A computing system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The client and server relationship is generated by computer programs running on the corresponding computers and having a client-server relationship with each other. The server may be a cloud server, also known as a cloud computing server or cloud host, which is a host product in the cloud computing service system to solve the defects of difficult management and weak business scalability in traditional physical hosts and VPS services.
[0127] Embodiment 5
[0128] An embodiment of the present invention also provides a computer program product, including a computer program, which, when executed by a processor, implements the model performance evaluation method provided in any embodiment of the present application.
[0129] In the process of implementation, the computer program product can be written in one or more programming languages or a combination thereof to perform the computer program code of the present invention, including object-oriented programming languages, such as Java, Smalltalk, C++, and conventional procedural programming languages, such as "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer can be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or can be connected to an external computer (for example, using an Internet service provider to connect through the Internet).
[0130] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps described in the present invention can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solution of the present invention can be achieved, and this document does not limit this.
[0131] The above specific implementations do not constitute a limitation on the protection scope of the present invention. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present invention should be included in the protection scope of the present invention.
Claims
1. A model performance evaluation method, It is characterized in that include: Extracting at least one atomic fact from the text data to be used; wherein the atomic fact refers to a factual description of the smallest granularity, and the atomic fact includes a question to be answered and a standard answer corresponding to the question to be answered; For each fact atom, based on the model to be evaluated, the question to be answered in the current fact atom is analyzed and answered to obtain an answer to be evaluated corresponding to the question to be answered; Determine, based on a pre-built target model, a fact embedding vector corresponding to the standard answer and a vector to be compared corresponding to the answer to be evaluated; Determine, based on the fact embedding vector and the vector to be compared, a fact support attribute of the answer to be evaluated for the question to be answered; The model performance evaluation result of the model to be evaluated is determined according to the fact support attributes of all the answers to be evaluated for the corresponding questions to be answered.
2. The method according to claim 1, It is characterized in that The step of extracting at least one atomic fact from the text data to be used comprises: Acquire the text data to be used, and perform data preprocessing on the text data to be used to obtain target text data; wherein the data preprocessing includes regular expression algorithm processing and / or text matching algorithm processing; The target text data is split into atomic facts to obtain at least one atomic fact.
3. The method according to claim 1, It is characterized in that The determining, based on the pre-built target model, a fact embedding vector corresponding to the standard answer and a vector to be compared corresponding to the answer to be evaluated includes: Retrieving a pre-trained target model; wherein the target model includes a pre-trained language model and a pre-set embedding model, and the embedding model is a neural network model including an attention layer and a fully connected layer; Performing entity extraction on the standard answer to obtain a first entity triplet, and inputting the first entity triplet into the target model to obtain a fact embedding vector corresponding to the standard answer; Perform entity extraction on the answer to be evaluated to obtain a second entity triplet, and input the second entity triplet into the target model to obtain a vector to be compared corresponding to the answer to be evaluated.
4. The method according to claim 1, It is characterized in that The determining, based on the fact embedding vector and the vector to be compared, the fact support attribute of the answer to be evaluated for the question to be answered comprises: Based on a similarity algorithm, the vector similarity between the fact embedding vector and the vector to be compared is determined, and the fact support attribute of the answer to be evaluated for the question to be answered is calculated.
5. The method according to claim 1, It is characterized in that Determining the model performance evaluation result of the model to be evaluated based on the fact support attributes of all the answers to be evaluated for the corresponding questions to be answered, includes: For each question to be answered, if the corresponding fact support attribute of the answer to be evaluated is greater than the preset fact support attribute, then the answer result of the model to be evaluated to the question to be answered is determined to be a correct result; If the fact support attribute of the answer to be evaluated is less than the preset fact support attribute, then it is determined that the answer result of the model to be evaluated to the question to be answered is an incorrect result; According to the answer results of the model to be evaluated to all the questions to be answered, the model performance evaluation result corresponding to the model to be evaluated is determined.
6. The method according to claim 5, It is characterized in that Determining the model performance evaluation result corresponding to the model to be evaluated based on the answer results of the model to be evaluated to all the questions to be answered includes: Obtaining the total result frequency according to the sum of the occurrence frequencies of correct results and incorrect results output by the model to be evaluated; Determine the frequency of erroneous results output by the model to be evaluated in batches where erroneous results occur; The model performance evaluation result of the model to be evaluated is determined according to the error result frequency and the total result frequency.
7. The method according to claim 6, It is characterized in that The step of determining the model performance evaluation result of the model to be evaluated according to the error result frequency and the total result frequency includes: According to the ratio of the error result frequency to the total result frequency, the error result ratio is obtained. If the error result ratio is less than the preset error result ratio, it is determined that the model performance evaluation result corresponding to the model to be evaluated is that the model performance is good; If the error result ratio is greater than a preset error result ratio, it is determined that the model performance evaluation result corresponding to the model to be evaluated is poor model performance.
8. A model performance evaluation device, It is characterized in that include: An atomic fact extraction module, used to extract at least one atomic fact from the text data to be used; wherein the atomic fact refers to a factual description of the smallest granularity, and the atomic fact includes a question to be answered and a standard answer corresponding to the question to be answered; The module for determining the answer to be evaluated is used to analyze and answer the question to be answered in the current fact atom based on the model to be evaluated for each fact atom, and obtain the answer to be evaluated corresponding to the question to be answered; A vector determination module, used to determine the fact embedding vector corresponding to the standard answer and the vector to be compared corresponding to the answer to be evaluated based on a pre-built target model; An attribute determination module, configured to determine a fact support attribute of the answer to be evaluated for the question to be answered based on the fact embedding vector and the vector to be compared; The evaluation result determination module is used to determine the model performance evaluation result of the model to be evaluated based on the fact support attributes of all the answers to be evaluated for the corresponding questions to be answered.
9. An electronic device, It is characterized in that The electronic device comprises: at least one processor; and a memory communicatively connected to the at least one processor; wherein, The memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can perform the model performance evaluation method according to any one of claims 1 to 7.
10. A computer-readable storage medium, It is characterized in that The computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to implement the model performance evaluation method according to any one of claims 1 to 7 when executed.
Citation Information
Cited By
Government affair large model detection-oriented method, system and equipment
CN120234396A
Large language model evaluation method and system, storage medium and terminal equipment
CN121146096A
