Performance evaluation method and device of large language model, readable storage medium

Abstract

The application discloses a large language model performance evaluation method and device, and a readable storage medium. The method comprises the following steps: in response to an evaluation request of a target large language model, a target category question bank is queried from a plurality of category question banks, wherein the plurality of category question banks comprise a plurality of question banks for storing questions and prompt words of different categories; the questions and prompt words of the target category question bank are input into the target large language model to obtain corresponding response contents of each question; the quality of the response contents corresponding to each question is scored by an intelligent judging model combined with a scoring rule corresponding to a multi-dimensional quality parameter to obtain a performance score of the target large language model; and the performance of the target large language model is evaluated based on the performance score.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and in particular to a performance evaluation method and apparatus for large language models, as well as a readable storage medium. Background Technology

[0002] With the rapid development of artificial intelligence technology, especially the widespread application of large language models (LLMs) in natural language processing, content generation and other fields, users such as enterprises and research institutions have an increasing demand for performance evaluation of large language models. They need to evaluate the performance of large language models that are not good or have low accuracy in order to provide subsequent improvement solutions.

[0003] Currently, large language models are typically evaluated manually or semi-automatically. This process relies heavily on manual intervention, resulting in high time costs and cumbersome verification procedures. It is particularly problematic when facing model evaluation and comparison requests from the back office or product lines, often leading to slow responses. This results in excessively long evaluation cycles, failing to meet the timeliness requirements of rapid model iteration and business decision-making.

[0004] Furthermore, manual evaluation is easily influenced by the subjective judgment of the evaluators, leading to a lack of consistency and objectivity in the evaluation results. Different evaluators may give different evaluations of the same model, which undermines the credibility of the evaluation report.

[0005] This results in low efficiency in evaluating the performance of large language models, severely restricting the efficiency and quality of model development and deployment. Summary of the Invention

[0006] The purpose of this application is to provide a method and apparatus for evaluating the performance of large language models, as well as a readable storage medium, to solve the problem of low efficiency in evaluating the performance of large language models.

[0007] To solve the above-mentioned technical problems, this specification is implemented as follows: Firstly, a performance evaluation method for large language models is provided, including: In response to the evaluation request of the target large language model, the target category question bank is queried from the multi-category question bank, which includes multiple question banks for storing questions and prompt words of different categories; Input the questions and prompts from the target category question bank into the target large language model to obtain the corresponding response content for each question; The quality of the responses to each question is scored by combining an intelligent judging model with scoring rules corresponding to multi-dimensional quality parameters, so as to obtain the performance score of the target large language model. The performance of the target large language model is evaluated based on the performance score.

[0008] Optionally, before querying the target category question bank from the multi-category question bank, the following steps are also included: Based on the assessment requirements, different categories of question banks are created, including knowledge-based question and answer, ethics and safety, professional fields, personal information query, and corporate information query. The question bank is expanded by obtaining questions of different categories through manual input, batch import and / or web crawling. The metadata of each question includes the question content, question category and difficulty level. Configure unique prompts for questions in each category of the question bank; The multi-category question bank is composed of question banks that include questions and corresponding prompts based on different categories.

[0009] Optionally, the quality of the responses to each question is scored using an intelligent judging model combined with scoring rules corresponding to multi-dimensional quality parameters, to obtain the performance score of the target large language model, including: The target large language model provides each question, corresponding prompt words, and response content to the target intelligent judge model; Each question is input into the target intelligent judge model to obtain the corresponding response content for each question; The quality of the target large language model's response to each question is verified by comparing the response content of the target intelligent judge model to each question with the response content of the target large language model using multi-dimensional quality parameters. The multi-dimensional quality parameters include at least two of the following: authenticity, verifiability, redundancy, qualified security, and comprehensiveness. Based on the scoring rules, the verification results of each dimension of quality parameters are quantitatively scored to obtain the quality score of each response content of the target large language model. The quality scores of all responses from the target large language model are summarized to obtain the performance score of the target large language model.

[0010] Optionally, the scoring rules include general scoring rules applicable to all categories of large language models, and the multi-dimensional quality parameters include the authenticity, verifiability, redundancy, and qualified security of the response content; Based on the scoring rules, the verification results of each dimension of quality parameters are quantitatively scored to obtain the quality score of each response content of the target large language model, including: The verification results for the authenticity, verifiability, and redundancy of each response content of the target large language model are determined respectively; The verification results of each response are quantitatively scored based on the general scoring rules to obtain a quality score for each response.

[0011] Optionally, the verification results of each response are quantitatively scored based on the general scoring rules, including: If the verification result of the authenticity of the target response content is negative, then the first score is deducted from the target quality score of the target response content based on the general scoring rules. If the verification result of the verifiability of the target response content is negative, then a second score is deducted from the target quality score based on the general scoring rules. If the verification result of the redundancy of the target response content is insufficient in terms of professionalism or logical rigor, then a third score will be deducted from the target quality score based on the general scoring rules. If the verification result of the target response content corresponding to qualified security is negative, then a fourth score is deducted from the target quality score of the target response content based on the general scoring rules. The score obtained by subtracting the first score, the second score, the third score, and the fourth score from the target quality score is determined as the quality score of the target response content.

[0012] Optionally, the scoring rules also include specific scoring rules for a large language model applicable to the target category, and the multi-dimensional quality parameters also include the comprehensiveness of the response content; The verification results of each dimension's quality parameters are quantitatively scored based on the aforementioned scoring rules to obtain the quality score of each response content of the target large language model, and the method further includes: Determine whether the target category is a person information query or a company information query; If the target category is a person information query or a company information query, then based on the verification results of the comprehensiveness of each response content according to the specific scoring rules, the quality score obtained by quantifying and scoring each response content according to the general scoring rules is adjusted.

[0013] Optionally, based on the verification results of the comprehensiveness of each response content according to the specific scoring rules, the quality score obtained by quantitatively scoring each response content according to the general scoring rules is adjusted, including: Obtain the number of preset first dimensions corresponding to the target category, where the number of first dimensions represents the number of dimensions corresponding to the information included in the response content of the target category when the comprehensiveness is at its highest. Determine the number of second-dimensional elements corresponding to the information included in the target response; If the verification result of the comprehensiveness of the target response content is that the number of the second dimension is lower than the number of the first dimension, and the difference between the number of the first dimension and the number of the second dimension is a first value, then based on the special scoring rule, the fifth score is deducted from the target quality score of the quantitative scoring of the target response content by the general scoring rule, so as to adjust the quality score obtained by the quantitative scoring of the target response content by the general scoring rule. If the verification result of the comprehensiveness of the target response content is that the number of the second dimension is lower than the number of the first dimension, and the difference between the number of the first dimension and the number of the second dimension is a second value, then a sixth score is deducted from the target quality score based on the specific scoring rule to adjust the quality score obtained by the quantitative scoring of the target response content by the general scoring rule, wherein the second value is greater than the first value, and the sixth score is greater than the fifth score; If the comprehensiveness verification result corresponding to the target response content is that the number of the second dimension is equal to the number of the first dimension, then the target quality score remains unchanged based on the specific scoring rules.

[0014] Optionally, after evaluating the performance of the target large language model based on the performance score, the method further includes: Output an analysis report of the performance evaluation of the target large language model, including an evaluation overview, evaluation data, problems and suggestions; The evaluation overview includes at least one of the following: evaluation purpose, evaluation scope, evaluation time, and intelligent referee model participating in the evaluation. The evaluation data includes the quality scores and corresponding analyses of the target large language model under different question banks and / or different scoring rules for category matching, and the analysis includes: the potential performance risks and core advantages of the target large language model; The issues and suggestions include problems identified in the target large language model based on the evaluation data, and corresponding optimization suggestions.

[0015] In a second aspect, a performance evaluation device for a large language model is provided, including a processor and a memory, wherein the memory stores a program or instructions that can run on the processor, and the program or instructions, when executed by the processor, implement the steps of the method described in the first aspect.

[0016] Thirdly, a readable storage medium is provided that stores a program or instructions which, when executed by a processor, implement the steps of the method described in the first aspect.

[0017] In this embodiment of the application, by retrieving the question bank corresponding to the category of the multi-category question bank, and using the questions and prompts stored in the question bank as test samples, the target large language model outputs the corresponding response content for each question. The quality of the response content corresponding to each question is scored by the intelligent judge model in combination with the scoring rules corresponding to the multi-dimensional quality parameters, so as to obtain the performance score of the target large language model and use it to evaluate the performance of the target large language model.

[0018] Therefore, the automated evaluation process significantly shortens the evaluation cycle, greatly improving the work efficiency of operations personnel. This allows for faster response to market changes in large language models and business decisions, significantly enhancing evaluation efficiency. By introducing intelligent referee models and multi-dimensional quality parameter scoring rules, the subjectivity of manual evaluation is reduced. The dual-referee scoring system further improves the reliability of model performance evaluation results, ensuring that the evaluation results are more objective, fair, and accurate. By constructing multi-category question banks and differentiated prompt word strategies, large language models can be comprehensively evaluated across multiple dimensions, avoiding evaluation blind spots. Attached Figure Description

[0019] The accompanying drawings, which are included to provide a further understanding of this application and form part of this application, illustrate exemplary embodiments and are used to explain this application, but do not constitute an undue limitation of this application. In the drawings: Figure 1 This is a flowchart illustrating the performance evaluation method for a large language model according to an embodiment of this application.

[0020] Figure 2 This is an example of a performance evaluation and analysis report of a large language model according to an embodiment of this application.

[0021] Figure 3 This is a structural block diagram of a performance evaluation device for a large language model according to an embodiment of this application. Detailed Implementation

[0022] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application. The drawing numbers in this application are only used to distinguish the various steps in the solution and are not used to limit the execution order of the various steps. The specific execution order is subject to the description in the specification.

[0023] To address the problems existing in the prior art, embodiments of this application provide a performance evaluation method for large language models, such as... Figure 1 As shown, it includes the following steps 102 to 108.

[0024] Step 102: In response to the evaluation request of the target large language model, query the target category question bank from the multi-category question bank, which includes multiple question banks for storing questions and prompt words of different categories.

[0025] The performance evaluation method for large language models in this application can be applied to a large language model performance evaluation system or platform. Users can input an evaluation request for the target large language model into the evaluation system, which will then respond to the evaluation request and trigger the evaluation of the target large language model based on the request. The system queries a multi-category question bank for the target type, which includes multiple question banks storing questions and prompts for different categories.

[0026] The multi-category question bank covers categories such as knowledge-based Q&A, ethics and safety, professional fields, and personal / corporate information queries, including various types of questions such as math, history, ethics, medical, safety-sensitive, soft political, and public figure questions.

[0027] Using the current evaluation category of the target large language model as the query condition, a multi-category question bank is queried to obtain the target question bank that matches the category. The questions and prompt words stored in the target question bank are used to evaluate the target large language model.

[0028] In one embodiment, before querying the target category question bank from the multi-category question bank, the method further includes: creating different categories of question banks according to assessment requirements, the categories including knowledge question and answer, ethics and safety, professional field, and person / company information query; obtaining questions of different categories through manual input, batch import, and / or web crawling to expand the corresponding question bank, the metadata of each question including question content, question category, and difficulty level; configuring exclusive prompt words for questions in each category of question bank; and constructing the multi-category question bank based on the question banks of different categories that include questions and corresponding prompt words.

[0029] This implementation involves establishing and managing a large-scale, multi-category assessment question bank. Operations personnel or content experts create and import multi-category question banks into the assessment system based on assessment needs. The question bank content covers the following categories: Knowledge-based question-and-answer categories, including math, history, geography, and science questions, are used to assess the breadth and accuracy of the large language model's knowledge. Ethics and safety categories, covering sensitive topics, discriminatory remarks, and misinformation, are used to assess the large language model's value alignment and safety compliance. Professional domain categories, including medical, legal, and financial categories, are used to assess the depth and professionalism of the large language model in specific professional fields. Person and company lookup categories, targeting information queries about individuals and companies, are used to assess the large language model's information integration and fact-checking capabilities.

[0030] The assessment system supports continuous expansion and updates of question banks across various categories, aiming to accumulate high-quality question set data assets, enhance the value of data assets, and ensure the comprehensiveness of assessments. For example, the assessment system supports continuous expansion of the question bank through various methods such as manual input, batch import, and web crawling. Each question includes metadata such as question content (optionally, it may also include the expected answer for partial automated comparison), question category, and difficulty level.

[0031] All question bank data is stored in a structured manner and tagged with a unique identifier to form high-quality question set data assets, which facilitate subsequent reuse, retrieval and analysis, and realize the assetization of question set data.

[0032] Different system prompt word configuration strategies are provided for different types of question banks. Users can set dedicated system prompt words for each question bank or evaluation task to guide the target large language model's comprehension ability and output application content quality in specific scenarios, and ensure the fairness and effectiveness of the evaluation.

[0033] Specifically, for different categories of question banks, it is possible to configure exclusive system prompts for each category or specific assessment task. For example, for knowledge-based question-and-answer questions, the prompts might emphasize "Please provide accurate and concise answers, and indicate the source of the information"; for ethics and safety questions, the prompts might emphasize "Please avoid any harmful, discriminatory, or inappropriate content"; for people and companies search questions, the prompts might emphasize "Please provide as much detailed information as possible about the target person or company, including the following dimensions: basic information, educational background, work experience, social contributions and evaluations, awards and honors," and so on.

[0034] Furthermore, the evaluation system supports version management of prompt words, recording the effects of different versions for easy iterative optimization. Operations personnel can adjust the prompt words based on the evaluation results to improve the understanding ability and output response quality of the large language model under evaluation in various scenarios.

[0035] Step 104: Input the questions and prompts from the target category question bank into the target large language model to obtain the corresponding response content for each question.

[0036] In response to the received evaluation request, an evaluation task is created to evaluate the performance of the target large language model in the target category. The question bank and prompt words corresponding to the target large language model are bound to the target large language model to be evaluated to ensure the relevance and fairness of the evaluation.

[0037] Each question in the target category question bank and its associated prompts are input as test samples into the target large language model to obtain the corresponding response results, i.e., the response content output by the target large language model for each question. The response content can be used as a label, thus forming a test sample-label test sample pair with the corresponding test sample.

[0038] Step 106: The quality of the response content for each question is scored by combining the intelligent referee model with the scoring rules corresponding to the multi-dimensional quality parameters, so as to obtain the performance score of the target large language model.

[0039] In this embodiment, the test sample-label test sample pairs of the target large language model obtained in step 104 are provided to the intelligent judging model. The intelligent judging model combines the scoring rules corresponding to the multi-dimensional quality parameters to score the quality of the response content for each question. The multi-dimensional quality parameters are indicators that reflect the quality of the response content from multiple aspects, reducing the subjectivity of manual evaluation and ensuring that the model performance evaluation results are more objective, fair, and accurate. Based on the quality score of the response content, the performance score of the target large language model can be evaluated. The higher the performance score, the better the performance of the target large language model.

[0040] The intelligent judging model can be a mainstream large language model. The evaluation system connects to the application programming interface (API) of mainstream and high-performing large language models from both domestic and international sources through an external integration layer. These models are configured as "intelligent judges" and are responsible for judging and scoring the quality of the responses from other large language models to be evaluated.

[0041] In one embodiment, an intelligent judging model, combined with scoring rules corresponding to multi-dimensional quality parameters, is used to score the quality of the responses to each question, thereby obtaining the performance score of the target large language model, including: The target large language model provides each question, corresponding prompts, and response content to the target intelligent judge model. Each question is input into the target intelligent judge model to obtain the corresponding response content. The response content of the target large language model to each question is compared with the response content of the target intelligent judge model to each question using multi-dimensional quality parameters to verify the quality of the response content of the target large language model. These multi-dimensional quality parameters include at least two of the following: authenticity, verifiability, redundancy, qualified security, and comprehensiveness. The verification results of each dimension of quality parameters are quantitatively scored based on the scoring rules to obtain the quality score of each response content of the target large language model. The quality scores of all responses from the target large language model are summarized to obtain the performance score of the target large language model.

[0042] In this embodiment, after the target large language model provides responses to each question in the target question bank, the evaluation system sends the question, the target large language model's response, and preset system prompts to the intelligent judging model. The intelligent judging model, based on its powerful understanding and reasoning capabilities, performs standardized verification of the quality of the target large language model's response. Quality verification is conducted from multiple dimensions of quality parameters, including at least two of the following: authenticity, verifiability, redundancy, adequate security, and comprehensiveness. The verification results are further combined with corresponding scoring rules to quantify the quality of each response into a corresponding quality score.

[0043] Each test sample from the target large language model is input into the intelligent judging model to obtain the corresponding response result, i.e., the intelligent judging model's response to the corresponding question for the test sample. Combining multi-dimensional quality parameters, the response content of the test sample in the target large language model is aligned and scored with the response content of the test sample in the intelligent judging model. The differences between the two responses under each dimension of quality parameters are determined, and thus the quality of the response content of the target large language model is scored based on the response content of the intelligent judging model.

[0044] After obtaining the quality scores for each dimension of the response content of the target large language model, all quality scores are further summarized to obtain the performance score of the target large language model. For example, the intelligent judge model outputs a score from 0 to 100, representing the performance score of the target large language model.

[0045] Optionally, the evaluation system supports configuring multiple intelligent referee models for collaborative evaluation. When the scores output by two intelligent referee models differ by more than a preset threshold (e.g., 20 points), the evaluation system can automatically initiate a manual review mechanism to further improve the accuracy and reliability of the evaluation results. If the score difference is within the preset threshold, the average score or weighted average score is taken as the final score.

[0046] In one embodiment, the scoring rules include general scoring rules applicable to all categories of large language models, and the multi-dimensional quality parameters include the authenticity, verifiability, redundancy, and qualified security of the response content; the verification results of each dimension of quality parameters are quantitatively scored based on the scoring rules to obtain the quality score of each response content of the target large language model, including: determining the verification results of authenticity, verifiability, and redundancy corresponding to each response content of the target large language model; and quantitatively scoring the verification results of each response content based on the general scoring rules to obtain the quality score of each response content.

[0047] In this embodiment, the scoring rule is a general scoring rule applicable to the performance evaluation of large language models across all categories. The general scoring rule focuses on multi-dimensional quality parameters including: the authenticity, verifiability, redundancy, and adequate security of the response content.

[0048] Specifically, the verification results of each response are quantitatively scored based on the aforementioned general scoring rules, including: If the verification result for the authenticity of the target response content is negative, then a first score is deducted from the target quality score of the target response content based on the general scoring rules; if the verification result for verifiability of the target response content is negative, then a second score is deducted from the target quality score based on the general scoring rules; if the verification result for redundancy of the target response content is insufficient in terms of professionalism or logical rigor, then a third score is deducted from the target quality score based on the general scoring rules; if the verification result for security of the target response content is negative, then a fourth score is deducted from the target quality score of the target response content based on the general scoring rules; the score after deducting the first, second, third, and fourth scores from the target quality score is determined as the quality score of the target response content.

[0049] In this embodiment, the verification result of the authenticity of the response content includes whether the response content contains fabricated facts ("illusion" attack). The intelligent adjudication model verifies whether there are fictitious or fabricated facts or information in the target response content of the target large language model. Once found, that is, if the verification result of the authenticity of the target response content is negative, the intelligent adjudication model will impose a severe deduction, such as deducting the first point, or even a veto. If there are minor factual flaws in the verification, a moderate deduction will be imposed.

[0050] The intelligent judging model can pre-set a maximum score for the quality parameters of each response content of the target large language model in each dimension, and deduct points based on the verification results of the quality parameters under the corresponding dimension, based on the maximum score.

[0051] The verification results for the authenticity of the response content include whether the response content appears reasonable but cannot be verified (“vague expression” penalty). The intelligent adjudication model verifies whether the target response content of the target large language model contains vague, unverifiable inferences or statements. Although such response content may seem reasonable on the surface, it lacks factual basis. If the verification result for the verifiability of the target response content is negative, the intelligent adjudication model will deduct a moderate score, such as deducting the second point. The moderate deduction from the highest score is lower than the high deduction from the highest score, meaning the second point is lower than the first point.

[0052] The verification results for the redundancy of the response content include whether the response content gives conclusions that exceed the scope of the question ("response focus" assessment). The intelligent judging model verifies whether the target response content of the target large language model is closely related to the question, avoiding meaningless overflow information. Redundancy of the response content that exceeds the scope of the question can be considered from the perspective of professionalism of expression and / or logical rigor.

[0053] For example, if the intelligent judging model verifies that the target response lacks logical rigor in terms of redundancy, a moderate deduction will be made. If the intelligent judging model verifies that the target response lacks professionalism in terms of redundancy, the content is somewhat perfunctory or the wording is unprofessional, resulting in insufficient professional expression, a minor deduction will be made. The minor deduction will be lower than the moderate deduction.

[0054] The intelligent judging model determines whether the response content is safe and secure. If the verification result for the response content is negative, the intelligent judging model will deduct a severe or moderate penalty from the highest score.

[0055] By combining the scoring rules and subtracting the highest score from the target response content based on multiple quality parameters, the score remaining after subtracting the highest score is determined as the quality score of the target response content. The performance score of the target large language model is obtained by summing the quality scores of all responses from the target large language model.

[0056] In one embodiment, the scoring rules further include specific scoring rules for a large language model applicable to the target category, and the multi-dimensional quality parameters further include the comprehensiveness of the response content; The verification results of each dimension of quality parameters are quantitatively scored based on the scoring rules to obtain the quality score of each response content of the target large language model. The method also includes: determining whether the target category is a person information query or a company information query; if the target category is a person information query or a company information query, the quality score obtained by quantitatively scoring each response content based on the verification results of the comprehensiveness of each response content corresponding to the specific scoring rules is adjusted.

[0057] In this embodiment, the scoring rules also include specialized scoring rules applicable to the performance evaluation of large language models in specific categories. More detailed and stringent scoring standards are developed for specific application scenarios. The specialized scoring rules focus on multi-dimensional quality parameters, including the comprehensiveness of the response content, typically for large language models categorized as person information queries or company information queries.

[0058] The core of the specific scoring rules for queries about personal or corporate information is "dimensionality first," which aims to prevent large language models from providing overly simplistic answers and requires comprehensive information in the response.

[0059] Given that the target large language model is classified as either a person information query or a company information query, an intelligent judging model combined with specific scoring rules is used to verify the comprehensiveness of the quality parameters corresponding to each response content of the target large language model. Based on the verification results, the quantitative scores of each response content under the general scoring rules are adjusted.

[0060] Specifically, based on the verification results of the comprehensiveness of each response content according to the specific scoring rules, the quality score obtained by quantitatively scoring each response content according to the general scoring rules is adjusted, including: Obtain the number of preset first dimensions corresponding to the target category, where the number of first dimensions represents the number of dimensions corresponding to the information included in the response content of the target category when the comprehensiveness is at its highest; determine the number of second dimensions corresponding to the information included in the target response content; If the verification result of the comprehensiveness of the target response content is that the number of the second dimension is lower than the number of the first dimension, and the difference between the number of the first dimension and the number of the second dimension is a first value, then based on the special scoring rule, the fifth score is deducted from the target quality score of the quantitative scoring of the target response content by the general scoring rule, so as to adjust the quality score obtained by the quantitative scoring of the target response content by the general scoring rule. If the verification result of the comprehensiveness of the target response content is that the number of the second dimension is lower than the number of the first dimension, and the difference between the number of the first dimension and the number of the second dimension is a second value, then a sixth score is deducted from the target quality score based on the specific scoring rule to adjust the quality score obtained by the quantitative scoring of the target response content by the general scoring rule, wherein the second value is greater than the first value, and the sixth score is greater than the fifth score; If the comprehensiveness verification result corresponding to the target response content is that the number of the second dimension is equal to the number of the first dimension, then the target quality score remains unchanged based on the specific scoring rules.

[0061] In this embodiment, the dimensions used to examine the comprehensiveness of personal information query or enterprise information query include, for example, five dimensions: basic information (place of origin, date of birth, etc.), educational background (graduating institution, major, etc.), work experience (positions held, social titles, etc.), social contributions and evaluations (social influence, industry evaluation, etc.), and awards and honors (awards received, certificates, etc.).

[0062] The comprehensiveness is highest when the target response includes five dimensions (the preset number of first dimensions). The comprehensiveness decreases when the target response includes fewer than five dimensions (the number of second dimensions). A smaller number of second dimensions results in more deductions. Correspondingly, the greater the difference between the number of second and first dimensions in the target response, the more points are deducted from the quantitative scoring of the target response using the general scoring rules.

[0063] For example, under the premise that the quality score of the target response content is ≥90 points according to the general scoring rules, the special scoring rules quantify the deduction standards corresponding to different numbers of second dimensions.

[0064] If the target response only provides basic information or a single dimension (e.g., just a simple resume), even if the content is accurate, the quality score after deduction will be below 60 points (unacceptable).

[0065] If the target response contains information from 2-3 dimensions, the quality score after deduction is, for example, 60-75 points (passing).

[0066] If the target response contains information from all four dimensions, the quality score after deduction is, for example, 75-90 points (good).

[0067] If the target response contains information from all five dimensions, the original quality score for the target response will be maintained according to the general scoring rules, for example, ≥ 90 points (excellent).

[0068] Therefore, when evaluating the performance of large language models for querying personal information or enterprise information, the intelligent referee model adjusts the quality scores of each response content quantified by the general scoring rules in conjunction with specific scoring rules, so that the adjusted quality scores of each response content more accurately reflect the quality of the response content in the corresponding category.

[0069] It can create evaluation tasks for target large language models on multiple different categories of question banks, obtain performance scores of target large language models under different categories, and then summarize the performance scores of target large language models under all categories to obtain the final performance score.

[0070] Step 108: Evaluate the performance of the target large language model based on the performance score.

[0071] After obtaining the performance score of the target large language model through the above steps, the performance of the model can be evaluated based on the score, thereby automating the performance evaluation of large language models and improving the stability and reliability of the evaluation system.

[0072] In one embodiment, after evaluating the performance of the target large language model based on the performance score, the method further includes: outputting an analysis report of the performance evaluation of the target large language model, the analysis report including an evaluation overview, evaluation data, problems and suggestions; The evaluation overview includes at least one of the following: evaluation purpose, evaluation scope, evaluation time, and participating intelligent referee models; the evaluation data includes the quality scores and corresponding analyses of the target large language model under different question banks and / or different scoring rules for category matching, the analysis including: the potential performance risks and core advantages of the target large language model; the problems and suggestions include the problems of the target large language model determined based on the evaluation data and corresponding optimization suggestions.

[0073] In this embodiment, after the evaluation of the target large language model is completed, the evaluation system automatically obtains the corresponding performance score, performs data analysis of the performance evaluation, and intelligently generates and publishes the analysis report of the performance evaluation of the target large language model.

[0074] The analysis report can automatically generate a complete, data-driven, and clearly defined evaluation report based on a preset template. The report content includes: an evaluation overview, such as the evaluation purpose, scope, timeframe, and participating intelligent judging models; an overview of the evaluation results, such as performance score rankings and key indicator comparisons from multiple intelligent judging models; detailed evaluation data, such as detailed scores and corresponding analyses of the target large language model's dimensional quality parameters under different question bank categories and scoring rules for category matching, highlighting the potential performance risks and core strengths of the target large language model; and problems and suggestions, such as identifying existing problems in the target large language model based on the evaluation data and providing specific optimization suggestions.

[0075] The detailed evaluation data analysis includes, for example, the average score, highest score, and lowest score of the target language model under different question bank categories; a comparison of the target language model's performance under general scoring rules and specific scoring rules; and statistics on indicators such as the "illusion" rate, vague expression rate, and answer focus of the target language model's response content.

[0076] In addition, the analysis report may include a competitive comparison analysis, highlighting the advantages of the target large language model or identifying gaps by comparing its performance with competing models. It may also provide conclusions and decision-making recommendations, offering objective evidence for the selection, switching, or optimization of the target large language model based on the evaluation results.

[0077] The analysis report can embed visualization elements such as charts and tables to intuitively display evaluation data and trends. For example, it can display a comparison chart of the accuracy rates of different large language models in the evaluation of translation capabilities for less commonly spoken languages.

[0078] After the analysis report is generated, the evaluation system can automatically publish the report to the company's official WeChat account, internal knowledge base, or decision support system according to the configuration, thereby expanding brand influence and enhancing external communication capabilities.

[0079] In the automated process, the evaluation system monitors for situations such as failed calls to the intelligent referee model and abnormal quantitative scoring. When preset abnormal conditions occur, the evaluation system will issue an alarm and can trigger manual intervention according to the configuration, such as initiating manual review when the performance scores of the two intelligent referee models differ too much.

[0080] Figure 2 This document presents an example of a performance evaluation and analysis report for a large language model according to an embodiment of this application. For instance, it includes core capability profiles based on multi-dimensional quality parameters such as factual accuracy, professional expression, logical rigor, qualification and security, and content comprehensiveness. The quantitative scores corresponding to each quality parameter are presented as polygons. The report also provides an analysis of the potential performance risks and core advantages of the target large language model, along with a final performance score of "81.61".

[0081] The evaluation system for the performance evaluation method of the large language model according to the embodiments of this application can adopt a modular design, be deployed in the cloud or an enterprise's own data center, and can be flexibly expanded according to actual needs. The core architecture of the evaluation system aims to achieve efficient, automated, and objective model evaluation, and its main components include: The data layer is responsible for storing and managing all the data required for evaluating large language models, including multi-category question banks, evaluation task configurations, response results of the large language model, scoring data from intelligent judging models, and historical evaluation analysis reports. The data layer can employ a combination of distributed storage systems (such as HDFS and S3) and relational databases (such as MySQL) to support large-scale data storage and efficient querying.

[0082] The service layer comprises several core functional modules that provide services to the outside world through API interfaces, automating the evaluation process of large language models. These core functional modules include question bank management services, prompt word management services, intelligent referee model invocation services, quantitative scoring services, evaluation rule engine services, and analysis report generation services.

[0083] The scheduling and control layer is responsible for scheduling, execution monitoring, and process control of evaluation tasks. This layer can utilize task scheduling frameworks (such as Airflow and Kubernetes CronJobs) to implement timed triggering, dependency management, and exception handling, ensuring the stable operation of evaluation tasks.

[0084] The user interface layer provides an intuitive interface for operations personnel to configure assessment tasks, manage question banks, and view assessment results and analysis reports. The interface may include a task dashboard, a question bank editor, a model configuration page, and a report viewer.

[0085] The external integration layer is responsible for interacting with external systems, including: large model API interfaces, used to call the APIs of mainstream large models at home and abroad as intelligent judge models, and to call the large language models to be evaluated; operation and maintenance deployment tools, used for the deployment, monitoring and management of various models; and knowledge retrieval tools, used to assist intelligent judge models in information retrieval and fact verification.

[0086] In this embodiment of the application, by retrieving the question bank corresponding to the category of the multi-category question bank, and using the questions and corresponding prompt words stored in the question bank as test samples, the target large language model outputs the corresponding response content for each question. The quality of the response content corresponding to each question is scored by the intelligent judge model in combination with the scoring rules corresponding to the multi-dimensional quality parameters, so as to obtain the performance score of the target large language model and use it to evaluate the performance of the target large language model.

[0087] Therefore, the automated evaluation process significantly shortens the evaluation cycle, greatly improving the work efficiency of operations personnel. This allows for faster response to market changes in large language models and business decisions, significantly enhancing evaluation efficiency. By introducing intelligent referee models and multi-dimensional quality parameter scoring rules, the subjectivity of manual evaluation is reduced. The dual-referee scoring system further improves the reliability of model performance evaluation results, ensuring that the evaluation results are more objective, fair, and accurate. By constructing multi-category question banks and differentiated prompt word strategies, large language models can be comprehensively evaluated across multiple dimensions, avoiding evaluation blind spots.

[0088] Furthermore, the automation of the entire process from task scheduling of evaluation requests to generation of evaluation analysis reports reduces manual operation costs and error rates, and improves the stability and reliability of the evaluation system.

[0089] Optionally, such as Figure 3 As shown in the figure, this application embodiment also provides a large language model performance evaluation device 2000, including a processor 2400 and a memory 2200. The memory 2200 stores a program or instructions that can be run on the processor 2400. When the program or instructions are executed by the processor 2400, they implement the various steps of the above-described large language model performance evaluation method embodiment and can achieve the same technical effect. To avoid repetition, they will not be described again here.

[0090] This application also provides a readable storage medium storing a program or instructions. When executed by a processor, the program or instructions implement the various processes of any of the above-described performance evaluation methods for large language models, achieving the same technical effect. To avoid repetition, these will not be described again here. The readable storage medium includes computer-readable storage media, such as read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0091] This application also provides a computer program product, which includes a non-transitory computer-readable storage medium storing a computer program. The computer program is operable to enable a computer to execute various processes of any of the above-described large language model performance evaluation method embodiments and achieve the same technical effect. To avoid repetition, it will not be described again here.

[0092] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.

[0093] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes several instructions to cause a terminal (which may be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods described in the various embodiments of this application.

[0094] The embodiments of this application have been described above with reference to the accompanying drawings. However, this application is not limited to the specific embodiments described above. The specific embodiments described above are merely illustrative and not restrictive. Those skilled in the art can make many other forms under the guidance of this application without departing from the spirit and scope of the claims, and all of these forms are within the protection scope of this application.

Claims

1. A performance evaluation method for a large language model, characterized in that, include: In response to the evaluation request of the target large language model, the target category question bank is queried from the multi-category question bank, which includes multiple question banks for storing questions and prompt words of different categories; Input the questions and prompts from the target category question bank into the target large language model to obtain the corresponding response content for each question; The quality of the responses to each question is scored by combining an intelligent judging model with scoring rules corresponding to multi-dimensional quality parameters, so as to obtain the performance score of the target large language model. The performance of the target large language model is evaluated based on the performance score.

2. The method as described in claim 1, characterized in that, Before querying the target category question bank from multiple category question banks, the following steps are also included: Based on the assessment requirements, different categories of question banks are created, including knowledge-based question and answer, ethics and safety, professional fields, personal information query, and corporate information query. The question bank is expanded by obtaining questions of different categories through manual input, batch import and / or web crawling. The metadata of each question includes the question content, question category and difficulty level. Configure unique prompts for questions in each category of the question bank; The multi-category question bank is composed of question banks that include questions and corresponding prompts based on different categories.

3. The method as described in claim 1, characterized in that, The quality of the responses to each question is scored using an intelligent judging model combined with scoring rules corresponding to multi-dimensional quality parameters, in order to obtain the performance score of the target large language model, including: The target large language model provides each question, corresponding prompt words, and response content to the target intelligent judge model; Each question is input into the target intelligent judge model to obtain the corresponding response content for each question; The quality of the target large language model's response to each question is verified by comparing the response content of the target intelligent judge model to each question with the response content of the target large language model using multi-dimensional quality parameters. The multi-dimensional quality parameters include at least two of the following: authenticity, verifiability, redundancy, qualified security, and comprehensiveness. Based on the scoring rules, the verification results of each dimension of quality parameters are quantitatively scored to obtain the quality score of each response content of the target large language model. The quality scores of all responses from the target large language model are summarized to obtain the performance score of the target large language model.

4. The method as described in claim 3, characterized in that, The scoring rules include general scoring rules applicable to all categories of large language models, and the multi-dimensional quality parameters include the authenticity, verifiability, redundancy, and qualified security of the response content. Based on the scoring rules, the verification results of each dimension of quality parameters are quantitatively scored to obtain the quality score of each response content of the target large language model, including: The verification results for the authenticity, verifiability, and redundancy of each response content of the target large language model are determined respectively; The verification results of each response are quantitatively scored based on the general scoring rules to obtain a quality score for each response.

5. The method as described in claim 4, characterized in that, The verification results of each response are quantitatively scored based on the aforementioned general scoring rules, including: If the verification result of the authenticity of the target response content is negative, then the first score is deducted from the target quality score of the target response content based on the general scoring rules. If the verification result of the verifiability of the target response content is negative, then a second score is deducted from the target quality score based on the general scoring rules. If the verification result of the redundancy of the target response content is insufficient in terms of professionalism or logical rigor, then a third score will be deducted from the target quality score based on the general scoring rules. If the verification result of the target response content corresponding to qualified security is negative, then a fourth score is deducted from the target quality score of the target response content based on the general scoring rules. The score obtained by subtracting the first score, the second score, the third score, and the fourth score from the target quality score is determined as the quality score of the target response content.

6. The method as described in claim 4, characterized in that, The scoring rules also include specific scoring rules for large language models applicable to the target category, and the multi-dimensional quality parameters also include the comprehensiveness of the response content; The verification results of each dimension's quality parameters are quantitatively scored based on the aforementioned scoring rules to obtain the quality score of each response content of the target large language model, and the method further includes: Determine whether the target category is a person information query or a company information query; If the target category is a person information query or a company information query, then based on the verification results of the comprehensiveness of each response content according to the specific scoring rules, the quality score obtained by quantifying and scoring each response content according to the general scoring rules is adjusted.

7. The method as described in claim 6, characterized in that, Based on the verification results of the comprehensiveness of each response content according to the specific scoring rules, the quality score obtained by quantitatively scoring each response content according to the general scoring rules is adjusted, including: Obtain the number of preset first dimensions corresponding to the target category, where the number of first dimensions represents the number of dimensions corresponding to the information included in the response content of the target category when the comprehensiveness is at its highest. Determine the number of second-dimensional elements corresponding to the information included in the target response; If the verification result of the comprehensiveness of the target response content is that the number of the second dimension is lower than the number of the first dimension, and the difference between the number of the first dimension and the number of the second dimension is a first value, then based on the special scoring rule, the fifth score is deducted from the target quality score of the quantitative scoring of the target response content by the general scoring rule, so as to adjust the quality score obtained by the quantitative scoring of the target response content by the general scoring rule. If the verification result of the comprehensiveness of the target response content is that the number of the second dimension is lower than the number of the first dimension, and the difference between the number of the first dimension and the number of the second dimension is a second value, then a sixth score is deducted from the target quality score based on the specific scoring rule to adjust the quality score obtained by the quantitative scoring of the target response content by the general scoring rule, wherein the second value is greater than the first value, and the sixth score is greater than the fifth score; If the comprehensiveness verification result corresponding to the target response content is that the number of the second dimension is equal to the number of the first dimension, then the target quality score remains unchanged based on the specific scoring rules.

8. The method as described in claim 6, characterized in that, After evaluating the performance of the target large language model based on the performance score, the following steps are also included: Output an analysis report of the performance evaluation of the target large language model, including an evaluation overview, evaluation data, problems and suggestions; The evaluation overview includes at least one of the following: evaluation purpose, evaluation scope, evaluation time, and intelligent referee model participating in the evaluation. The evaluation data includes the quality scores and corresponding analyses of the target large language model under different question banks and / or different scoring rules for category matching, and the analysis includes: the potential performance risks and core advantages of the target large language model; The issues and suggestions include problems identified in the target large language model based on the evaluation data, and corresponding optimization suggestions.

9. A performance evaluation device for a large language model, characterized in that, It includes a processor and a memory, the memory storing a program or instructions that can run on the processor, the program or instructions being executed by the processor to implement the steps of the method as described in any one of claims 1-8.

10. A readable storage medium, characterized in that, The readable storage medium stores a program or instructions that, when executed by a processor, implement the steps of the method as described in any one of claims 1-8.

Images