Large language model evaluation method, device, equipment and storage medium

Abstract

This application discloses a method, apparatus, device, and storage medium for evaluating large language models, relating to the field of artificial intelligence technology. The method includes: acquiring task configuration information and candidate evaluation model information; determining target evaluation model information based on stability evaluation index information and alignment evaluation index information of the candidate evaluation model information; injecting corresponding semantic evaluation elements into the model prompt word template of the target evaluation model information based on the task configuration information to determine semantic enhancement information of the target evaluation model; and generating corresponding evaluation results based on the semantic enhancement information of the target evaluation model. This application dynamically determines the optimal evaluation model through stability evaluation index and alignment evaluation index, and injects semantic evaluation elements of the business scenario into the corresponding model prompt word template to obtain evaluation results. It coordinates model selection and prompt enhancement, eliminates the semantic gap between general templates and business requirements, and improves the model evaluation's discriminative power, semantic consistency, and reproducibility.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and in particular to methods, apparatus, devices and storage media for evaluating large language models. Background Technology

[0002] As large language models (LLMs) are increasingly used in complex application scenarios such as intelligent agents, automated evaluation methods that use LLM as a judge are receiving more and more attention for the reliability of their evaluation results and their consistency with human preferences. The quality of the evaluation depends not only on the discrimination ability of the evaluation model itself, but also on the accurate transmission of task semantics in the evaluation prompts.

[0003] Currently, existing practices involve using generic evaluation prompt templates to maintain a consistent evaluation standard and semantic framework across different tasks, or allowing users to create highly customized evaluation templates based on specific business scenarios. However, while generic templates ensure consistency across tasks, they lack evaluation dimensions and focus points specific to particular scenarios and are not semantically adapted to specific tasks. This results in evaluation results that fail to accurately reflect the quality requirements of specific scenarios. Custom templates, while introducing scenario-specific criteria, easily lead to fragmented evaluation standards, making it difficult to compare and reproduce evaluation results across different tasks. This inherent contradiction between uniformity and adaptability makes it difficult for evaluation systems to maintain consistency while flexibly covering diverse business needs. Therefore, how to accurately and effectively evaluate large language models has become an urgent problem to be solved. Summary of the Invention

[0004] The main purpose of this application is to provide a method, apparatus, device and storage medium for evaluating large language models, aiming to solve the technical problem of how to accurately and effectively evaluate large language models.

[0005] To achieve the above objectives, this application proposes a method for evaluating large language models, the method comprising: Obtain task configuration information and candidate evaluation model information; Based on the stability evaluation index information and alignment evaluation index information of the candidate evaluation model information, the target evaluation model information is determined. Based on the task configuration information, the corresponding semantic evaluation elements are injected into the model prompt word template of the target evaluation model information to determine the semantic enhancement information of the target evaluation model; The corresponding evaluation results are generated based on the semantic enhancement information of the target evaluation model.

[0006] In one embodiment, the step of determining the target evaluation model information based on the stable evaluation index information and the aligned evaluation index information of the candidate evaluation model information includes: Obtain the evaluation sample set; Based on the evaluation sample set, determine the stability evaluation index information and alignment evaluation index information of the candidate evaluation model information; Based on the stability evaluation index information and the alignment evaluation index information, the candidate evaluation model information is screened to obtain the target evaluation model information.

[0007] In one embodiment, the step of determining the stable evaluation index information and the aligned evaluation index information of the candidate evaluation model information based on the evaluation sample set includes: Obtain annotation preference data; The candidate evaluation model information is independently scored multiple times based on paired samples in the evaluation sample set to obtain the scoring results; Based on the scoring results and the labeled preference data, the stability evaluation index information and alignment evaluation index information of the candidate evaluation model information are determined. The stability evaluation index information includes stability variance index information, and the alignment evaluation index information includes pairwise accuracy index information, tie rate index information, and sensitivity index information.

[0008] In one embodiment, the step of filtering the candidate evaluation model information based on the stability evaluation index information and the alignment evaluation index information to obtain the target evaluation model information includes: The stability evaluation index and the alignment evaluation index are weighted and fused based on preset weights to determine the comprehensive score information; Based on the comprehensive scoring information, the candidate evaluation model information is filtered to obtain the target evaluation model information.

[0009] In one embodiment, the step of injecting the corresponding semantic evaluation elements into the model prompt word template of the target evaluation model information based on the task configuration information, and determining the semantic enhancement information of the target evaluation model, includes: Based on the task configuration information, the corresponding target scenario category information is determined; Based on the target scene category information, the corresponding semantic evaluation elements are retrieved from the preset scene configuration library to determine the semantic evaluation element information; The semantic evaluation element information is injected into the model prompt word template of the target evaluation model information to obtain the semantic enhancement information of the target evaluation model.

[0010] In one embodiment, the step of injecting the semantic evaluation element information into the model prompt word template of the target evaluation model information to obtain the semantic enhancement information of the target evaluation model includes: Acquire comparison mode information, which includes baseline mode, half-process mode, and full-process mode; The corresponding injection strategy information is determined based on the comparison pattern information. Based on the injection strategy information, the semantic evaluation element information is injected into the model prompt word template of the target evaluation model information to obtain the semantic enhancement information of the target evaluation model.

[0011] In one embodiment, the step of determining the corresponding injection strategy information based on the comparison pattern information includes: Obtain information during the assessment phase; When the comparison mode information is the baseline mode, the injection strategy information is that injection is not allowed; When the evaluation phase information is a partial reasoning phase and the comparison mode information is a half-process mode, the injection strategy information allows injection in the half-phase. When the evaluation phase information is full-stage and the comparison mode information is full-process mode, the injection strategy information allows injection to be attempted in all stages.

[0012] Furthermore, to achieve the above objectives, this application also proposes a large language model evaluation device, which includes: The acquisition module is used to acquire task configuration information and candidate evaluation model information; The processing module is used to determine the target evaluation model information based on the stable evaluation index information and the aligned evaluation index information of the candidate evaluation model information; The processing module is also used to inject the corresponding semantic evaluation elements into the model prompt word template of the target evaluation model information based on the task configuration information, and determine the semantic enhancement information of the target evaluation model; The execution module is used to generate corresponding evaluation results based on the semantic enhancement information of the target evaluation model.

[0013] In addition, to achieve the above objectives, this application also proposes a large language model evaluation device, the device comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, the computer program being configured to implement the steps of the large language model evaluation method as described above.

[0014] In addition, to achieve the above objectives, this application also proposes a storage medium, which is a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, it implements the steps of the large language model evaluation method described above.

[0015] One or more technical solutions proposed in this application have at least the following technical effects: This embodiment proposes a method for evaluating large language models, which involves acquiring task configuration information and candidate evaluation model information; determining target evaluation model information based on the stability evaluation index information and alignment evaluation index information of the candidate evaluation model information; injecting the corresponding semantic evaluation elements into the model prompt word template of the target evaluation model information based on the task configuration information to determine the semantic enhancement information of the target evaluation model; and generating corresponding evaluation results based on the semantic enhancement information of the target evaluation model. This application utilizes stable evaluation metrics and alignment evaluation metrics of candidate evaluation models to dynamically determine the optimal target evaluation model. During runtime, it identifies the current business scenario based on task configuration information and dynamically injects scenario-specific semantic evaluation elements into the target evaluation model's generic prompt word template, generating semantic enhancement information for the target evaluation model and thus obtaining the evaluation result. This organically coordinates the model selection and prompt enhancement stages. It eliminates the subjective arbitrariness of manually specifying the evaluation model through quantitative metrics, ensuring the stability of the model output and consistency with human preferences. It also bridges the semantic gap between the generic prompt template and specific business requirements, allowing the optimal model to execute highly adapted evaluation instructions, significantly improving the model's discriminative power, semantic consistency, and reproducibility. Attached Figure Description

[0016] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0017] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0018] Figure 1 This is a flowchart illustrating the first embodiment of the large language model evaluation method of this application. Figure 2 This is a schematic diagram showing the comparison results of model selection for the large language model evaluation method in this application; Figure 3 This is a schematic diagram of the fine-grained scene classification system for the large language model evaluation method in this application; Figure 4This is a schematic diagram illustrating the semantic evaluation element recognition method of the large language model evaluation method in this application; Figure 5 This is a schematic diagram illustrating the experimental verification results of the large language model evaluation method proposed in this application. Figure 6 This is a flowchart illustrating Embodiment 2 of the large language model evaluation method of this application; Figure 7 This is a flowchart illustrating the comparison of large language model evaluation methods in this application. Figure 8 This is a schematic diagram of the module structure of the large language model evaluation device according to an embodiment of this application; Figure 9 This is a schematic diagram of the hardware operating environment involved in the large language model evaluation method in this application embodiment.

[0019] The purpose, features, and advantages of this application will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0020] It should be understood that the specific embodiments described herein are merely illustrative of the technical solutions of this application and are not intended to limit this application.

[0021] To better understand the technical solution of this application, a detailed description will be provided below in conjunction with the accompanying drawings and specific implementation methods.

[0022] The main solution of this application embodiment is as follows: obtain task configuration information and candidate evaluation model information; determine target evaluation model information based on the stability evaluation index information and alignment evaluation index information of the candidate evaluation model information; inject the corresponding semantic evaluation elements into the model prompt word template of the target evaluation model information based on the task configuration information to determine the semantic enhancement information of the target evaluation model; and generate the corresponding evaluation result based on the semantic enhancement information of the target evaluation model.

[0023] In this embodiment, for ease of description, the following description will focus on a large language model evaluation device.

[0024] While existing general templates can ensure consistency across task evaluation frameworks, they lack evaluation dimensions and key considerations specific to particular scenarios and are semantically adapted to specific tasks. This results in evaluation results that fail to accurately reflect the quality requirements of specific scenarios. Custom templates, on the other hand, can introduce scenario-specific criteria, but they easily lead to fragmented evaluation standards, making it difficult to compare and reproduce evaluation results across different tasks. This inherent contradiction between uniformity and adaptability makes it difficult for evaluation systems to maintain consistency while flexibly covering diverse business needs.

[0025] This application provides a solution that dynamically determines the optimal target evaluation model from candidate models by weighted fusion calculation of stability evaluation indicators and alignment evaluation indicators, eliminating the subjective arbitrariness of manual selection. It introduces a scene classifier to identify the fine-grained scene to which the current task belongs, loads scene-specific semantic evaluation elements from a modular configuration library, dynamically injects them into a general prompt word template, generates semantic enhancement information for the target evaluation model, and bridges the semantic gap between the unified framework and business requirements. The optimal evaluation model obtained by automatic selection executes the scene-enhanced prompt instructions for evaluation. While ensuring the stability of output and consistency with human preferences, it significantly improves the discriminative power, interpretability, and reproducibility of evaluation results.

[0026] As can be seen from the above embodiments, this application can dynamically determine the optimal target evaluation model by using the stability evaluation index and alignment evaluation index of the candidate evaluation model in the evaluation. At runtime, it can identify the current business scenario based on the task configuration information and dynamically inject scenario-specific semantic evaluation elements into the general prompt word template of the target evaluation model to generate semantic enhancement information of the target evaluation model, thereby obtaining the evaluation result. The two stages of model selection and prompt enhancement are organically coordinated. It not only eliminates the subjective arbitrariness of manually specifying the evaluation model through quantitative indicators and ensures the stability of the model output and the consistency with human preferences, but also bridges the semantic gap between the general prompt template and specific business needs. The optimal model executes highly adapted evaluation instructions, which significantly improves the discriminative power, semantic consistency and reproducibility of the model evaluation.

[0027] Based on this, embodiments of this application provide a method for evaluating large language models, referring to... Figure 1 , Figure 1 This is a flowchart illustrating the first embodiment of the large language model evaluation method of this application.

[0028] In this embodiment, the large language model evaluation method includes steps S10 to S40: Step S10: Obtain task configuration information and candidate evaluation model information; It should be noted that the task configuration information is configuration data used to define the standards and elements required for evaluating tasks in specific business scenarios. It can be organized through modular configuration files, with each scenario corresponding to an independent YAML format file. The specific semantic evaluation elements that need to be considered in the evaluation of the corresponding scenario are clearly defined. For example, in the chat scenario configuration file extension chichat.yaml-extended corresponding to the "General Communication - Casual Chat" scenario, evaluation elements such as "vividness", "engagement" and "length" are clearly defined, and general basic configurations such as the basic writing configuration file basic_writing.yaml and the basic robot configuration file basic_bot.yaml are referenced to form a reusable configuration system.

[0029] It is understood that the candidate evaluation model information is a series of large language models that can be used to perform the evaluation task, along with the parameters required for their invocation. It can be a pre-configured list of models, such as models with different specifications like "Qwen3-4B", "Qwen2.5-7B", and "Qwen3-30B".

[0030] In a specific embodiment, the task configuration information can be a pre-configured, dynamically loadable modular configuration file. When adapting to new business scenarios, only the corresponding YAML configuration file needs to be added or modified, without altering the underlying evaluation engine's code logic, thus achieving flexible expansion of the evaluation criteria. The candidate evaluation model information can be obtained through pre-set selection experiments, and candidate models can be dynamically added or removed. For example, the candidate evaluation model information can include the Qwen3-4B model, the Qwen2.5-7B model, and the Qwen3-30B-Instruct model, resulting in a pool similar to a basic model pool, facilitating evaluation.

[0031] Step S20: Based on the stability evaluation index information and alignment evaluation index information of the candidate evaluation model information, determine the target evaluation model information; It should be noted that the target evaluation model information is the identifier and related calling parameters of the determined optimal evaluation model. Essentially, it is to automatically select the model that achieves the best balance between stable output and consistency with human judgment from multiple candidate models. This model is used to evaluate which large language model should be used to perform the task, thereby eliminating the subjective arbitrariness and unreproducible results caused by manually specifying the evaluation model.

[0032] In a specific embodiment, an evaluation sample set is obtained. This evaluation sample set can be a publicly available dataset with paired responses and human preference annotations, such as the HelpSteer2 dataset released by NVIDIA Research, which contains 10,162 prompts and their paired responses and provides human ratings in five dimensions (Helpfulness, Correctness, Coherence, Complexity, and Verbosity). Each metric uses a rating scale of 0 to 4 points, which is naturally suitable as the "gold standard" for judging the model's ability. To enhance the sample discriminability, sample pairs with a human rating difference of 3 or more on indicators such as Helpfulness, Correctness, and Coherence can be selected to construct a highly discriminative paired subset as the evaluation sample set.

[0033] Obtain annotation preference data; perform multiple rounds of independent scoring on the candidate evaluation model information based on paired samples in the evaluation sample set to obtain scoring results; determine the stability evaluation index and alignment evaluation index information of the candidate evaluation model information based on the scoring results and the annotation preference data. The stability evaluation index information includes stability variance index information, and the alignment evaluation index information includes paired accuracy index information, tie rate index information, and sensitivity index information, i.e., obtain the preference label of which of the two manually annotated responses is better. Based on each paired sample in the evaluation sample set, input the two responses simultaneously into the candidate evaluation model and use the same generation parameters (e.g., temperature=0 to eliminate random exogenous factors) to perform a preset number of independent scoring operations on each sample pair, such as five times, to obtain scoring results. Calculate the stability variance index based on the scoring results: for each sample pair, calculate the variance of the model's multiple scoring operations, and then summarize the variances of all sample pairs to obtain the overall stability variance. Alignment evaluation metrics are calculated based on scoring results and annotation preference data: Pairwise accuracy is the percentage of sample pairs whose model preference judgments align with human annotation preferences; tie rate is the percentage of sample pairs where the model determines two responses are indistinguishable; sensitivity, for difficult sample subsets where human annotations themselves are discrepancies, calculates the proportion of samples for which the model can provide non-tie judgments. For example, ... Figure 2 As shown, Figure 2 This diagram illustrates the comparison results of model selection for the large language model evaluation method in this application. Using Qwen3-4B, Qwen2.5-7B, and Qwen3-30B as candidate evaluation models, the experiment shows that Qwen3-30B has the lowest stability variance, the highest pairwise accuracy, and the highest sensitivity, while the tie rate is at a moderate level, making it the best overall performer.

[0034] The stability evaluation index and the alignment evaluation index are weighted and fused based on preset weights to determine the comprehensive score information. Based on this comprehensive score information, the candidate evaluation model information is screened to obtain the target evaluation model information. Specifically, when screening candidate evaluation model information, the weights of each index can be preset, and the stability variance, pairwise accuracy, tee rate, and sensitivity are weighted and fused to calculate the comprehensive score information for each candidate model. For example, to emphasize consistency with human preferences, pairwise accuracy and sensitivity can be given higher weights; to ensure reproducibility, stability variance can be given an appropriate weight; and to suppress model avoidance decisions, tee rate can be given an inverse weight. Based on the comprehensive score information, the candidate evaluation models are ranked, and the model with the highest comprehensive score is selected as the target evaluation model information. This selection process can be executed automatically periodically, for example, by retesting candidate models daily using the latest evaluation sample set, updating the mapping relationship between evaluation indices and the optimal model in real time, and ensuring that the currently optimal Judge model is always used.

[0035] In one feasible implementation, step S20 may include steps A11 to A13: Step A11: Obtain the evaluation sample set; It should be noted that the evaluation sample set is a set of paired samples with labeled preference data used to verify the capabilities and quantify the indicators of candidate evaluation models. Publicly available datasets containing paired responses and human preference annotations can be used as the verification benchmark, such as the HelpSteer2 dataset released by NVIDIA Research. To further improve the sample discrimination and evaluation effectiveness, sample pairs with human score differences greater than or equal to preset thresholds on "higher is better" indicators such as usefulness, correctness, and coherence can be selected to construct a highly discriminative paired subset as the evaluation sample set, thereby more accurately measuring the differences in the capabilities of candidate evaluation models in terms of stability and human preference alignment.

[0036] Step A12: Determine the stability evaluation index information and alignment evaluation index information of the candidate evaluation model information based on the evaluation sample set; Understandably, stability evaluation metrics are a set of quantitative indicators used to measure the consistency and reliability of a candidate evaluation model's output in multiple repeated evaluations. Since large language models inherently possess a certain degree of output randomness, even with the same input, multiple calls may produce different scoring results. If the evaluation model's scores fluctuate too much on the same sample, it will directly affect the credibility of the evaluation conclusion. Therefore, stability evaluation metrics can be stability variance metrics. This metric is obtained by calculating the dispersion of score values ​​after multiple rounds of independent scoring of the candidate evaluation model on the same sample pair. The lower the stability variance, the more stable the evaluation model's output and the stronger the reproducibility of the evaluation results. Alignment evaluation metrics are a set of quantitative indicators used to measure the consistency between the evaluation results of candidate evaluation models and human annotation preferences. Under the LLM-as-a-Judge paradigm, the evaluation model should not only be stable, but also its judgments should align with the true preferences of humans. Therefore, alignment evaluation metrics can include pairwise accuracy, tie rate, and sensitivity. Among them, the pairwise accuracy metric represents the proportion of the evaluation model's preference ranking in pairwise sample comparisons that is consistent with human annotation preferences. The tie rate metric reflects the frequency with which the evaluation model makes no difference judgments between two responses. An excessively high tie rate means that the model avoids decision-making and has insufficient evaluation ability. The sensitivity metric characterizes the evaluation model's ability to capture subtle quality differences and its ability to make clear judgments even in difficult samples where human annotations are inconsistent.

[0037] In one feasible implementation, step A12 may include steps B11-B13: Step B11: Obtain annotation preference data; It should be noted that the labeled preference data is artificial preference label information formed by the labeler after evaluating the quality of paired responses in the evaluation sample set. It is expressed as a preference labeling result that clearly identifies which response is better or the two responses are of equivalent quality in the case of a given prompt word and two candidate responses.

[0038] Step B12: Perform multiple rounds of independent scoring on the candidate evaluation model information based on paired samples in the evaluation sample set to obtain the scoring results; It should be noted that the scoring result is a set of multiple independent scoring data generated by inputting each pair of samples in the evaluation sample set into the candidate evaluation model in sequence, and then having the model quantify and score the two responses according to the preset evaluation indicators and scoring standards. It may include the specific score value given by the candidate evaluation model to the same pair of responses in each round of independent evaluation.

[0039] It is understood that the aforementioned multi-round independent scoring is to ensure that the evaluation results can truly reflect the output stability of the candidate evaluation model itself. The same generation parameter configuration (such as setting the temperature parameter to zero to eliminate random exogenous factors) is used to perform multiple (e.g., five) independent evaluation operations on each pair of samples in the evaluation sample set. Each evaluation is a complete and independent reasoning process that does not depend on the previous scoring results. In this way, the consistency of the model's output is quantified by calculating the dispersion between multiple scoring values.

[0040] Step B13: Based on the scoring results and the labeled preference data, determine the stability evaluation index information and alignment evaluation index information of the candidate evaluation model information. The stability evaluation index information includes the stability variance index information, and the alignment evaluation index information includes the pairwise accuracy index information, the tie rate index information, and the sensitivity index information.

[0041] Understandably, the stability variance metric is a quantitative indicator that measures the stability of the candidate evaluation model's output in multiple repeated evaluations. It is obtained by calculating the dispersion of the scores in each round after multiple rounds of independent scoring of the same pair of samples. The lower the stability variance, the more consistent the model's scoring results and the stronger its reproducibility. The pairwise accuracy metric is a core indicator that measures the degree to which the candidate evaluation model's evaluation results align with human preferences. It is obtained by measuring the proportion of sample pairs whose preference ranking given by the statistical model matches the labeled preference data to the total number of sample pairs. The higher the pairwise accuracy, the more accurately the model can identify high-quality samples. The tie rate is a quantitative indicator that measures whether a candidate evaluation model tends to avoid decisions. It is obtained by statistically determining the proportion of sample pairs where two responses are indistinguishable out of the total number of sample pairs. An excessively high tie rate usually means that the model's ability to judge excellence is insufficient and lacks discrimination. The sensitivity indicator is a quantitative indicator that describes the candidate evaluation model's ability to capture subtle quality differences. In a difficult sample subset where human annotations are inherently divergent or the scoring differences are small, the model can give a clear non-tie judgment. The higher the sensitivity, the better the model can perceive and distinguish subtle quality differences between responses.

[0042] Step A13: Based on the stability evaluation index information and the alignment evaluation index information, the candidate evaluation model information is screened to obtain the target evaluation model information.

[0043] It is understood that by screening the candidate evaluation model information through the stability evaluation index information and the alignment evaluation index information, a comprehensive score and automatic ranking of the candidate evaluation model capabilities can be achieved. This allows for the precise determination of the target evaluation model that achieves the best balance between stability, reliability, and semantic alignment from the candidate model pool. This fundamentally eliminates the subjective arbitrariness and selection bias caused by relying on human experience to specify the evaluation model, and avoids the problem of evaluation results being unreproducible or deviating from true human preferences due to human error in selecting the wrong model. At the same time, by incorporating the tie rate into the screening criteria, the tendency of the model to avoid decision-making in the evaluation can be effectively suppressed, ensuring that the selected model has sufficient ability to judge excellence and fine quality discrimination, and can continuously maintain optimal evaluation performance.

[0044] In one feasible implementation, step A13 may include steps C11-C12: Step C11: Based on preset weights, the stability evaluation index and the alignment evaluation index are weighted and fused to determine the comprehensive score information; It should be noted that the comprehensive score information is a single quantitative score obtained by weighting and summing or weighting and fusing the various quantitative indicators of the candidate evaluation model in the stability evaluation dimension and the alignment evaluation dimension according to the pre-set weight coefficients. It is used to measure the comprehensive ability of the candidate evaluation model, and represents the comprehensive performance of the candidate evaluation model in multiple dimensions such as output stability, degree of alignment with human preferences, and precision of judgment.

[0045] It is understandable that the weighted fusion is a calculation process that assigns differentiated preset weight coefficients to each evaluation indicator based on the emphasis requirements of different quality dimensions in the actual evaluation task. For example, to emphasize the high consistency between the evaluation results and human judgment, pairwise accuracy indicators and sensitivity indicators can be given higher weights. To ensure the reproducibility of the evaluation results, the stability variance indicator is assigned an appropriate weight. At the same time, to suppress the model's tendency to avoid decision-making in the evaluation, the average rate indicator is given a reverse or negative constraint weight. By multiplying the values ​​of each indicator with their corresponding weights and summing them up, a leap from a single stability or single accuracy judgment to a multi-dimensional comprehensive balanced evaluation can be achieved, enabling the screening mechanism to flexibly adapt to the priority needs of specific business scenarios.

[0046] Step C12: Based on the comprehensive scoring information, the candidate evaluation model information is filtered to obtain the target evaluation model information.

[0047] It is understandable that by screening the candidate evaluation model information through the comprehensive scoring information, the target evaluation model with the best comprehensive quality can be accurately identified, completely avoiding the arbitrariness and cognitive bias when manually specifying the model, ensuring the repeatability and consistency of the selection results. At the same time, since this weighted fusion mechanism supports dynamic adjustment of weights according to the evaluation scenario, the selected target evaluation model can better adapt to the specific quality requirements of the current task, achieving an organic unity of evaluation credibility and scenario adaptability.

[0048] Step S30: Based on the task configuration information, inject the corresponding semantic evaluation elements into the model prompt word template of the target evaluation model information to determine the semantic enhancement information of the target evaluation model; It is understood that the semantic evaluation elements are evaluation dimension definitions and key points descriptions that are deeply bound to the current task scenario and retrieved from the preset scenario configuration library. They include the names of the quality indicators that need to be examined in this scenario and their corresponding evaluation criteria. For example, evaluation elements such as "vividness", "participation" and "length" that are clearly defined in the "general communication - casual chat" scenario can be organized and stored through modular YAML configuration files. Each scenario independently defines its own evaluation focus and can use general basic configurations to achieve element reuse, forming a hierarchical and scalable semantic evaluation element system.

[0049] In a specific embodiment, the corresponding target scenario category information is determined based on the task configuration information. Based on the target scenario category information, the corresponding semantic evaluation elements are retrieved from a preset scenario configuration library to determine the semantic evaluation element information. That is, the task configuration information carries the business scenario identifier to which the current evaluation task belongs. Therefore, a scenario classifier can be used to identify and match the business scenario identifier, mapping it to the corresponding specific scenario category in a fine-grained scenario classification system. This fine-grained scenario classification system includes 8 major categories and 58 minor categories. Each minor category corresponds to an independent YAML configuration file. For example, the major categories include MLP task categories, functional writing categories, creative writing categories, code categories, rewriting categories, summarizing categories, exam question categories, and general communication categories, totaling 8 major categories, which are further subdivided into 58 minor categories, such as... Figure 3 As shown, Figure 3 This diagram illustrates the fine-grained scenario classification system for the large language model evaluation method in this application. Specifically, the MLP task category is further subdivided into sorting tasks, text translation, data analysis, classification and recognition, title generation, question generation, reading comprehension, keyword extraction, information extraction, topic modeling, and others. The functional writing category is subdivided into writing product descriptions, job applications, news articles, biographies, emails, legal documents, technical documents, scientific papers, functional writing, and cooking recipes. The creative writing category is subdivided into writing lyrics, social media posts, blog posts, personal essays, creative writing, advertising, marketing materials, speech scripts, and counterfactual writing. The code category is subdivided into code simplification, code generation, and decoding. The code is categorized into code interpretation, code correction and rewriting, and code translation. The rewriting category is further subdivided into text simplification, language polishing, instruction rewriting, text correction, and rewriting. The summary category is subdivided into post summaries, text summaries, and note summaries. The exam question category is subdivided into mathematical reasoning, solving problems with math exams, and solving problems without math exams. The general communication category is subdivided into how-to questions, open-ended questions, general analysis, general explanation, seeking advice, recommendations, value judgments, fact verification, casual conversation, role-playing, planning, and brainstorming. This fine-grained scenario classification system can be implemented through modular YAML configuration files. Each subcategory can independently define its own semantic evaluation elements, achieving a high degree of adaptation to the semantics of specific tasks and effectively bridging the semantic gap between general evaluation and business needs. For example, ... Figure 4 As shown, Figure 4This diagram illustrates the semantic evaluation element identification method of the large language model evaluation approach in this application. The scene classifier identifies the input task as a "general communication - casual conversation" scene, then searches for and loads the corresponding scene configuration file `chitchat.yaml`, explicitly defining scene-specific semantic evaluation elements such as "friendliness" (including sub-dimensions like politeness, enthusiasm, expressing concern, and proactively offering help), "accuracy" (requiring all information provided in the response to be accurate; this standard does not apply to content beyond the scope of knowledge), "vividness" (requiring the use of emoticons, informal expressions, etc., to make the dialogue lively and maintain its attractiveness and interest), as well as "engagement" and "appropriate length." Simultaneously, `chitchat.yaml` reuses the evaluation dimension definitions from `basic_bot.yaml` (basic robot interaction configuration file, providing general basic elements such as interactivity) and `basic_writing.yaml` (basic writing configuration file, providing general basic elements such as completeness and sentence richness) through a configuration referencing mechanism, forming a hierarchical and composable modular configuration system to obtain the semantic evaluation elements.

[0050] The semantic evaluation element information is injected into the model prompt word template of the target evaluation model information to obtain semantic enhancement information of the target evaluation model. That is, while keeping the overall structure of the model prompt word template unchanged, the extracted scene-specific evaluation dimensions, key points, and other content can be filled into the reserved scene element placeholder areas in the template. For example, for a casual conversation scene, the originally blank "key evaluation dimensions" part of the template will be filled with "vividness, participation, and appropriate length", and the originally blank "key points" part will be filled with specific guidance such as "whether the reply uses natural and colloquial expressions, whether it effectively maintains the emotional interaction of the dialogue, and whether the length of the reply matches the user's input". In this way, the semantic enhancement information of the target evaluation model can be obtained for accurate evaluation of scene perception.

[0051] Step S40: Generate corresponding evaluation results based on the semantic enhancement information of the target evaluation model.

[0052] It should be noted that the evaluation results are obtained by the target evaluation model after reasonable reasoning on the content to be evaluated through scene semantic enhancement. Since the target evaluation model at this time is the optimal model with excellent stability and consistency with human preferences, and is generated based on prompts and instructions that integrate scene-specific evaluation elements, it has stronger discriminative power, higher fit with business scenario needs, and richer and more interpretable scoring reasons compared to the evaluation method of manual selection plus general templates, thus effectively improving the credibility and reproducibility of the evaluation results.

[0053] In a specific embodiment, since the evaluation result is generated by the automatically selected optimal evaluation model executing scenario enhancement prompts, such as the Qwen3-30B model with the highest overall score, the evaluation result fundamentally eliminates the selection bias and output fluctuation problems caused by manually specifying the evaluation model. Figure 5 As shown, Figure 5 This diagram illustrates the experimental results of the large language model evaluation method proposed in this application. In experiments on publicly available datasets, a high-discrimination subset of HelpSteer2 was used, employing the intent integrity metric. Three models—Qwen3-30B, Qwen3-4B, and Qwen2.5-7B—were selected to verify the generalization ability of different scenario enhancement modes across different scales and generations of models. Experimental results show that as the degree of scenario information fusion increases, the evaluation performance significantly improves across all models and metrics. From the baseline mode to the full-process mode, pairwise accuracy improved by 13.86 percentage points, the tie rate decreased to 18.11%, and the average number of opinions and reason length also increased significantly. This gradual improvement indicates that the effect of scenario enhancement is systematic, rather than accidental or lucky, further validating the necessity of "end-to-end semantic alignment." Further verification using real mobile banking question-and-answer samples showed a significant improvement over the original scores without introducing significant perturbations. The enhanced model provides a more comprehensive breakdown of user concerns, richer viewpoints and reasons, and more interpretable evaluation results. These results are consistent with experimental conclusions on open-source datasets and have extremely high engineering practical value.

[0054] This embodiment proposes a method for evaluating large language models, which involves acquiring task configuration information and candidate evaluation model information; determining target evaluation model information based on the stability evaluation index information and alignment evaluation index information of the candidate evaluation model information; injecting the corresponding semantic evaluation elements into the model prompt word template of the target evaluation model information based on the task configuration information to determine the semantic enhancement information of the target evaluation model; and generating corresponding evaluation results based on the semantic enhancement information of the target evaluation model. This application addresses the technical challenge of accurately and effectively evaluating large language models. Compared to existing technologies, it dynamically determines the optimal target evaluation model by using stable evaluation metrics and alignment evaluation metrics for candidate evaluation models. During runtime, it identifies the current business scenario based on task configuration information and dynamically injects scenario-specific semantic evaluation elements into the target evaluation model's generic prompt word template, generating semantic enhancement information for the target evaluation model and thus obtaining the evaluation result. This organically coordinates the model selection and prompt enhancement stages. It eliminates the subjective arbitrariness of manually specifying the evaluation model through quantitative metrics, ensuring the stability of the model output and consistency with human preferences. It also bridges the semantic gap between the generic prompt template and specific business needs, allowing the optimal model to execute highly adapted evaluation instructions, significantly improving the model's discriminative power, semantic consistency, and reproducibility.

[0055] Based on the first embodiment of this application, in the second embodiment of this application, the same or similar content as the first embodiment described above can be referred to the above description, and will not be repeated hereafter.

[0056] In this embodiment, refer to Figure 6 , Figure 6 This is a flowchart illustrating Embodiment 2 of the large language model evaluation method of this application. Step S30 specifically includes steps S31 to S33: Step S31: Determine the corresponding target scene category information based on the task configuration information; It should be noted that the target scenario category information is a classification label used to identify the business scenario to which the current evaluation task belongs, realizing the adaptation from general evaluation to personalized scenario.

[0057] In a specific embodiment, a scene classifier can be used to perform semantic recognition and matching of the business scene identifier in the task configuration information. This involves comparing the features of the business scene identifier with the definitions and examples of various scene categories in a fine-grained classification system, mapping it to the most matching specific scene category. For example, if the business scene identifier in the task configuration information is "Mobile Banking Chatbot Response Quality Assessment," the scene classifier, after semantic analysis, maps it to the target scene category "General Communication - Chat" in the classification system, thus transforming a vague or custom business description into a standardized, systematic scene label.

[0058] Step S32: Based on the target scene category information, retrieve the corresponding semantic evaluation elements from the preset scene configuration library to determine the semantic evaluation element information; It should be noted that the semantic evaluation element information is a data set that defines evaluation dimensions and describes key points of focus, deeply bound to specific business scenarios. It clarifies which quality indicators the model should prioritize in the current task scenario and how to judge the quality of these indicators. For example, for the "general communication - casual chat" scenario, the corresponding semantic evaluation element information specifically includes multiple evaluation dimensions such as "vividness," "engagement," and "appropriate length." The "vividness" element requires examining whether the response uses emoticons, internet slang, or lively informal expressions to create a relaxed and fun conversational atmosphere. The "engagement" element guides the evaluation model to focus on whether the response effectively addresses the user's emotions, proactively extends the topic, or raises new questions, thereby maintaining the interactivity of the conversation. The "appropriate length" element defines a standard that the length of the response should match the level of detail in the user's current input, avoiding overly short responses that are perfunctory or overly long responses that are informationally redundant.

[0059] Understandably, the preset scenario configuration library is a file configuration database used to centrally store and distribute evaluation standards specific to various business scenarios. It has a folder directory with a clear hierarchical structure. Its top level is divided by major categories, and the lower level sets independent configuration files for each sub-category, such as YAML files in data serialization format. Following the principles of reusability and composability, a configuration file for a specific scenario can not only define its unique evaluation elements, but also inherit the common evaluation standards from one or more general basic configuration files through reference instructions, thereby forming a hierarchical and low-redundancy configuration management system. Semantic evaluation elements are the smallest logical units that constitute the scenario configuration library. Each element is defined with a clear evaluation dimension name and its corresponding evaluation description text. It is the carrier that transforms abstract evaluation principles into specific and executable scenario-based instructions.

[0060] In a specific embodiment, matching can be performed in a preset scenario configuration library, that is, locating the corresponding major category directory, and then indexing the configuration file named after that sub-scenario, such as chitchat.yaml. By loading and parsing this configuration file, the exclusive semantic evaluation elements directly defined therein can be extracted. For example, for a casual chat scenario, the system will extract the actual descriptive text of elements such as "friendliness" (including sub-dimensions such as politeness, enthusiasm, and expression of concern), "accuracy" (requiring all information provided in the reply to be accurate), "vividness" (requiring the use of emoticons and informal expressions to maintain the fun of the conversation), "engagement," and "appropriate length" from chitchat.yaml, and integrate these texts into semantic evaluation element information.

[0061] Step S33: Inject the semantic evaluation element information into the model prompt word template of the target evaluation model information to obtain the semantic enhancement information of the target evaluation model.

[0062] It should be noted that the semantic enhancement information of the target evaluation model is evaluation instruction information with scene adaptability. It enables the target evaluation model to accurately focus on the quality requirements and discrimination criteria of the current scene while maintaining the consistency of the cross-task evaluation framework. Thus, when performing the evaluation, it can output stable and comparable scores, and fully understand and adapt to the semantic features of specific tasks, effectively bridging the semantic gap between general prompt templates and business requirements.

[0063] In a specific embodiment, comparison mode information is obtained, including baseline mode, semi-process mode, and full-process mode. This information is used to verify the effectiveness of the mechanism. Comparison modes, designed from low to high, can be implemented, including baseline mode, semi-process mode, and full-process mode. Based on controlled variables, the impact of different semantic injection depths on the evaluation results is analyzed. For example, the baseline mode serves as a benchmark, injecting no scene information and using general evaluation prompts throughout the evaluation. The semi-process mode adds a scene classification judgment step to the baseline mode, injecting scene-specific semantic evaluation elements into the model prompt word template only in certain reasoning stages (such as the opinion extraction stage), while using general prompts in other stages. The full-process mode further increases the coverage of scene classification judgment, systematically injecting scene information in all reasoning stages, including opinion extraction, opinion judgment, and reason generation, achieving end-to-end deep semantic alignment. This allows for the determination of corresponding injection strategies and flexible adjustment of the semantic enhancement intervention granularity within a unified evaluation framework.

[0064] Obtain evaluation phase information; when the comparison mode information is the baseline mode, the injection strategy information is to disallow injection; when the evaluation phase information is a partial inference phase and the comparison mode information is a half-process mode, the injection strategy information is to allow injection in the half-stage; when the evaluation phase information is a full stage and the comparison mode information is a full-process mode, the injection strategy information is to allow injection attempts in all stages, i.e. Figure 7 As shown, Figure 7This is a flowchart illustrating the comparison mode process of the large language model evaluation method in this application. Based on the information obtained in the evaluation stage, the corresponding injection strategy information is dynamically determined by jointly judging the currently selected comparison mode information and the specific inference stage of the evaluation task. For example, when the comparison mode information is the baseline mode, the injection strategy information is set to not allow injection regardless of the evaluation stage, that is, the original appearance of the general evaluation prompt words is maintained throughout, which serves as the benchmark for effect comparison. When the evaluation stage information indicates that the current stage is a partial inference stage (such as the opinion extraction stage), and the comparison mode information is a half-process mode, the injection strategy information is determined to be allowed in the half-process stage. Injection refers to integrating scene-specific semantic evaluation elements into the prompt template only at that specific stage, while maintaining general prompts in other stages. This is used to periodically verify the contribution of semantic enhancement. When the evaluation stage information indicates that the current stage covers all stages (such as key links like opinion extraction, opinion judgment, and reason generation), and the comparison mode information is a full-process mode, the injection strategy information corresponds to allowing injection attempts at all stages. This systematically integrates scene-specific semantic evaluation elements throughout the entire evaluation and reasoning process, achieving end-to-end deep semantic alignment. This allows for precise control of the granularity of semantic enhancement intervention within a unified evaluation framework, and enables analysis of the impact of different scene information fusion degrees on the evaluation results.

[0065] Based on the injection strategy information, the semantic evaluation element information is injected into the model prompt word template of the target evaluation model information to obtain the semantic enhancement information of the target evaluation model. That is, when the injection strategy information is not allowed to be injected, the system directly uses the original general template as the semantic enhancement information of the target evaluation model. When the injection strategy information is allowed to be injected in half stages, the system only fills the blank evaluation dimensions and key points in the template with the semantic evaluation elements of the current scene in the prompt fragments corresponding to the specified reasoning stages such as opinion extraction, while the prompts for other stages such as quantitative scoring still maintain the general description. When information is allowed to be injected at all stages, the system systematically integrates semantic evaluation elements such as scenario-specific evaluation dimensions, key points of concern, and judgment criteria into the corresponding placeholder areas in the prompt templates of all reasoning stages, including opinion extraction, opinion judgment, and reason generation. The semantic enhancement information of the target evaluation model obtained in this way not only retains the standardization of the evaluation process and output structure guaranteed by the general prompt template, but also accurately carries the quality focus and judgment requirements of the current business scenario. This allows for targeted fine-grained evaluation of key semantic features of the scenario, effectively bridging the semantic gap between the unified framework and business needs.

[0066] In one feasible implementation, step S33 may include steps D11~D13: Step D11: Obtain comparison mode information, which includes baseline mode, half-process mode and full-process mode; It should be noted that the comparison pattern information is a pattern parameter used to control the timing and extent of injecting scene-specific semantic evaluation elements into the model prompt word template.

[0067] Understandably, the baseline mode serves as a benchmark and does not inject any scenario information throughout the process, only using general evaluation prompts. The semi-process mode adds scenario classification judgment on the basis of the baseline and injects scenario information only in some reasoning stages (such as the opinion extraction stage). The full-process mode further extends the injection of scenario information to all reasoning stages such as opinion extraction, opinion judgment, and reason generation, achieving end-to-end deep semantic alignment.

[0068] Step D12: Determine the corresponding injection strategy information based on the comparison pattern information; It should be noted that the injection strategy information is the operational rule used to inject semantic evaluation elements into the model prompt word template in the corresponding evaluation stage, which can clearly define the admission conditions and scope of the injection behavior.

[0069] In one feasible implementation, step D12 may include steps E11 to E14: Step E11: Obtain assessment phase information; It should be noted that the assessment stage information refers to the specific reasoning stage in the single-indicator assessment process of the current assessment task, which may include multiple stages such as opinion extraction, opinion judgment, quantitative scoring, and reason generation.

[0070] Step E12: When the comparison mode information is the baseline mode, the injection strategy information is "injection not allowed"; Understandably, when the comparison mode information is set to baseline mode, regardless of which specific reasoning stage the current evaluation stage information indicates, the injection strategy information is uniformly determined to be not allowed to be injected. This is because the baseline mode is designed as a benchmark for effect comparison, and it needs to maintain the original state of the general evaluation prompt words and not introduce any scene-specific semantic evaluation elements, thereby providing a pure reference baseline for difference comparison.

[0071] Step E13: When the evaluation phase information is a partial reasoning phase and the comparison mode information is a half-process mode, the injection strategy information is to allow injection in the half-stage. Understandably, the semi-process mode can test the contribution of phased semantic enhancement. Therefore, when the evaluation phase information indicates that the current stage is a partial reasoning stage (such as the opinion extraction stage), the injection strategy information is determined to allow the integration of scene semantic evaluation elements into the prompt template at that specific stage. In other stages such as quantitative scoring, the general prompt remains unchanged. This phased injection design can accurately attribute the marginal improvement of the evaluation results brought by the injection of scene information in different reasoning stages.

[0072] Step E14: When the evaluation phase information is full-phase and the comparison mode information is full-process mode, the injection strategy information is to allow injection attempts in all phases.

[0073] Understandably, the end-to-end deep semantic alignment is pursued in the full-process model. Therefore, when the evaluation stage information indicates that the current stage covers all key reasoning stages such as opinion extraction, opinion judgment, and reason generation, the injection strategy information is determined to allow the systematic injection of scenario-specific semantic evaluation elements in all stages. By integrating semantic information such as evaluation dimensions, key points of concern, and judgment criteria throughout the entire evaluation reasoning process, the semantic gap between general prompt templates and specific business needs can be bridged to the greatest extent, and the global enhancement effect of scenario information on evaluation quality can be fully utilized.

[0074] Step D13: Based on the injection strategy information, the semantic evaluation element information is injected into the model prompt word template of the target evaluation model information to obtain the semantic enhancement information of the target evaluation model.

[0075] It is understandable that the model prompt word template of the target evaluation model information is a basic prompt instruction framework used to guide the target evaluation model to perform evaluation tasks according to a standardized process. It may include general content such as evaluation task description, evaluation stage division, and output format requirements, providing a unified operation process and output structure for all evaluation tasks, and ensuring the overall consistency and comparability of evaluation results across scenarios and batches. However, due to the lack of semantic details for specific task scenarios, it cannot provide differentiated guidance for the quality focus of different business scenarios when used alone. Therefore, it needs to be dynamically combined with semantic evaluation elements to achieve a high degree of adaptability to specific tasks.

[0076] This embodiment proposes a method for evaluating large language models. Based on task configuration information, it determines the corresponding target scene category information; based on the target scene category information, it retrieves corresponding semantic evaluation elements from a preset scene configuration library to determine semantic evaluation element information; and it injects the semantic evaluation element information into the model prompt word template of the target evaluation model information to obtain semantic enhancement information for the target evaluation model. This solves the technical problem of how to accurately and effectively evaluate large language models. Compared with existing technologies, this application identifies the target scene category through task configuration information, retrieves scene-specific semantic evaluation elements from a preset scene configuration library, and injects them into the prompt word template of the target evaluation model to generate semantic enhancement information for the target evaluation model. This effectively bridges the semantic gap between general prompt templates and specific business requirements, enabling the optimal evaluation model to accurately focus on the quality priorities of the current scene during evaluation, thereby significantly improving the discriminative power, interpretability, and cross-task reproducibility of the evaluation results. It truly achieves highly flexible scene adaptation while maintaining a consistent evaluation framework.

[0077] This application also provides a large language model evaluation device; please refer to [reference needed]. Figure 8 The large language model evaluation device includes: Module 10 is used to acquire task configuration information and candidate evaluation model information; Processing module 20 is used to determine target evaluation model information based on the stable evaluation index information and the aligned evaluation index information of the candidate evaluation model information; The processing module 20 is further configured to inject the corresponding semantic evaluation elements into the model prompt word template of the target evaluation model information based on the task configuration information, and determine the semantic enhancement information of the target evaluation model; The execution module 30 is used to generate corresponding evaluation results based on the semantic enhancement information of the target evaluation model.

[0078] The processing module 20 is also used to acquire an evaluation sample set; Based on the evaluation sample set, determine the stability evaluation index information and alignment evaluation index information of the candidate evaluation model information; Based on the stability evaluation index information and the alignment evaluation index information, the candidate evaluation model information is screened to obtain the target evaluation model information.

[0079] The processing module 20 is also used to acquire annotation preference data; The candidate evaluation model information is independently scored multiple times based on paired samples in the evaluation sample set to obtain the scoring results; Based on the scoring results and the labeled preference data, the stability evaluation index information and alignment evaluation index information of the candidate evaluation model information are determined. The stability evaluation index information includes stability variance index information, and the alignment evaluation index information includes pairwise accuracy index information, tie rate index information, and sensitivity index information.

[0080] The processing module 20 is further configured to perform weighted fusion of the stability evaluation index and the alignment evaluation index based on preset weights to determine comprehensive scoring information; Based on the comprehensive scoring information, the candidate evaluation model information is filtered to obtain the target evaluation model information.

[0081] The processing module 20 is further configured to determine the corresponding target scene category information based on the task configuration information; Based on the target scene category information, the corresponding semantic evaluation elements are retrieved from the preset scene configuration library to determine the semantic evaluation element information; The semantic evaluation element information is injected into the model prompt word template of the target evaluation model information to obtain the semantic enhancement information of the target evaluation model.

[0082] The processing module 20 is also used to acquire mode information, which includes baseline mode, half-process mode and full-process mode. Based on the pattern information, the corresponding injection strategy information is determined; Based on the injection strategy information, the semantic evaluation element information is injected into the model prompt word template of the target evaluation model information to obtain the semantic enhancement information of the target evaluation model.

[0083] The processing module 20 is also used to acquire evaluation stage information; When the mode information is the baseline mode, the injection strategy information is that injection is not allowed; When the evaluation phase information is a partial reasoning phase and the pattern information is a half-process pattern, the injection strategy information is to allow injection in the half-phase. When the evaluation phase information is full-phase and the mode information is full-process mode, the injection strategy information allows injection to be attempted in all phases.

[0084] The large language model evaluation device provided in this application, employing the large language model evaluation method described in the above embodiments, can solve the technical problem of how to accurately and effectively perform large language model evaluation. Compared with the prior art, the beneficial effects of the large language model evaluation device provided in this application are the same as those of the large language model evaluation method provided in the above embodiments, and other technical features in the large language model evaluation device are the same as those disclosed in the methods of the above embodiments, and will not be repeated here.

[0085] This application provides a large language model evaluation device, which includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the large language model evaluation method in Embodiment 1 above.

[0086] The following is for reference. Figure 9 The diagram illustrates a structural schematic suitable for implementing the large language model evaluation device of the embodiments of this application. The large language model evaluation device in the embodiments of this application may include, but is not limited to, mobile terminals such as mobile phones, laptops, digital broadcast receivers, PDAs (Personal Digital Assistants), PADs (Portable Application Description), PMPs (Portable Media Players), in-vehicle terminals (e.g., in-vehicle navigation terminals), and fixed terminals such as digital TVs and desktop computers. Figure 9 The large language model evaluation device shown is merely an example and should not impose any limitations on the functionality and scope of use of the embodiments of this application.

[0087] like Figure 9 As shown, the large language model evaluation device may include a processing unit 1001 (e.g., a central processing unit, a graphics processing unit, etc.), which can perform various appropriate actions and processes according to a program stored in ROM (Read Only Memory) 1002 or a program loaded from storage device 1003 into RAM (Random Access Memory) 1004. RAM 1004 also stores various programs and data required for the operation of the large language model evaluation device. The processing unit 1001, ROM 1002, and RAM 1004 are interconnected via bus 1005. Input / output (I / O) interface 1006 is also connected to the bus. Typically, the following systems can be connected to I / O interface 1006: input devices 1007 including, for example, touch screens, touchpads, keyboards, mice, image sensors, microphones, accelerometers, gyroscopes, etc.; output devices 1008 including, for example, liquid crystal displays (LCDs), speakers, vibrators, etc.; storage devices 1003 including, for example, magnetic tapes, hard disks, etc.; and communication devices 1009. Communication device 1009 allows the large language model evaluation device to communicate wirelessly or wiredly with other devices to exchange data. Although the figure shows a large language model evaluation device with various systems, it should be understood that it is not required to implement or possess all of the systems shown. More or fewer systems may be implemented alternatively.

[0088] Specifically, according to the embodiments disclosed in this application, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments disclosed in this application include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via a communication device, or installed from storage device 1003, or installed from ROM 1002. When the computer program is executed by processing device 1001, it performs the functions defined in the methods of the embodiments disclosed in this application.

[0089] The large language model evaluation device provided in this application, employing the large language model evaluation method described in the above embodiments, can solve the technical problem of how to accurately and effectively perform large language model evaluation. Compared with the prior art, the beneficial effects of the large language model evaluation device provided in this application are the same as those of the large language model evaluation method provided in the above embodiments, and other technical features of this large language model evaluation device are the same as those disclosed in the previous embodiment method, and will not be repeated here.

[0090] It should be understood that the various parts disclosed in this application can be implemented using hardware, software, firmware, or a combination thereof. In the description of the above embodiments, specific features, structures, materials, or characteristics can be combined in any suitable manner in one or more embodiments or examples.

[0091] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

[0092] This application provides a computer-readable storage medium having computer-readable program instructions (i.e., a computer program) stored thereon, the computer-readable program instructions being used to execute the large language model evaluation method described in the above embodiments.

[0093] The computer-readable storage medium provided in this application may be, for example, a USB flash drive, but is not limited to, electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices, or any combination thereof. More specific examples of computer-readable storage media may include, but are not limited to: electrical connections having one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In this embodiment, the computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, system, or device. The program code contained on the computer-readable storage medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, RF (Radio Frequency), etc., or any suitable combination thereof.

[0094] The aforementioned computer-readable storage medium may be included in the large language model evaluation device; or it may exist independently and not be assembled into the large language model evaluation device.

[0095] The aforementioned computer-readable storage medium carries one or more programs. When these programs are executed by a large language model evaluation device, the large language model evaluation device performs the following actions: acquires task configuration information and candidate evaluation model information; determines target evaluation model information based on the stability evaluation index information and alignment evaluation index information of the candidate evaluation model information; injects the corresponding semantic evaluation elements into the model prompt word template of the target evaluation model information based on the task configuration information, thereby determining the semantic enhancement information of the target evaluation model; and generates corresponding evaluation results based on the semantic enhancement information of the target evaluation model.

[0096] Computer program code for performing the operations of this application can be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, and C++, and conventional procedural programming languages ​​such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a Local Area Network (LAN) or a Wide Area Network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0097] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0098] The modules described in the embodiments of this application can be implemented in software or hardware. The names of the modules do not necessarily limit the functionality of the unit itself.

[0099] The readable storage medium provided in this application is a computer-readable storage medium that stores computer-readable program instructions (i.e., computer programs) for executing the above-described large language model evaluation method, thereby solving the technical problem of how to accurately and effectively perform large language model evaluation. Compared with the prior art, the beneficial effects of the computer-readable storage medium provided in this application are the same as those of the large language model evaluation method provided in the above embodiments, and will not be repeated here.

[0100] The above description is only a part of the embodiments of this application and does not limit the patent scope of this application. All equivalent structural transformations made under the technical concept of this application and using the contents of the specification and drawings of this application, or direct / indirect applications in other related technical fields, are included in the patent protection scope of this application.

Claims

1. A method for evaluating large language models, characterized in that, The method includes: Obtain task configuration information and candidate evaluation model information; Based on the stability evaluation index information and alignment evaluation index information of the candidate evaluation model information, the target evaluation model information is determined. Based on the task configuration information, the corresponding semantic evaluation elements are injected into the model prompt word template of the target evaluation model information to determine the semantic enhancement information of the target evaluation model; The corresponding evaluation results are generated based on the semantic enhancement information of the target evaluation model.

2. The method as described in claim 1, characterized in that, The step of determining the target evaluation model information based on the stability evaluation index information and alignment evaluation index information of the candidate evaluation model information includes: Obtain the evaluation sample set; Based on the evaluation sample set, determine the stability evaluation index information and alignment evaluation index information of the candidate evaluation model information; Based on the stability evaluation index information and the alignment evaluation index information, the candidate evaluation model information is screened to obtain the target evaluation model information.

3. The method as described in claim 2, characterized in that, The step of determining the stable evaluation index information and alignment evaluation index information of the candidate evaluation model information based on the evaluation sample set includes: Obtain annotation preference data; The candidate evaluation model information is independently scored multiple times based on paired samples in the evaluation sample set to obtain the scoring results; Based on the scoring results and the labeled preference data, the stability evaluation index information and alignment evaluation index information of the candidate evaluation model information are determined. The stability evaluation index information includes stability variance index information, and the alignment evaluation index information includes pairwise accuracy index information, tie rate index information, and sensitivity index information.

4. The method as described in claim 2, characterized in that, The step of filtering the candidate evaluation model information based on the stability evaluation index information and the alignment evaluation index information to obtain the target evaluation model information includes: The stability evaluation index and the alignment evaluation index are weighted and fused based on preset weights to determine the comprehensive score information; Based on the comprehensive scoring information, the candidate evaluation model information is filtered to obtain the target evaluation model information.

5. The method as described in claim 1, characterized in that, The step of injecting the corresponding semantic evaluation elements into the model prompt word template of the target evaluation model information based on the task configuration information, and determining the semantic enhancement information of the target evaluation model, includes: Based on the task configuration information, the corresponding target scenario category information is determined; Based on the target scene category information, the corresponding semantic evaluation elements are retrieved from the preset scene configuration library to determine the semantic evaluation element information; The semantic evaluation element information is injected into the model prompt word template of the target evaluation model information to obtain the semantic enhancement information of the target evaluation model.

6. The method as described in claim 5, characterized in that, The step of injecting the semantic evaluation element information into the model prompt word template of the target evaluation model information to obtain the semantic enhancement information of the target evaluation model includes: Acquire comparison mode information, which includes baseline mode, half-process mode, and full-process mode; The corresponding injection strategy information is determined based on the comparison pattern information. Based on the injection strategy information, the semantic evaluation element information is injected into the model prompt word template of the target evaluation model information to obtain the semantic enhancement information of the target evaluation model.

7. The method as described in claim 6, characterized in that, The step of determining the corresponding injection strategy information based on the comparison pattern information includes: Obtain information during the assessment phase; When the comparison mode information is the baseline mode, the injection strategy information is that injection is not allowed; When the evaluation phase information is a partial reasoning phase and the comparison mode information is a half-process mode, the injection strategy information allows injection in the half-phase. When the evaluation phase information is full-stage and the comparison mode information is full-process mode, the injection strategy information allows injection to be attempted in all stages.

8. A large language model evaluation device, characterized in that, The device includes: The acquisition module is used to acquire task configuration information and candidate evaluation model information; The processing module is used to determine the target evaluation model information based on the stable evaluation index information and the aligned evaluation index information of the candidate evaluation model information; The processing module is also used to inject the corresponding semantic evaluation elements into the model prompt word template of the target evaluation model information based on the task configuration information, and determine the semantic enhancement information of the target evaluation model; The execution module is used to generate corresponding evaluation results based on the semantic enhancement information of the target evaluation model.

9. A large language model evaluation device, characterized in that, The device includes: a memory, a processor, and a computer program stored in the memory and executable on the processor, the computer program being configured to implement the steps of the large language model evaluation method as described in any one of claims 1 to 7.

10. A storage medium, characterized in that, The storage medium is a computer-readable storage medium, and a computer program is stored on the storage medium. When the computer program is executed by a processor, it implements the steps of the large language model evaluation method as described in any one of claims 1 to 7.

Images