Model evaluation method and electronic device

Abstract

This invention provides a model evaluation method and electronic device, relating to the field of large language model technology. The method includes: acquiring N sets of question data corresponding to N original questions, where N≥1; each set of question data includes: reference questions corresponding to various question types, model answers obtained after each reference question is input into the model to be evaluated, and standard answers; the reference questions for each question type (excluding original types) are obtained by transforming the original questions according to the question type; for each question type, comparing the model answers of all reference questions under that question type with the standard answers in the corresponding question data to obtain a quality index value characterizing the response quality of the model to be evaluated for that question type; and determining performance index parameters of the model to be evaluated based on the quality index values ​​corresponding to each question type.

Description

Technical Field

[0001] This invention relates to the field of large language model technology, specifically to a model evaluation method and electronic device. Background Technology

[0002] With the development of large language model technology, knowledge-based question answering capabilities based on large models are gradually being introduced into scenarios such as energy storage system consulting, fault explanation, safety risk analysis, and operation and maintenance assistance. The energy storage field involves a wide range of professional knowledge, including battery cells, BMS, thermal management systems, PCS, grid-side control, fire safety, and system integration. The questions are usually highly specialized and safety-sensitive.

[0003] However, current large-scale model evaluation methods in the energy storage field still have several limitations in engineering practice: First, existing evaluation methods lack domain-specific expertise; most existing methods focus on the general performance of large models while neglecting the evaluation needs specific to certain domains. Second, existing large-scale model evaluation methods are too dimensional to reflect the comprehensive capabilities of the model or guide optimization. Third, automated and manual evaluations have long been separated, lacking quantitative indicators for comprehensively evaluating large-scale model performance. Therefore, existing large-scale model evaluation methods largely remain at the level of simple text matching or manual judgment, failing to achieve a deep integration of professionalism, robustness, and scalability, and thus cannot support the large-scale model evaluation requirements of today's intelligent operation and maintenance scenarios. Summary of the Invention

[0004] The purpose of this invention is to provide a model evaluation method and electronic device, which achieves comprehensive and efficient evaluation of the large language model of energy storage operation and maintenance by constructing a domain-adaptive evaluation index system, a term weight enhancement mechanism, a multi-type problem evaluation strategy, and a closed-loop optimization path.

[0005] To achieve the above objectives, the present invention provides a model evaluation method, comprising: Obtain N sets of question data corresponding to N original questions, where N≥1. Each set of question data includes: reference questions corresponding to various question types, model answers obtained after each reference question is input into the model to be evaluated, and standard answers; the reference questions for each of the non-original question types are obtained by transforming the original questions according to the question type. For each question type, the model answers to all reference questions under that question type are compared with the standard answers in the question data to obtain a quality index value that characterizes the response quality of the model to be evaluated for that question type. Based on the quality index values ​​corresponding to each of the aforementioned problem types, the performance index parameters of the model to be evaluated are determined.

[0006] The present invention also provides an electronic device, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform the model evaluation method as described above.

[0007] The present invention also provides a computer-readable storage medium, which is a non-volatile or non-transient storage medium, on which a computer program is stored, and which is executed by a processor to perform the model evaluation method described above.

[0008] In one embodiment, the method further includes: Based on the reference score values ​​of the model answers to the reference questions under each of the aforementioned question types, determine the quality score value corresponding to each of the aforementioned question types; Based on the quality index values ​​corresponding to each of the aforementioned problem types, the performance index parameters of the model to be evaluated are determined, including: Based on the quality index values ​​and quality score values ​​corresponding to each of the aforementioned problem types, the performance index parameters of the model to be evaluated are determined.

[0009] In one embodiment, determining the performance index value of the model to be evaluated based on the quality index value corresponding to each of the aforementioned problem types and the quality score value includes: For each of the aforementioned problem types, based on the fusion coefficient corresponding to the problem type, the quality indicator value and the quality score value corresponding to the problem type are fused to obtain the fusion score corresponding to the problem type; Based on the weight coefficients corresponding to each of the aforementioned problem types and the fusion score, the comprehensive performance index of the model to be evaluated is calculated. The performance index parameters include: the comprehensive performance index.

[0010] In one embodiment, the multiple problem types are divided into multiple levels, and each level includes multiple problem types; For each level, the hierarchical performance index of the model to be evaluated at that level is calculated based on the weight coefficients corresponding to the problem types included in that level and the fusion score; the performance index parameters also include: the hierarchical performance index of the model to be evaluated at all levels.

[0011] In one embodiment, for each question type, a quality index value characterizing the response quality of the model to be evaluated for the corresponding question type is obtained by comparing the model's answers to all reference questions under that question type with the standard answers in the question data to which it belongs, including: For each of the aforementioned problem types: The model answer to each reference question under the question type is compared with the standard answer in the question data to obtain the overlap and semantic similarity between the model answer and the standard answer of each reference question; Based on the overlap and semantic similarity between the model answers and the standard answers to each of the reference questions, the quality index value corresponding to the question type is obtained.

[0012] In one embodiment, the overlap between the model answer to the reference question and the standard answer is obtained as follows: For each of the reference questions under the aforementioned question type: The model answer and the standard answer are segmented according to a preset professional terminology database, and the words in the model answer and the standard answer are replaced with the corresponding standard terms according to the preset mapping relationship between standard terms and synonyms. For each word in the model's answer and the standard answer, if the word matches a preset high-weight term, the weight of the word is increased, and the increased weight of the word is greater than 1. Search for whether key numerical items exist in the model's answer and the standard answer; If they exist, extract the key numerical items from the model's answer and the standard answer, and increase the weight of the extracted key numerical items. The increased weight of the key numerical items is greater than 1. Based on the vocabulary and key numerical items in the model answer and the standard answer, the ROUGE index parameters, which characterize the overlap between the model answer and the standard answer, are calculated; the ROUGE index parameters include: ROUGE-1 score, ROUGE-2 score, and ROUGE-L score.

[0013] In one embodiment, the semantic similarity between the model's answer to the reference question and the standard answer is obtained as follows: For each reference question under the aforementioned question type, the cosine similarity between the model answer to the reference question and the standard answer is calculated, which serves as the semantic similarity between the model answer to the reference question and the standard answer.

[0014] In one embodiment, the formula for calculating the quality index value corresponding to the problem type is: ; in, This indicates the value of the quality indicator. , , All are preset weight values. This represents the mean ROUGE-L score of all the reference questions under the given question type. This represents the mean ROUGE-1 score of all the reference questions under the given question type. This represents the mean ROUGE-2 score of all the reference questions under the given question type. This represents the mean semantic similarity between all the reference questions and the standard answer under the given question type.

[0015] In one embodiment, the non-original question type includes at least one of the following: semantically invariant rewriting type, term substitution and abbreviation usage type, typo and grammatical noise addition type, question generalization type, question concretization type, cross-knowledge point association and reasoning type, and marginal case and negative query type.

[0016] In one embodiment, the various question types are divided into four levels: semantic layer, lexical layer, knowledge layer, and logical layer. The semantic layer includes: the original type and the semantically invariant rewritten type; The vocabulary layer includes: the terminology substitution and abbreviation usage types and the addition of typos and grammatical noise types; The knowledge layer includes: the problem generalization type and the problem specificity type; The logical layer includes: the cross-knowledge point association and reasoning type and the edge case and negative query type. Attached Figure Description

[0017] Figure 1 This is a schematic diagram of the model evaluation method in the first embodiment of the present invention; Figure 2 This is a schematic diagram of a set of problem data in the first embodiment of the present invention; Figure 3 Yes Figure 2 The diagram shown illustrates the model evaluation of the problem data. Figure 4 This is a comparison chart of the comprehensive performance indices of multiple models to be evaluated according to the first embodiment of the present invention; Figure 5 This is a comparison chart of hierarchical performance indices of multiple models to be evaluated according to the first embodiment of the present invention. Detailed Implementation

[0018] The embodiments of the present invention will be described in detail below with reference to the accompanying drawings to provide a clearer understanding of the purpose, features, and advantages of the present invention. It should be understood that the embodiments shown in the drawings are not intended to limit the scope of the present invention, but are merely illustrative of the essential spirit of the technical solution of the present invention.

[0019] In the following description, certain specific details are set forth for the purpose of illustrating various disclosed embodiments in order to provide a thorough understanding of the various disclosed embodiments. However, those skilled in the art will recognize that embodiments may be practiced without one or more of these specific details. In other instances, well-known apparatuses, structures, and techniques associated with this application may not have been shown or described in detail to avoid unnecessarily obscuring the description of the embodiments.

[0020] Unless the context requires otherwise, throughout the specification and claims, the word “comprising” and its variations, such as “including” and “having”, shall be understood to have an open, inclusive meaning, that is, to be interpreted as “including, but not limited to”.

[0021] Throughout this specification, references to "an embodiment" or "an embodiment" indicate that a particular feature, structure, or characteristic described in connection with the embodiment is included in at least one embodiment. Therefore, the appearance of "in an embodiment" or "an embodiment" in various places throughout the specification does not necessarily refer to the same embodiment. Furthermore, a particular feature, structure, or characteristic may be combined in any manner in one or more embodiments.

[0022] The singular forms “a” and “the” used in this specification and the appended claims include plural references unless otherwise expressly stated herein. It should be noted that the term “or” is generally used to include the meaning of “or / and” unless otherwise expressly stated herein.

[0023] In the following description, in order to clearly demonstrate the structure and working method of the present invention, a number of directional terms will be used. However, terms such as "front", "back", "left", "right", "outside", "inside", "outward", "inward", "up", and "down" should be understood as convenient terms and not as limiting terms.

[0024] With the deepening application of large language models in the energy field, their potential in intelligent operation and maintenance of electrochemical energy storage systems is becoming increasingly prominent. However, there are many types of existing large language models, and their performance in intelligent operation and maintenance tasks varies. There is an urgent need for an evaluation method to measure the performance differences of different large language models in this specific task. Existing evaluation methods mostly adopt the evaluation paradigm of general natural language processing tasks, which has the following challenges: First, the evaluation indicators have strong generalization but weak domain adaptability. Although general indicators such as ROUGE and BLEU can measure the surface overlap of text, they cannot capture the synonym substitution, logical causal chain, and accuracy of numbers involved in energy storage field professional terms. Second, the problem types are limited and do not cover the diversity of actual operation and maintenance scenarios. Actual energy storage operation and maintenance problems cover a variety of types, such as original text understanding, noise robustness, and cross-knowledge point reasoning. Existing evaluation datasets and methods have failed to systematically classify and differentiate them. Third, there is a lack of an effective integration mechanism between manual and automated evaluation. Automated scoring and manual scoring are simply listed side by side without establishing a comprehensive evaluation system that integrates weights and dynamically adjusts them, resulting in one-sided evaluation results or significant subjective bias.

[0025] This invention aims to address the problems existing in the evaluation process of large language models in the intelligent operation and maintenance scenario of electrochemical energy storage systems, such as the single evaluation dimension, misunderstanding of professional terminology, untraceable logical reasoning, and high cost of manual evaluation. Traditional evaluation methods mostly rely on general natural language processing indicators (such as BLEU and ROUGE), lacking specific consideration of the professional knowledge structure, semantic logic, and interpretability of operation and maintenance decisions in the energy storage field. This makes it difficult to scientifically quantify and iteratively optimize the performance of models in actual operation and maintenance tasks.

[0026] The first embodiment of this invention relates to a model evaluation method applied to electronic devices, such as servers or common computer devices like desktop computers and laptops. The electronic devices can use this model evaluation method to evaluate the performance of large language models (models to be evaluated) in the field of energy storage.

[0027] The large language model specifically for the energy storage field referred to in this embodiment includes, but is not limited to, a general large language model and a domain-specific large language model that has been fine-tuned with technical knowledge in the energy storage field.

[0028] like Figure 1 The diagram shown is a flowchart illustrating the model evaluation method in this embodiment.

[0029] Step 101: Obtain N sets of question data corresponding to N original questions, where N≥1. Each set of question data includes: reference questions corresponding to various question types, model answers obtained after each reference question is input into the model to be evaluated, and standard answers; the reference questions for each of the non-original question types are obtained by transforming the original questions according to the question type.

[0030] Specifically, one can first acquire professional knowledge in the field of energy storage. Sources of knowledge may include relevant national standards, industry standards, enterprise technical specifications, energy storage system design and operation and maintenance manuals, accident analysis reports, published papers, professional books, and technical information related to batteries, BMS, thermal management systems, PCS, fire protection systems, and power plant operation and maintenance. By organizing this technical information, professional knowledge in the field of energy storage can be formed, covering the main technical aspects of energy storage systems.

[0031] Based on acquiring professional knowledge in the field of energy storage, corresponding question-answer pairs are designed, each pair including the original question and its corresponding standard answer. This application categorizes problems in the field of electrochemical energy storage into multiple question types, including the original type and at least one non-original type. Reference questions under the original type correspond to the unrewritten original question. Furthermore, by rewriting the original question according to the rewriting rules corresponding to each non-original type, reference questions are obtained for each non-original type. The rewritten reference questions can be one or more; that is, there can be one or more reference questions for each question type other than the original type. Based on the above process, for each original question, reference questions for each question type can be obtained according to pre-defined multiple question types. These reference questions are then input into the model to be evaluated, yielding the model's output answer. Each original question has a pre-built standard answer. This results in a set of question data for each original question, including: reference questions corresponding to multiple question types, the model answers obtained after inputting these reference questions into the model to be evaluated, and the standard answer. Since non-original type reference questions are also rewritten from original questions, they also use the standard answer of the original question as their corresponding standard answer; that is, all reference questions in the same set of question data correspond to the same standard answer.

[0032] Thus, N sets of problem data corresponding to N original problems can be obtained, where N≥1; reference problems of each problem type in these N sets of problem data are summarized to participate in the performance evaluation of the model to be evaluated; reference problems of multiple problem types participate in the performance evaluation process of the model to be evaluated, and each problem type has corresponding feature descriptions and evaluation focuses; furthermore, corresponding weight coefficients are set for each problem type, and the sum of the weight coefficients corresponding to all problem types is 1.

[0033] For example, non-original question types include at least one of the following: semantically invariant rewriting type, terminology substitution and abbreviation usage type, typo and grammatical noise addition type, question generalization type, question concretization type, cross-knowledge point association and reasoning type, and marginal case and negative query type.

[0034] For example, please refer to Table 1 below, which shows the corresponding feature descriptions, evaluation focuses, and weighting coefficients for each problem type.

[0035] Table 1:

[0036] For example, the original question was "What are the discharge procedures for testing lithium batteries and battery packs used in energy storage systems?"; please refer to Figure 2 This shows a set of problem data corresponding to the original problem; The primitive type uses the original text of the original problem, so there are no rewriting rules. It is used to determine the model's ability to call knowledge.

[0037] The rewriting rules for semantically invariant rewriting types include: rearranging word order and replacing technical terms with non-technical terms to obtain semantically invariant sentences, in order to test whether the model has general comprehension capabilities. For example, Figure 2 As shown, in the original problem, "lithium batteries and battery packs for energy storage systems" has been changed to "lithium batteries and battery packs for energy storage systems", and "program" has been changed to "process".

[0038] The rewriting rules corresponding to the types of terminology substitution and abbreviation usage include: replacing non-technical terms with technical terms and abbreviating technical terms into English, in order to test the model's ability to understand key domain terms.

[0039] The rewriting rules corresponding to the types of typos and grammatical noise include: omitting some words or adding some redundant words, or changing individual words into typos, in order to test the model's ability to resist interference.

[0040] The rewriting rules corresponding to problem generalization types include: removing some conditional restrictions from the original problem to assess the model's ability to perform fuzzy knowledge lookup. For example, Figure 2 As shown, the "discharge procedure for testing" in the original question is modified to "general steps of testing".

[0041] The rewriting rules corresponding to specific question types include: based on the standard answer to the original question, adding some restrictions to the original question to determine the model's ability to accurately find knowledge.

[0042] The rewriting rules corresponding to cross-knowledge point association and reasoning types include: obtaining other knowledge points that are related to the knowledge points in the original question, and setting questions based on these related knowledge points to test the model's ability to understand multiple knowledge points. For example, Figure 2 As shown, the original question involves lithium battery test discharge, which requires knowledge of battery cluster safety.

[0043] The rewriting rules for marginal cases and negative query types include: based on the standard answer to the original question, setting specific cases, such as whether specific conditions or behaviors are feasible, to test the model's logical reasoning ability.

[0044] Step 102: For each question type, compare the model answers to all reference questions under the question type with the standard answers in the question data to obtain a quality index value that characterizes the response quality of the model to be evaluated for the question type.

[0045] Specifically, all reference questions in the N sets of problem data are summarized according to problem type, thereby obtaining all reference questions under each problem type; Then, according to the question type, the model answers to all reference questions under that question type are compared with the corresponding standard answers. The response instructions of the model to be evaluated for each question type are evaluated to obtain the quality index value that characterizes the response quality of the model to be evaluated for the question type.

[0046] More specifically: For each question type, the model answer to each reference question under that question type is compared with the standard answer in the corresponding question data to obtain the overlap and semantic similarity between the model answer and the standard answer for each reference question; specifically: The method for obtaining the overlap between the model answer to the reference question and the standard answer is as follows: For each of the reference questions under the aforementioned question type: The model answer and the standard answer corresponding to the reference question are segmented according to a pre-set professional terminology lexicon. In other words, professional terms related to energy storage are extracted from professional knowledge in the field of energy storage to construct a professional terminology lexicon. When segmenting the model answer or the standard answer, the professional terms in the professional terminology lexicon are used first to avoid problems such as the general word segmenter segmenting words too short, thereby improving the quality of word segmentation and preparing for subsequent weighting and synonym processing.

[0047] Based on the above, both the model's answer and the standard answer can be divided into multiple words; According to the preset mapping relationship between standard terms and synonyms, the words in the model's answer and the standard answer are replaced with the corresponding standard terms. That is, for standard terms in the energy storage field, synonyms with the same meaning for each standard term are obtained, and a mapping relationship between standard terms and synonyms is constructed to form a domain synonym library. Then, through a string matching algorithm, it is determined whether the words in the model's answer and the standard answer match the corresponding synonyms of the standard terms. If they match, these words are replaced with standard terms based on the preset mapping relationship between standard terms and synonyms; if they do not match, no replacement is required. In this way, all synonym variants can be mapped to standard terms, ensuring that the evaluation indicators focus on the semantic content itself, rather than superficial differences in expression, thus improving the semantic consistency and fairness of the evaluation.

[0048] For example, the words obtained by word segmentation of the model answer to reference question X are "A1, B, C, D, E"; the words obtained by word segmentation of the standard answer to reference question X are "A2, B, C, D, F"; and A1 and A2 both correspond to the same standard term A. After adjustment: the words obtained by word segmentation of the model answer to reference question X are "A, B, C, D, E"; and the words obtained by word segmentation of the standard answer to reference question X are "A, B, C, D, F".

[0049] For each word in the model answer and the standard answer, if the word matches a preset high-weight term, the weight of the word is increased, and the increased weight is greater than 1. Specifically, some professional terms in the field of energy storage are assigned higher weights and are denoted as high-weight terms. For each word in the model answer and the standard answer, the word is compared with preset high-weight terms to determine whether the word is a high-weight term. If the word is not a high-weight term, its weight is 1. If the word is a high-weight term, its weight is increased, for example, by increasing the weight of the word through a preset weight factor, so that the increased weight is greater than 1. Here, all high-weight terms can be set to correspond to the same weight factor, or different high-weight terms can be set to correspond to different weight factors, with the weight factor greater than 1, for example, between 1.2 and 2.5. By increasing the weight of words belonging to high-weight terms, the contribution of these words in calculating the ROUGE score is increased, thereby significantly improving the evaluation weight of the accuracy of key professional content.

[0050] The system retrieves whether key numerical items exist in the model's answer and the standard answer. If they do, the key numerical items are extracted from both the model's answer and the standard answer, and the weight of the extracted key numerical items is increased to be greater than 1. Specifically, considering the characteristics of parameters involving specific key numbers (such as voltage, temperature, and capacity threshold) in maintenance Q&A, a regular expression pattern matching method is designed. Commonly used units or percentage symbols after the key numbers are used as preset identifiers to retrieve whether preset identifiers exist in the model's answer and the standard answer. If preset identifiers exist, the preset identifiers and the numbers preceding them are extracted as key numerical items. This allows the extraction of key numerical items from the model's answer and the standard answer, such as "25℃", "3.2V", and "80%".

[0051] Then, the weights of each key numerical item are increased, for example, by using a preset weight factor to increase the weight of the key numerical item so that the increased weight of the key numerical item is greater than 1. This can be done by setting the same weight factor for all key numerical items, or by setting different weight factors for different key numerical items, with the weight factor greater than 1, for example, between 1.2 and 2.5.

[0052] Based on the above process, we can extract the vocabulary and key numerical items from the model answers and standard answers corresponding to each reference question, as well as the weights of each vocabulary and key numerical item.

[0053] Based on the vocabulary and key numerical items in the model answer and the standard answer, the ROUGE index parameters, which characterize the overlap between the model answer and the standard answer, are calculated; the ROUGE index parameters include: ROUGE-1 score, ROUGE-2 score, and ROUGE-L score.

[0054] Specifically: For the model answer and standard answer corresponding to the same reference question, the ROUGE index parameters are calculated based on the vocabulary and key numerical items in the model answer and standard answer, as well as the weights of each vocabulary and key numerical item. This yields the ROUGE index parameters for all reference questions under each question type. Then, the average of the ROUGE index parameters for all reference questions under each question type is calculated to obtain the ROUGE index parameters for each question type, including: ROUGE-1 score, ROUGE-2 score, and ROUGE-L score. The averaging of the ROUGE index parameters for all reference questions under each question type is achieved by separately averaging the ROUGE-1 score, ROUGE-2 score, and ROUGE-L score for each reference question under each question type. These average ROUGE-1 score, ROUGE-2 score, and ROUGE-L score constitute the ROUGE index parameters for each question type.

[0055] For example, the words obtained by word segmentation in the model's answer to reference question X are "A, B, C, D, E"; the standard answer corresponding to reference question X is obtained by word segmentation as "A, B, C, D, F"; where A has a weight of 1, B is a high-weight term with an adjusted weight of 1.5, C is a key numerical item with an adjusted weight of 1.5, D has a weight of 1, and E has a weight of 1 in the model's answer; When calculating the ROUGE-1 score, the statistical model's answer overlaps with the standard answer in terms of individual words "A, B, C, D", with a weight of 5; the total weight of the corresponding individual words in the model's answer is 6. So: .

[0056] When calculating the ROUGE-2 score, the model's answer corresponds to the double phrases "AB, BC, CD, DE", while the standard answer corresponds to the double phrases "AB, BC, CD, DF". The double phrases that overlap between the model's answer and the standard answer are "AB, BC, CD", with a weight of 2.5 + 3 + 2.5 = 8. The total weight of the double phrases corresponding to the model's answer is 2.5 + 3 + 2.5 + 2 = 10. So: .

[0057] When calculating the ROUGE-L score, the longest common subsequence (LCS) between the model answer and the standard answer is "A, B, C, D", with a corresponding weight of 5; the total weight of each word in the model answer is 6. So: .

[0058] The semantic similarity between the model answer to the reference question and the standard answer is obtained as follows: For each reference question under the aforementioned question type, the cosine similarity between the model answer and the standard answer of the reference question is calculated, which serves as the semantic similarity between the model answer and the standard answer. In other words, for the model answer and standard answer corresponding to the same reference question, the cosine similarity between them is calculated as the semantic similarity between them. Thus, the semantic similarity for all reference questions under each question type can be obtained. Then, the average of the semantic similarities for all reference questions under each question type is calculated to obtain the semantic similarity for each question type.

[0059] Based on the overlap and semantic similarity between the model answers and the standard answers to each of the reference questions, the quality index value corresponding to the question type is obtained.

[0060] In other words, for each question type, its corresponding ROUGE index parameters (representing overlap) and semantic similarity have been determined. Therefore, based on the ROUGE index parameters (representing overlap) and semantic similarity corresponding to the question type, as well as the preset weight parameters, the quality index value corresponding to the question type can be determined. The weight parameters are differentiated weights assigned to different indicators according to the characteristics of domain question answering.

[0061] For example, the formula for calculating the quality index value corresponding to a single problem type is: ; in, This indicates the value of the quality indicator. , , All are preset weight values. This represents the mean ROUGE-L score of all the reference questions under the given question type. This represents the mean ROUGE-1 score of all the reference questions under the given question type. This represents the mean ROUGE-2 score of all the reference questions under the given question type. This represents the mean semantic similarity between all the reference questions and the standard answer under the given question type.

[0062] For example, in the field of energy storage operation and maintenance, =0.4 (because it captures sentence-level semantic structure and logical coherence, it is crucial for the accuracy and interpretability of operation and maintenance decisions). = =0.2 (used to balance the importance of word-level and phrase-level matching). =0.2 (as a supplementary weighting factor to enhance the semantic evaluation dimension).

[0063] Step 103: Based on the reference score values ​​of the model answers to the reference questions under each of the aforementioned question types, determine the quality score value corresponding to each of the aforementioned question types.

[0064] Specifically, the reference score values ​​of the model answers to the reference questions under each of the aforementioned question types are obtained by domain experts through manual evaluation of the model answers to the reference questions and input into the electronic device. Thus, the reference score values ​​corresponding to all reference questions under each question type can be obtained. Then, for each question type, the average of the reference score values ​​corresponding to all reference questions under that question type is calculated as the quality score value corresponding to that question type. Thus, the quality score value corresponding to each question type can be obtained.

[0065] For example, the model's answer to the reference question is scored on a 0-5 scale, with domain experts making a comprehensive evaluation based on three core dimensions. Specifically, for questions involving question generalization, cross-knowledge point association, and reasoning, the standard answer may not be entirely consistent with the original question due to the expanded scope; therefore, the relevance of the format to the original question will be considered during scoring. For questions involving question specification, marginal cases, and negative queries, only a partial answer is required; redundant content outside the question's scope will be considered a deduction. The specific scoring criteria are shown in Table 2 below.

[0066] Table 2:

[0067] The manual evaluation process consists of four rigorous stages: 1. Preparation Phase: Assemble an evaluation team consisting of at least three experts with backgrounds in electrochemical energy storage. Conduct training on standardized scoring criteria and perform pre-evaluation using a calibration set to ensure consistent understanding of the standards among the evaluators.

[0068] 2. Initial Evaluation Stage: Each evaluation expert independently scores the model's answer to each reference question. The evaluation will focus on: the accuracy of factual content, the accuracy of technical terminology, the completeness of the logical reasoning chain, and the ability to handle negative queries and marginal cases.

[0069] 3. Arbitration and Scoring Stage: For model answers to the same reference question, if the scores given by different evaluators differ by more than 2 points, the evaluation team leader will organize a discussion to review the answers again and reach a consensus to determine the final score. If the score difference is less than 2 points, the average of the scores from different evaluators will be taken as the final score. Furthermore, the final score needs to be normalized by dividing it by 5, which will serve as a standardized reference score for the model answers to each reference question.

[0070] 4. Results Evaluation Stage: Descriptive statistics are performed on the final scores, and typical good / bad answer cases are recorded to provide qualitative reference for model optimization.

[0071] Step 104: Based on the quality index values ​​and quality score values ​​corresponding to each of the problem types, determine the performance index parameters of the model to be evaluated.

[0072] Specifically, for each of the problem types, based on the fusion coefficient corresponding to the problem type, the quality indicator value and the quality score value corresponding to the problem type are fused to obtain the fusion score corresponding to the problem type; Based on the aforementioned process, the quality index value and quality score value corresponding to each problem type were determined. Each problem type also has a corresponding fusion coefficient, allowing for the weighted calculation of the fusion score for each problem type. For example, for the i-th problem type, its corresponding fusion score is... The formula for calculation is: ; in, Let be the fusion coefficient corresponding to the i-th problem type. Let i be the quality index value corresponding to the i-th problem type. This represents the quality score corresponding to the i-th problem type; To balance the fusion coefficients of the two types of scores, different question types correspond to different fusion coefficients to evaluate the performance of different question levels; and This is the value after standard normalization.

[0073] Based on the weight coefficients corresponding to each problem type and the fusion score, the comprehensive performance index of the model to be evaluated is calculated. Specifically, each problem type is assigned a corresponding weight coefficient, as shown in Table 1. Therefore, the fusion scores corresponding to all problem types can be weighted and summed to obtain the comprehensive performance index of the model to be evaluated. For example, the formula for calculating the comprehensive performance index of the model to be evaluated is: Overall performance index (1) in, This represents the weight coefficient corresponding to the i-th problem type, and k represents the total number of problem types. This represents the fusion score corresponding to the i-th problem type.

[0074] The performance index parameters of the model to be evaluated include: the comprehensive performance index, which quantifies the performance of the model to be evaluated in the field of energy storage system operation and maintenance, and provides a clear indicator for measuring the performance of different models.

[0075] In one example, multiple problem types can be divided into multiple levels, and each level includes multiple problem types; for example, the multiple problem types can be divided into the following four levels, and optimization strategies can be provided for typical problems at each level: Semantic layer: manifested as comprehension bias and semantic inconsistency, used to examine the degree to which the model under evaluation understands the meaning of the question; optimization strategies include enhancing the semantic matching accuracy in the Retrieval Augmentation (RAG) mechanism and improving contextual understanding capabilities.

[0076] Lexical layer: This is characterized by incorrect terminology usage and non-standard expressions. It is used to examine the model under evaluation's mastery of domain terminology and its compatibility with lexical errors. Optimization strategies include expanding the domain dictionary and enhancing terminology constraints in the prompting process.

[0077] Knowledge layer: characterized by knowledge gaps and insufficient timeliness, used to examine the model under evaluation's ability to understand and apply domain knowledge; optimization strategies include regularly updating the knowledge base and building a dynamic knowledge fusion mechanism.

[0078] The logical layer is characterized by incoherent reasoning and broken causal chains. It is used to examine the ability of the model under evaluation to reason about and integrate knowledge from within and outside the domain. Optimization strategies include introducing the Chain of Thought (CoT) hinting engineering to enhance the constraints on reasoning paths.

[0079] For example, for Figure 1 The eight problem types shown are divided into four levels as described in Table 3 below. Table 3 also shows the fusion coefficients corresponding to each problem type.

[0080] Table 3:

[0081] For each level, based on the weight coefficients corresponding to each of the problem types included in the level and the fusion score, the level performance index of the model to be evaluated at that level is calculated; that is, for each level, the target problem types included in that level are obtained, and based on the weight coefficients corresponding to each target problem type at that level, the fusion scores of all target problem types are weighted and summed to obtain the level performance index of that level; thus, the level performance index of the model to be evaluated at each level can be obtained; for example, for a specified level, the formula for calculating its level performance index is: Hierarchical performance index (2) in, This represents the weight coefficient corresponding to the i-th target problem type under the specified level, where r represents the total number of problem types included in the specified level. This represents the fusion score corresponding to the i-th target problem type under the specified level.

[0082] The performance metrics parameters of the model to be evaluated also include: the hierarchical performance index of the model under all levels; the hierarchical performance index focuses on the performance of each problem level scenario, especially the hierarchical scenarios where the index value is lower than that of other models, to provide a reference for subsequent optimization. The optimization strategies provided can be referred to the above content, and will not be repeated here.

[0083] Based on the above process, the quality index values ​​for each question type are calculated by the algorithm, which can objectively quantify the model's answer and the standard answer in dimensions such as lexical overlap and semantic similarity. The quality score for each question type is given manually by domain experts who subjectively score the model's answer for content correctness, information accuracy, and structural degree. Finally, the performance index parameters of the model to be evaluated are integrated with both algorithmic and manual evaluation results, and through weighted calculation and correlation analysis, a comprehensive profile of the model's performance is formed. This dual-track evaluation mechanism overcomes the shortcomings of purely automated evaluation, which lacks semantic understanding depth, and reduces the subjectivity and labor costs of purely manual evaluation, achieving an effective balance between evaluation efficiency and accuracy.

[0084] In one example, after obtaining the quality index value representing the response quality of the model to be evaluated for the question type, the performance index parameters of the model to be evaluated can be determined directly based on the quality index value corresponding to each question type. For example, the comprehensive performance index of the model to be evaluated can be obtained by weighted summation based on the weight coefficient corresponding to each question type and the quality index value. The calculation formula can refer to the calculation formula (1) of the comprehensive performance index mentioned above. The difference is that the fusion score is not used, but the quality index value corresponding to the question type.

[0085] For each level, the hierarchical performance index of the model to be evaluated under the level is calculated based on the weight coefficients corresponding to each problem type included in the level and the quality index value. The calculation formula can refer to the calculation formula (2) of the hierarchical performance index above. The difference is that the fusion score is not used, but the quality index value corresponding to the problem type is used.

[0086] The performance index parameters include: the overall performance index of the model to be evaluated and the hierarchical performance index of the model to be evaluated at all levels.

[0087] The following example illustrates the model evaluation process described above: During the data preparation phase, 30 question-and-answer pairs were collected and organized from standards and specifications, equipment manuals, historical operation and maintenance reports, and expert experience in the field of electrochemical energy storage. Each question-and-answer pair included the original question and the corresponding standard answer. Subsequently, for different models to be evaluated, 30 sets of question data corresponding to the 30 original questions were constructed. Then, the comprehensive performance index and the hierarchical performance index of each level corresponding to each model to be evaluated can be obtained. The levels include: semantic layer, lexical layer, knowledge layer, and logical layer. Please refer to Figure 3 ,by Figure 2 Taking a set of problem data as an example, by performing the model evaluation process of this embodiment, we can obtain the quality index value, reference score value, and fusion score corresponding to each problem type; as well as the comprehensive performance index and hierarchical performance index.

[0088] For example, the models to be evaluated include four domain-fine-tuned small models and five full-scale large models, namely: IIama3-vision-11b, IIama3-1-8b, qwen2.5-14b, qwen2.5-7b, qwen3-max, deepseek-v3.1, deepseek-r1, kimi2-thingking, and glm4-6; the first four are domain-fine-tuned small models; the latter five are full-scale large models; please refer to [reference needed]. Figure 4 It displays the overall performance index of all models to be evaluated; please refer to... Figure 5 It shows the hierarchical performance indices of all models to be evaluated at the semantic, lexical, knowledge, and logical layers.

[0089] based on Figure 4 and Figure 5 It is evident that at the semantic and lexical levels, the domain-fine-tuned small model outperforms the full-scale large model; at the knowledge and logic levels, the full-scale large model performs better.

[0090] The model evaluation method for intelligent operation and maintenance of electrochemical energy storage systems proposed in this invention significantly improves the professionalism and accuracy of the evaluation. The constructed comprehensive performance index enables objective quantification of the evaluation results, significantly improving the comparability of the results and providing a scientific basis for the selection, optimization, and deployment of large-scale language models.

[0091] This embodiment systematically addresses core issues such as weak domain specialization, limited scoring dimensions, and difficulty in quantifying evaluation results by addressing the entire process from data construction and test design to comprehensive scoring. Its basic principles are: by constructing a professional terminology database and synonym mapping table for the electrochemical energy storage field, a unified representation and weighted enhancement of core concepts in the field are achieved, resolving misjudgments caused by inconsistent terminology in traditional assessments; by designing evaluation strategies for eight typical problems and dynamic weight configurations, the system covers various operation and maintenance scenarios from basic understanding to complex reasoning, improving the comprehensiveness and relevance of the evaluation; by introducing a dual-track fusion mechanism of automated and manual evaluation, a comprehensive performance index is constructed, achieving objective quantification and scientific integration of evaluation results; and by hierarchically analyzing the evaluation results, a model optimization strategy is proposed.

[0092] In other words, by systematically organizing various information such as standard documents, papers, patents, and operation and maintenance reports in the field of electrochemical energy storage, this invention constructs a professional terminology database in this field. Based on this database, it also builds a thesaurus and a terminology weight database, significantly improving the domain-specificity of the assessment task and achieving a key upgrade from "surface matching" to "semantic depth measurement." Based on a question-and-answer mechanism for different types of perturbations, the assessment method can simulate real operation and maintenance question-and-answer scenarios, providing a reference for enhancing the robustness of large models and avoiding significant performance degradation under complex semantic conditions, overcoming the problem of single indicator structures in traditional methods. Based on a comprehensive performance index model combining automation and manual work, the performance of large models in handling professional tasks is quantified. This model can be used for performance comparison between different large models and provides reference and guidance for the continuous optimization of large models. Through the above modular collaboration, this invention forms a comprehensive assessment system covering professional assessment, automatic assessment, manual assessment, and quantitative indicators, achieving a breakthrough in the assessment of large energy storage operation and maintenance models from simple single indicators to multi-dimensional comprehensive indicators.

[0093] Through the aforementioned mechanism, this invention achieves a complete innovation across the entire process, from "single indicator" to "multi-dimensional fusion" and then to "closed-loop optimization." Compared to existing technologies, its advantages are mainly reflected in three aspects: First, it enhances the domain adaptability and professionalism of the assessment by significantly improving assessment accuracy through terminology weighting, synonym mapping, and key number matching; second, it enhances the comprehensiveness and robustness of the assessment by covering complex operation and maintenance scenarios through multi-type problem classification and differentiated strategies; and third, it achieves quantifiable and operationalizable assessment results by providing reference directions for model optimization through dual-track fusion and closed-loop mechanisms.

[0094] In terms of system, a collaborative system has been formed, from automatic evaluation and manual evaluation to dual-track comprehensive evaluation. In terms of function, a deep integration of professionalism and robustness has been achieved, providing a highly reliable, interpretable and scalable intelligent technical solution for real-world scenario evaluation of the intelligent operation and maintenance model of electrochemical energy storage systems.

[0095] By constructing a domain-adaptive evaluation index system, a terminology weight enhancement mechanism, a multi-type problem evaluation strategy, and a closed-loop optimization path, a comprehensive and efficient evaluation of the energy storage operation and maintenance large language model can be achieved. Stable and repeatable performance comparisons can be performed between different models, effectively identifying specific shortcomings of the model in terms of professional terminology use and logical reasoning, providing a reference direction for model optimization, and possessing good engineering applicability and promotion value.

[0096] The second embodiment of the present invention relates to an electronic device, such as a server, or a common computer device, such as a desktop computer, a laptop computer, etc.

[0097] The electronic device includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor to enable the at least one processor to perform the model evaluation method in the first embodiment.

[0098] It should be noted that the memory may include random access memory (RAM) and may also include non-volatile memory, such as at least one disk storage device. Similarly, the processor can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components.

[0099] Since the first embodiment corresponds to this embodiment, this embodiment can be implemented in conjunction with the first embodiment. The relevant technical details mentioned in the first embodiment remain valid in this embodiment, and the technical effects achievable in the first embodiment can also be achieved in this embodiment. To reduce repetition, they will not be repeated here. Correspondingly, the relevant technical details mentioned in this embodiment can also be applied to the first embodiment.

[0100] The third embodiment of the present invention relates to a computer-readable storage medium, which is a non-volatile storage medium or a non-transient storage medium, on which a computer program is stored, characterized in that the computer program is executed by a processor to perform the model evaluation method in the first embodiment.

[0101] The preferred embodiments of the present invention have been described in detail above, but it should be understood that, if necessary, aspects of the embodiments can be modified to utilize aspects, features, and concepts from various patents, applications, and publications to provide other embodiments.

[0102] In light of the detailed description above, these and other changes can be made to the embodiments. Generally, the terminology used in the claims should not be considered limited to the specific embodiments disclosed in the specification and claims, but should be understood to include all possible embodiments together with the full scope of equivalents enjoyed by these claims.

Claims

1. A model evaluation method, characterized in that, include: Obtain N sets of question data corresponding to N original questions, where N≥1. Each set of question data includes: reference questions corresponding to various question types, model answers obtained after each reference question is input into the model to be evaluated, and standard answers; the reference questions for each of the non-original question types are obtained by transforming the original questions according to the question type. For each question type, the model answers to all reference questions under that question type are compared with the standard answers in the question data to obtain a quality index value that characterizes the response quality of the model to be evaluated for that question type. Based on the quality index values ​​corresponding to each of the aforementioned problem types, the performance index parameters of the model to be evaluated are determined.

2. The model evaluation method according to claim 1, characterized in that, The method further includes: Based on the reference score values ​​of the model answers to the reference questions under each of the aforementioned question types, determine the quality score value corresponding to each of the aforementioned question types; Based on the quality index values ​​corresponding to each of the aforementioned problem types, the performance index parameters of the model to be evaluated are determined, including: Based on the quality index values ​​and quality score values ​​corresponding to each of the aforementioned problem types, the performance index parameters of the model to be evaluated are determined.

3. The model evaluation method according to claim 2, characterized in that, Based on the quality index values ​​and quality score values ​​corresponding to each of the aforementioned problem types, the performance index values ​​of the model to be evaluated are determined, including: For each of the aforementioned problem types, based on the fusion coefficient corresponding to the problem type, the quality indicator value and the quality score value corresponding to the problem type are fused to obtain the fusion score corresponding to the problem type; Based on the weight coefficients corresponding to each of the aforementioned problem types and the fusion score, the comprehensive performance index of the model to be evaluated is calculated. The performance index parameters include: the comprehensive performance index.

4. The model evaluation method according to claim 3, characterized in that, The various problem types are divided into multiple levels, and each level includes multiple of the aforementioned problem types; For each level, the hierarchical performance index of the model to be evaluated at that level is calculated based on the weight coefficients corresponding to each problem type included in the level and the fusion score. The performance index parameters also include: the hierarchical performance index of the model to be evaluated at all levels.

5. The model evaluation method according to claim 1, characterized in that, For each question type, a quality index value representing the response quality of the model to be evaluated for the corresponding question type is obtained by comparing the model's answers to all reference questions under that question type with the standard answers in the question data. This includes: For each of the aforementioned problem types: The model answer to each reference question under the question type is compared with the standard answer in the question data to obtain the overlap and semantic similarity between the model answer and the standard answer of each reference question; Based on the overlap and semantic similarity between the model answers and the standard answers to each of the reference questions, the quality index value corresponding to the question type is obtained.

6. The model evaluation method according to claim 5, characterized in that, The method for obtaining the overlap between the model answer to the reference question and the standard answer is as follows: For each of the reference questions under the aforementioned question type: The model answer and the standard answer are segmented according to a preset professional terminology database, and the words in the model answer and the standard answer are replaced with the corresponding standard terms according to the preset mapping relationship between standard terms and synonyms. For each word in the model's answer and the standard answer, if the word matches a preset high-weight term, the weight of the word is increased, and the increased weight of the word is greater than 1. Search for whether key numerical items exist in the model's answer and the standard answer; If they exist, extract the key numerical items from the model's answer and the standard answer, and increase the weight of the extracted key numerical items. The increased weight of the key numerical items is greater than 1. Based on the vocabulary and key numerical items in the model answer and the standard answer, calculate the ROUGE index parameter, which characterizes the overlap between the model answer and the standard answer; The ROUGE index parameters include: ROUGE-1 score, ROUGE-2 score, and ROUGE-L score.

7. The model evaluation method according to claim 5, characterized in that, The semantic similarity between the model answer to the reference question and the standard answer is obtained as follows: For each reference question under the aforementioned question type, the cosine similarity between the model answer to the reference question and the standard answer is calculated, which serves as the semantic similarity between the model answer to the reference question and the standard answer.

8. The model evaluation method according to claim 6, characterized in that, The formula for calculating the quality index value corresponding to the problem type is: ； in, This indicates the value of the quality indicator. , , All are preset weight values. This represents the mean ROUGE-L score of all the reference questions under the given question type. This represents the mean ROUGE-1 score of all the reference questions under the given question type. This represents the mean ROUGE-2 score of all the reference questions under the given question type. This represents the mean semantic similarity between all the reference questions and the standard answer under the given question type.

9. The model evaluation method according to any one of claims 1 to 4, characterized in that, The non-original question types include at least one of the following: semantically invariant rewriting type, terminology substitution and abbreviation usage type, typo addition and grammatical noise type, question generalization type, question concretization type, cross-knowledge point association and reasoning type, marginal case and negative query type.

10. The model evaluation method according to claim 9, characterized in that, The various question types are divided into four levels: semantic level, lexical level, knowledge level, and logical level. The semantic layer includes: the original type and the semantically invariant rewritten type; The vocabulary layer includes: the terminology substitution and abbreviation usage types and the addition of typos and grammatical noise types; The knowledge layer includes: the problem generalization type and the problem specificity type; The logical layer includes: the cross-knowledge point association and reasoning type and the edge case and negative query type.

11. An electronic device, characterized in that, include: At least one processor; And a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform the model evaluation method as described in any one of claims 1 to 10.

12. A computer-readable storage medium, said computer-readable storage medium being a non-volatile storage medium or a non-transient storage medium, having stored thereon a computer program, characterized in that, The computer program is executed by the processor to perform the model evaluation method as described in any one of claims 1 to 10.

Images