Evaluation method and device for model reasoning effect on power system

Abstract

The invention discloses an evaluation method and device for a model reasoning effect on an electric power system, and relates to the technical field of electric power field rescue. According to the main technical scheme, for a large language model for performing reasoning operation on power data on a power system, a plurality of evaluation indexes are constructed for the model, and at least one evaluation index is randomly selected to be combined to obtain a target evaluation index set; an evaluation dimension is created for the target evaluation index set in combination with an actual application scene, so that test questions are constructed for the target evaluation index set on the basis of the evaluation dimension, and each test question has a preset correct answer; the method comprises the following steps: inputting target power data required by test questions into a model for processing, outputting reasoning result data given by the model, comparing the reasoning result data with a preset correct answer of each test question, and evaluating to obtain an evaluation result of a model reasoning effect.

Description

Technical Field

[0001] The present application relates to the technical field of power systems, and in particular to a method and device for evaluating the effect of model reasoning on a power system. Background Art

[0002] Large Language Model (LLM) is an important part of the field of artificial intelligence and plays a key role in natural language processing. In the field of power system technology, LLM can process and analyze massive power data, provide decision support for power grid operation, optimize energy distribution, predict and prevent potential failures, etc., thereby improving the reliability and efficiency of the power grid.

[0003] However, the complexity of the power system and the extremely high requirements for safety make LLM face unique challenges. How to measure the reasoning effect of LLM in practical applications is an urgent problem to be solved. Summary of the invention

[0004] In view of this, the present application provides a method and device for UAV flight path planning. The main purpose is to utilize the logical reasoning test implemented by constructing test questions, which can be combined with the actual application scenarios of the model to effectively evaluate the reasoning ability of the model in terms of logic and interpretability using logical reasoning tests, thereby providing a solution for effectively evaluating the reasoning effect of the model.

[0005] In order to achieve the above objectives, this application mainly provides the following technical solutions:

[0006] The first aspect of the present application provides a method for evaluating the effect of model reasoning on a power system, the method comprising:

[0007] Constructing a plurality of evaluation indicators for the model, each of the evaluation indicators corresponds to a first evaluation dimension, and there is no information overlap between the first evaluation dimensions corresponding to different evaluation indicators; the model is a large language model that performs reasoning operations on power data on the power system;

[0008] Selecting at least one evaluation indicator from the multiple evaluation indicators to form a target evaluation indicator set, wherein the target evaluation indicator set corresponds to at least one second evaluation dimension, and the second evaluation dimension is an evaluation dimension created in combination with an actual application scenario;

[0009] Constructing test questions for the target evaluation indicator set, the test questions are used to test the reasoning effect of the model on the second evaluation dimension, and each of the test questions corresponds to a preset correct answer;

[0010] Obtaining target power data required for the test question;

[0011] Processing the target power data using the model to output inference result data;

[0012] The inference result data is evaluated by using the preset correct answer corresponding to the test question to output an evaluation result of the inference effect of the model.

[0013] A second aspect of the present application provides a device for evaluating the effect of model reasoning on a power system, the device comprising:

[0014] A first construction unit is used to construct a plurality of evaluation indicators for the model, each of the evaluation indicators corresponds to a first evaluation dimension, and there is no information overlap between the first evaluation dimensions corresponding to different evaluation indicators; the model is a large language model that performs reasoning operations on power data on the power system;

[0015] A selection unit, configured to select at least one evaluation indicator from the plurality of evaluation indicators to form a target evaluation indicator set, wherein the target evaluation indicator set corresponds to at least one second evaluation dimension, and the second evaluation dimension is an evaluation dimension created in combination with an actual application scenario;

[0016] A second construction unit is used to construct a test question for the target evaluation indicator set, wherein the test question is used to test the reasoning effect of the model on the second evaluation dimension, and each of the test questions corresponds to a preset correct answer;

[0017] A first acquisition unit, used for acquiring target power data required for the test question;

[0018] A processing unit, used to process the target power data using the model to output inference result data;

[0019] An evaluation unit is used to evaluate the inference result data by using the preset correct answer corresponding to the test question to output an evaluation result of the inference effect of the model.

[0020] A third aspect of the present application provides a computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the method for evaluating the effect of model reasoning on a power system as described above is implemented.

[0021] A fourth aspect of the present application provides an electronic device, comprising: a memory, a processor, and a computer program stored on the memory and executable on the processor, wherein the computer program, when executed by the processor, implements the above-described method for evaluating the effect of model reasoning on a power system.

[0022] By means of the above technical solution, the technical solution provided by this application has at least the following advantages:

[0023] The present application provides an evaluation method and device for the effect of model reasoning on an electric power system. For a large language model that performs reasoning operations on electric power data on an electric power system, first, multiple evaluation indicators are constructed for the model. Since the use of a single evaluation indicator or only the superposition of evaluation indicators to evaluate the model is not comprehensive and objective, the present application combines the evaluation indicators to obtain a target evaluation indicator set; then, an evaluation dimension is created for the target evaluation indicator set in combination with an actual application scenario, thereby constructing test questions for the target evaluation indicator set based on such an evaluation dimension, and each test question has a preset correct answer; the target electric power data required for the test question is input into the model for processing, and the reasoning result data given by the model is output, and the reasoning result data is compared with the preset correct answer of each test question and evaluated, thereby obtaining an evaluation result of the model reasoning effect.

[0024] As mentioned above, the logical reasoning test implemented by constructing test questions in this application can be combined with the actual application scenarios of the model, and the logical reasoning test can be used to effectively evaluate the reasoning ability of the model in terms of logic and explanation, thereby providing a solution for effectively evaluating the reasoning effect of the model.

[0025] The above description is only an overview of the technical solution of the present application. In order to more clearly understand the technical means of the present application, it can be implemented in accordance with the contents of the specification. In order to make the above and other purposes, features and advantages of the present application more obvious and easy to understand, the specific implementation methods of the present application are listed below. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] Various other advantages and benefits will become apparent to those of ordinary skill in the art by reading the detailed description of the preferred embodiments below. The accompanying drawings are only for the purpose of illustrating the preferred embodiments and are not to be considered as limiting the present application. Also, the same reference symbols are used throughout the accompanying drawings to represent the same components. In the accompanying drawings:

[0027] Figure 1 A flow chart of a method for evaluating the effect of model reasoning on a power system provided in an embodiment of the present application;

[0028] Figure 2 A block diagram of a device for evaluating the effect of model reasoning on a power system provided in an embodiment of the present application;

[0029] Figure 3 A block diagram of the composition of another device for evaluating the effect of model reasoning on a power system provided in an embodiment of the present application. DETAILED DESCRIPTION

[0030] The exemplary embodiments of the present application will be described in more detail below with reference to the accompanying drawings. Although the exemplary embodiments of the present application are shown in the accompanying drawings, it should be understood that the present application can be implemented in various forms and should not be limited by the embodiments set forth herein. On the contrary, these embodiments are provided in order to enable a more thorough understanding of the present application and to fully convey the scope of the present application to those skilled in the art.

[0031] The present application embodiment provides a method for evaluating the effect of model reasoning on a power system, such as Figure 1 As shown, the present application embodiment provides the following specific steps:

[0032] 101. Construct multiple evaluation indicators for the model, each evaluation indicator corresponds to a first evaluation dimension, and there is no information overlap between the first evaluation dimensions corresponding to different evaluation indicators; the model is a large language model that performs reasoning operations on power data in the power system.

[0033] In the embodiment of the present application, a large language model (LLM) is used to process power data. Power data refers to various data related to power supply, use and management generated, collected and processed by power facilities. It mainly includes power generation data, power consumption data, quality data, equipment data, transaction data, etc. These data are collected and processed by the acquisition and transmission system to provide a basis for monitoring, control and optimization of the power grid.

[0034] For example, power data can generally have, but is not limited to, the following characteristics:

[0035] (1) Real-time: Real-time updating of power data is crucial to the safe and stable operation of the power grid.

[0036] (2) Diversity: Power data comes from various links such as power generation, transmission, transformation, distribution, consumption and dispatching, and the data types are diverse.

[0037] (3) High value density: Power data contains rich information and is of great value for the optimization and management of the power system.

[0038] (4) Accuracy: The accuracy of power data is directly related to the operating efficiency and safety of the power grid.

[0039] In addition, the application of the large language model in power processing may include but is not limited to the following:

[0040] (1) Data cleaning and integration: LLM has strong text understanding and reasoning capabilities, which can automatically identify and correct errors and outliers in power data. Through contextual learning, LLM can understand and integrate information from different data sources to form a unified and accurate data set.

[0041] (2) Load forecasting and power generation output forecasting: LLM can make more accurate load forecasting and power generation output forecasting based on multi-dimensional information such as historical electricity data, meteorological information, and social and economic activities. In terms of load forecasting, LLM's small sample learning ability helps solve the problem of data scarcity in scenarios with new users and low-frequency electricity consumption. For medium- and long-term load forecasting, LLM can solve the problems of feature selection and non-power consumption data generation. In terms of power generation output forecasting, LLM can consider multiple factors (such as weather changes, equipment status, etc.) to improve the accuracy of the forecast.

[0042] (3) Power system planning: In terms of uncertainty simulation, LLM can generate intuitive simulation results based on knowledge, save sampling calculation process and achieve approximate results, and realize power system risk assessment based on uncertainty simulation results. In terms of planning scenario generation, LLM can automatically generate typical planning scenarios through natural language descriptions and adjust and optimize specific scenarios.

[0043] In terms of planning scheme optimization, LLM can be applied to the automatic interpretation of optimization schemes to generate natural language descriptions of the optimal schemes. At the same time, LLM can quickly generate corresponding planning models and constraints to reduce the workload of power system planning.

[0044] (4) Power system dispatching: LLM can assist in power system dispatching from multiple aspects, including dispatcher experience extraction, dispatch modeling, dispatch decision-making, operation execution, and power system situation awareness. The generalization and natural language processing capabilities of LLM can help dispatchers quickly learn dispatching knowledge, improve dispatching efficiency and accuracy, and thus achieve economic, safe, and low-carbon operation of the power system.

[0045] (5) Fault diagnosis and recovery: LLM can quickly identify the fault type and location based on fault data and professional knowledge base, and generate corresponding fault handling solutions. In the diagnosis of transmission and distribution system faults, the versatility of LLM is expected to achieve the simultaneous diagnosis capability of multiple types of faults. In terms of system recovery, LLM can conduct a comprehensive multi-dimensional security assessment of the recovery plan and quickly generate a recovery plan.

[0046] (6) Power market research: In terms of power market modeling, LLM can build more accurate transaction behavior models with few or no samples. In terms of power market decision-making, LLM has the ability to generate strategies autonomously to cope with the rapidly changing market environment. In terms of power market mechanism design, LLM can be combined with mechanism design theory to simulate the behavior of market players, thereby better ensuring the rationality and practicality of power market mechanism design.

[0047] In the field of power system technology, LLM can process and analyze massive power data, provide decision support for power grid operation, optimize energy distribution, predict and prevent potential failures, etc., thereby improving the reliability and efficiency of the power grid.

[0048] In response to the characteristics of power data and the application of LLM in power processing, and taking into account how to evaluate the reasoning effect of the model, the embodiment of the present application preliminarily constructs multiple evaluation indicators. For example, a preset input template can be used for construction. Each evaluation indicator corresponds to an evaluation dimension, and there is no information overlap in each evaluation dimension to avoid information overlap between evaluation indicators and evaluation redundancy.

[0049] It should be noted here that these are only preliminary evaluation indicators and preliminary evaluation dimensions obtained using the preliminary evaluation indicators. In order to distinguish and refer to the evaluation dimensions that are subsequently constructed in depth, the embodiments of this application use the words "first" and "second" for distinction.

[0050] For example, the embodiments of the present application may, but are not limited to, construct the following evaluation indicators as preliminary evaluation indicators, which may constitute a comprehensive evaluation indicator system.

[0051] Accuracy: refers to the accuracy of the model in the reasoning task, that is, for a given reasoning problem, the degree of consistency between the answer or reasoning result given by the model and the standard answer or reasoning result. Specific detailed indicators include understanding accuracy, error rate, etc. For fixed-answer questions, the accuracy is evaluated by comparing the matching degree between the model output and the standard answer using a pre-labeled test data set. For open-ended questions, the accuracy score is obtained through expert evaluation of the model output.

[0052] Logic: refers to the logical rigor of the model during the reasoning process, that is, whether the reasoning process conforms to the logical rules and whether there is a reasonable reasoning chain and logical structure. Specific detailed indicators include logical consistency score, reasoning chain integrity, etc. Logical analysis tools can be used to detect the reasoning chain and logical structure of the model output, and the logic can be judged through the expert review of the model reasoning process.

[0053] Consistency: refers to the stability and consistency of the model when reasoning in different situations, that is, whether the model's reasoning results are consistent for similar reasoning problems. Specific detailed indicators include consistency score and coefficient of variation. Test the same or similar problems multiple times at different times and in different environments, compare the consistency of their reasoning results, and observe the stability of the model's reasoning results by changing the problem background or details.

[0054] Coverage: refers to the breadth of different fields, topics, and scenarios that the model can cover in reasoning tasks, that is, the model's performance in various reasoning scenarios. Specific detailed indicators include field coverage, scenario adaptability score, etc. The model is tested in different professional fields and topics to evaluate its performance, and the model's reasoning ability is tested by simulating different application scenarios.

[0055] Interpretability: refers to whether the model can clearly explain the reasoning process and results of the reasoning process, so that users can understand and accept the reasoning results. Specific indicators include explanation clarity score, user satisfaction, etc. Use model explanation tools (such as LIME, SHAP) to analyze the interpretability of model outputs and score them based on the user's understanding and acceptance of the model explanation process.

[0056] Transferability: refers to the ability of a model to transfer between different reasoning tasks or fields, that is, whether the performance of the model on one reasoning task can be transferred and applied to other reasoning tasks. Specific detailed indicators include transfer success rate, adaptability score, etc. In different tasks and fields, the model is tested for transfer and its performance is evaluated. Through fine-tuning or a small amount of training data, the degree of improvement of the model's performance on new tasks is observed.

[0057] Efficiency: refers to the speed and resource utilization efficiency of the model when performing reasoning tasks, that is, the reasoning results can be obtained within a reasonable time, and the computing resources occupied by the processing process are moderate. Specific detailed indicators include response time, resource occupancy rate, etc. By actually running the model, measuring its processing speed and resource consumption, and analyzing the computing resource usage of the model during the reasoning process.

[0058] 102. Select at least one evaluation indicator from multiple evaluation indicators to form a target evaluation indicator set, where the target evaluation indicator set corresponds to at least one second evaluation dimension, and the second evaluation dimension is an evaluation dimension created in combination with an actual application scenario.

[0059] As for the multiple evaluation indicators constructed in 101 above, each evaluation indicator has a different purpose, and different evaluation methods can be used, as shown in Table 1 below: "Evaluation indicators", corresponding descriptions and evaluation methods.

[0060] Table 1:

[0061]

[0062]

[0063] In the above, each evaluation indicator evaluates the model from different evaluation dimensions, and the comprehensive evaluation results are equal to the sum of the evaluation results of each evaluation indicator. Based on the different divisions of the evaluation indicators, this is equivalent to evaluating the model at a certain granularity, with a good evaluation breadth, but the depth of the evaluation is still insufficient. Therefore, the embodiment of the present application is to recreate a new evaluation dimension based on one or more evaluation indicators and actual application scenarios, obtain multiple new evaluation dimensions corresponding to a target evaluation indicator set, and then evaluate the model on the new evaluation dimension.

[0064] For example, taking the evaluation indicators "logic" and "consistency" as examples, "logic" describes whether the reasoning process conforms to the logical rules, which can be measured by the rigor of the logical rules; "consistency" describes whether the reasoning results in similar situations are consistent. The embodiment of the present application integrates these two evaluation indicators and selects an application scenario to construct an evaluation dimension. For example, combined with the actual application scenario of "electricity market", the reasoning test is "In the electricity market of an electric company, if the electricity price rises by 10%, the relationship between electricity demand and electricity supply", and what is the reasoning effect of taking into account "logic" and "consistency".

[0065] Therefore, the new evaluation dimension is limited to: "actual application scenario" + "logic" + "consistency". And the reasoning effect of the model is tested on the new evaluation dimension, that is, the reasoning effect of the model is evaluated.

[0066] 103. Construct test questions for the target evaluation indicator set. The test questions are used to test the reasoning effect of the model on the second evaluation dimension. Each test question corresponds to a preset correct answer.

[0067] As exemplified in 102, test questions are constructed on a new evaluation dimension corresponding to the target evaluation indicator set.

[0068] For example: construct a test question; the question stem is "In the electricity market of an electric power company, if the electricity price rises by 10%, which of the following situations is most likely to occur?"; and the four options ABCD and the preset correct answer of this question stem

[0069] Option A: Electricity demand falls by 5%, but electricity supply increases by 3%;

[0070] Option B: Electricity demand increases by 8%, but electricity supply decreases by 6%;

[0071] Option C: Electricity demand increases by 12%, but electricity supply increases by 10%;

[0072] Option D: Electricity demand falls by 3%, but electricity supply falls by 5%;

[0073] The default correct answer to this question is "B".

[0074] The test questions constructed above are questions that require logical reasoning ability. They are input into the model to test the reasoning effect of the model by using whether the reasoning results output by the model are consistent with the preset correct answers, thereby evaluating the reasoning ability of the model in terms of logic and explanatory power using logical reasoning tests.

[0075] 104. Obtain the target power data required for the test questions.

[0076] 105. Use the model to process the target power data to output inference result data.

[0077] 106. The inference result data is evaluated by using the preset correct answers corresponding to the test questions to output the evaluation results of the inference effect of the model.

[0078] As shown in 104-106, based on a test question as exemplified in 103, the embodiment of the present application obtains the relevant power data corresponding to the stem of the test question, and inputs four options into the model, so that the model processes these data information and outputs the inference result. When the inference result is consistent with the preset correct answer of the test question, it indicates that the model has a good inference effect on this question.

[0079] The test questions listed in 103 are only exemplary examples of the embodiments of the present application. In order to better test the reasoning effect of the model, the embodiments of the present application can create multiple new evaluation dimensions in combination with different actual scenarios for a target evaluation indicator set, and create multiple test questions on any new evaluation dimension. That is, a target evaluation indicator set corresponds to multiple new evaluation dimensions, and multiple test questions are constructed on each new evaluation dimension. In fact, these test questions are all applicable to the target evaluation indicator set.

[0080] Moreover, based on the multiple evaluation indicators in 101, multiple different sets of target evaluation indicators can be constructed, so that the final new evaluation dimensions are diverse. Therefore, compared with the evaluation indicators in 101, the embodiment of the present application is equivalent to creating more numerous, diverse and deep new evaluation dimensions, so as to achieve a more comprehensive and in-depth evaluation of the model reasoning effect.

[0081] In the above 101-106, the embodiment of the present application provides an evaluation method for the model reasoning effect on the power system. For the large language model that performs reasoning operations on the power data in the power system, first, multiple evaluation indicators are constructed for the model. Since the use of a single evaluation indicator or only the superposition of evaluation indicators to evaluate the model is not comprehensive and objective, the present application combines the evaluation indicators to obtain a target evaluation indicator set; then, in combination with the actual application scenario, an evaluation dimension is created for the target evaluation indicator set, so as to realize the construction of test questions for the target evaluation indicator set based on such evaluation dimensions, and each test question has a preset correct answer; the target power data required for the test question is input into the model for processing, and the reasoning result data given by the model is output, and the reasoning result data is compared with the preset correct answer of each test question, and an evaluation result of the model reasoning effect is obtained.

[0082] Therefore, the logical reasoning test implemented by constructing test questions in the embodiment of the present application can be combined with the actual application scenarios of the model to effectively evaluate the reasoning ability of the model in terms of logic and explanation, thereby providing a solution for effectively evaluating the reasoning effect of the model.

[0083] In some modified embodiments, for the target evaluation indicator set in 102 corresponding to at least one second evaluation dimension, the embodiment of the present application provides a detailed explanation of constructing this second evaluation dimension and provides the following detailed implementation steps:

[0084] A1 determines the target evaluation indicators included in the target evaluation indicator set and the first evaluation dimension corresponding to each target evaluation indicator;

[0085] A2 expands the first semantic information on each first evaluation dimension;

[0086] A3 combines the first semantic information obtained on different first evaluation dimensions with different preset application scenarios to construct a plurality of second semantic information;

[0087] A4 constructs a corresponding second evaluation dimension according to each second semantic information;

[0088] A5 uses the second evaluation dimension as the evaluation dimension for the target evaluation indicator set;

[0089] In the embodiment of the present application, the constructed second evaluation dimension is actually the data information and actual application scenarios on the first evaluation dimension of one or more evaluation indicators in the comprehensive 101, so as to construct a new evaluation dimension that integrates one or more evaluation indicators and actual application scenarios.

[0090] Specifically, the embodiments of the present application can use natural language processing to expand semantic information on the first evaluation dimension corresponding to each evaluation indicator. For example, for the evaluation indicator "logic", it is possible to obtain explanation information, condition constraint information, usage scenario requirement information, etc. related to "logic" by, but not limited to, constructing related words and other related methods, so as to obtain extended semantic information related to "logic".

[0091] When a target evaluation indicator set includes one target evaluation indicator, it is sufficient to obtain the extended semantic information related to the target evaluation indicator. However, when the target evaluation indicator set includes multiple target evaluation indicators, the embodiment of the present application needs to integrate the extended semantic information corresponding to the multiple evaluation indicators, and then combine it with different preset application scenarios as the extended semantic information corresponding to each target evaluation indicator set, and create a second evaluation dimension based on such semantic information.

[0092] It should be noted that a target evaluation indicator set can create one or more second evaluation dimensions, thereby achieving in-depth evaluation of the model based on different actual scenarios.

[0093] In some modified embodiments, a detailed explanation is given for constructing a test question for the target evaluation indicator set in 103. The embodiment of the present application provides the following detailed implementation steps:

[0094] B1, in at least one second evaluation dimension corresponding to the target evaluation indicator set, different scoring values ​​are configured for each second evaluation dimension, and the scoring value is used to measure the importance of the second evaluation dimension;

[0095] B2 allocates the proportion of questions to the second evaluation dimension corresponding to different scoring values ​​according to the preset weight allocation method;

[0096] B3 constructs test questions on different second evaluation dimensions according to the proportion of the number of questions.

[0097] As shown in B1-B3, the embodiments of the present application configure different scoring values ​​for each second evaluation dimension based on the different difficulty levels of the reasoning operations performed by the model on different second evaluation dimensions. The higher the scoring value, the more important the evaluation on the corresponding second evaluation dimension is.

[0098] Moreover, after assigning the scoring values, the embodiments of the present application may also assign different weights to different scoring values. The weights are used to indicate the number of test questions set, thereby constraining the second evaluation dimension by assigning high and low scoring values ​​and the number of questions, thereby achieving a better in-depth evaluation of model reasoning.

[0099] In some modified embodiments, 106 is explained in detail for evaluating the reasoning result data by using the preset correct answers corresponding to the test questions. The embodiment of the present application provides the following detailed implementation steps:

[0100] C1 obtains the inference result corresponding to each test question from the inference result data;

[0101] C2 compares the inference result with the preset correct answer corresponding to the test question. If they are consistent, the score corresponding to the test question is obtained according to the score value of the second evaluation dimension corresponding to the test question; if they are inconsistent, the test question is not scored;

[0102] C3 compares each test question one by one in the inference result data, and obtains the cumulative scores corresponding to the multiple test questions;

[0103] C4 determines the comprehensive evaluation result corresponding to the model by comparing the accumulated score with the preset comprehensive evaluation score range.

[0104] For C1-C4 above, the embodiments of the present application are explained in conjunction with the test questions shown in Table 2 below.

[0105] Table 2

[0106]

[0107]

[0108]

[0109] Furthermore, the constructed test questions may be multiple-choice questions, as shown in Table 3 below.

[0110] Table 3

[0111]

[0112]

[0113]

[0114] For Table 2 and Table 3, according to the test question settings, the questions are scientifically scored. For example, for policy interpretation and understanding and reasonable extension reasoning related content, the model's reasoning ability based on text understanding is examined, and 2 points are assigned; for scenario analysis and accurate judgment of transactions, dispatching, etc. in the power market, the model's scenario understanding and reasoning ability are examined, and 3 points are assigned; for problems such as power system node flow calculation involving scenario changes, the model's reasoning and comprehensive calculation ability are examined, and 4 points are assigned. And it can be but not limited to adding operations. Since the difficulty of multiple-choice questions will be slightly higher than that of single-choice questions, the corresponding weight of the multiple-choice questions that are answered correctly can be increased compared to the single-choice questions that are answered correctly. After forming a complete test question, the question is input into the model for testing, so as to obtain a more scientific and accurate evaluation of the reasoning ability of the large model.

[0115] The above 101-106 is actually an evaluation of the model reasoning effect based on the "model output results". In some modified embodiments, the model can also be evaluated by comprehensively considering the three aspects of data source, model quality, and application scenarios of model processing. As a reference factor, the weight can be increased to comprehensively update the evaluation results of the reasoning effect of the model output by 101-106, thereby achieving a more comprehensive and comprehensive evaluation of the model.

[0116] Data sources refer to the data used to build models and perform reasoning, including factors such as the quantity, quality, and diversity of the data. The quality of the data source directly affects the reasoning effect of the model. Model quality is to evaluate the accuracy, stability, and generalization ability of the model itself, including the modeling method and training technology of the model. The quality of the model determines the credibility of the reasoning results. Output results refer to the results obtained by model reasoning, including the accuracy, completeness, and consistency of information. The quality of the output results determines the actual utility of the model. Application scenarios refer to the performance of the model in actual applications, including reasoning speed, adaptability, stability, etc. The quality of the application scenarios affects the use effect of the model in actual scenarios. The comprehensive evaluation system needs to comprehensively consider the above four factors and evaluate from four aspects: data quality, model quality, output results, and application scenarios to comprehensively evaluate the pros and cons of the reasoning effect of the large language model and provide a reference for the improvement and optimization of the model.

[0117] For example, in operation, for the historical data source processed by the model, a first evaluation result of the historical source is obtained; for the design architecture of the model itself, a second evaluation result of the design architecture is obtained; for the historical application scenarios corresponding to the model, a third evaluation result of the historical application scenarios is obtained; the first evaluation result, the second evaluation result and the third evaluation result are combined as a reference evaluation factor for the model's reasoning ability on power data.

[0118] Furthermore, as a response to the above Figure 1The implementation of the method shown in the embodiment of the present application provides an evaluation device for the model reasoning effect on the power system. The device embodiment corresponds to the aforementioned method embodiment. For ease of reading, the device embodiment will no longer repeat the details of the aforementioned method embodiment one by one, but it should be clear that the device in this embodiment can correspond to all the contents of the aforementioned method embodiment. The device is used to achieve a more in-depth evaluation of the reasoning effect of the model, specifically as follows Figure 2 As shown, the device comprises:

[0119] A first construction unit 21 is used to construct a plurality of evaluation indicators for the model, each of the evaluation indicators corresponds to a first evaluation dimension, and there is no information overlap between the first evaluation dimensions corresponding to different evaluation indicators; the model is a large language model that performs reasoning operations on power data on the power system;

[0120] A selection unit 22, configured to select at least one evaluation indicator from the multiple evaluation indicators to form a target evaluation indicator set, wherein the target evaluation indicator set corresponds to at least one second evaluation dimension, and the second evaluation dimension is an evaluation dimension created in combination with an actual application scenario;

[0121] A second construction unit 23, configured to construct a test question for the target evaluation indicator set, wherein the test question is used to test the reasoning effect of the model on the second evaluation dimension, and each of the test questions corresponds to a preset correct answer;

[0122] A first acquisition unit 24 is used to acquire the target power data required for the test question;

[0123] A processing unit 25, configured to process the target power data using the model to output inference result data;

[0124] The evaluation unit 26 is used to evaluate the inference result data by using the preset correct answers corresponding to the test questions to output an evaluation result of the inference effect of the model.

[0125] Further, such as Figure 3 As shown, the second construction unit 23 includes:

[0126] A configuration module 231, configured to configure different scoring values ​​for each of the second evaluation dimensions corresponding to the target evaluation indicator set, wherein the scoring values ​​are used to measure the importance of the second evaluation dimension;

[0127] An allocating module 232 is used to allocate the proportion of the number of questions to the second evaluation dimension corresponding to different scoring values ​​according to a preset weight allocation method;

[0128] The construction module 233 is used to construct test questions on different second evaluation dimensions according to the proportion of the number of questions.

[0129] Further, such as Figure 3 As shown, the evaluation unit 26 includes:

[0130] A first acquisition module 261 is used to acquire, for each of the test questions, an inference result corresponding to the test question from the inference result data;

[0131] A comparison module 262 is used to compare the inference result with the preset correct answer corresponding to the test question, and if they are consistent, obtain the score corresponding to the test question according to the score value of the second evaluation dimension corresponding to the test question; if they are inconsistent, the test question is not scored;

[0132] A second acquisition module 263 is used to obtain the cumulative scores corresponding to the plurality of test questions after comparing each of the test questions one by one in the inference result data;

[0133] The determination module 264 is used to determine the comprehensive evaluation result corresponding to the model by comparing the accumulated score with a preset comprehensive evaluation score range.

[0134] Further, such as Figure 3 As shown, after the target evaluation indicator set is formed, the device further includes: a third construction unit 27, which is specifically used to:

[0135] In the target evaluation indicator set, determining the included target evaluation indicators and the first evaluation dimension corresponding to each of the target evaluation indicators;

[0136] expanding the first semantic information on each of the first evaluation dimensions;

[0137] Based on the first semantic information obtained on different first evaluation dimensions, combining with different preset application scenarios to construct a plurality of second semantic information;

[0138] According to each of the second semantic information, construct a corresponding second evaluation dimension;

[0139] The second evaluation dimension is used as the evaluation dimension for the target evaluation indicator set.

[0140] Further, such as Figure 3 As shown, the device also includes:

[0141] A second acquisition unit 28 is used to acquire a first evaluation result of a historical data source processed by the model;

[0142] The second acquisition unit 28 is used to acquire a second evaluation result of the design architecture of the model itself;

[0143] The second acquisition unit 28 is used to acquire a third evaluation result of the historical application scenario corresponding to the model;

[0144] The determination unit 29 is used to integrate the first evaluation result, the second evaluation result and the third evaluation result as reference evaluation factors for the reasoning ability of the model on power data.

[0145] In summary, the evaluation device for the model reasoning effect on the power system includes a processor and a memory. The above-mentioned first construction unit, selection unit, second construction unit, first acquisition unit, processing unit and evaluation unit are all stored in the memory as program units, and the processor executes the above-mentioned program units stored in the memory to realize the corresponding functions.

[0146] The processor contains a kernel, which calls the corresponding program unit from the memory. The kernel can be set to one or more. By adjusting the kernel parameters, the logical reasoning test implemented by constructing test questions can be combined with the actual application scenarios of the model to effectively evaluate the reasoning ability of the model in terms of logic and explanation, thereby providing a solution for effectively evaluating the reasoning effect of the model.

[0147] An embodiment of the present application provides a storage medium having a program stored thereon, which implements a method for evaluating the effect of model reasoning on a power system when the program is executed by a processor.

[0148] An embodiment of the present application provides a processor, which is used to run a program, wherein the program executes an evaluation method for model reasoning effect on a power system when the program is running.

[0149] The present application also provides a computer program product, which, when executed on a data processing device, is suitable for executing a program for initializing the steps of the method for evaluating the effect of model reasoning on a power system.

[0150] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0151] In a typical configuration, the device includes one or more processors (CPU), memory and bus. The device may also include input / output interface, network interface and the like.

[0152] The memory may include non-permanent memory in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM, and the memory includes at least one memory chip. The memory is an example of a computer-readable medium.

[0153] Computer readable media include permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. Information can be computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer readable media does not include temporary computer readable media (transitory media), such as modulated data signals and carrier waves.

[0154] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.

[0155] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that contain computer-usable program code.

[0156] The above are only embodiments of the present application and are not intended to limit the present application. For those skilled in the art, the present application may have various changes and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application should be included within the scope of the claims of the present application.

Claims

1. A method for evaluating the effect of model reasoning on a power system, characterized in that: The method comprises: Constructing a plurality of evaluation indicators for the model, each of the evaluation indicators corresponds to a first evaluation dimension, and there is no information overlap between the first evaluation dimensions corresponding to different evaluation indicators; the model is a large language model that performs reasoning operations on power data on the power system; Selecting at least one evaluation indicator from the multiple evaluation indicators to form a target evaluation indicator set, wherein the target evaluation indicator set corresponds to at least one second evaluation dimension, and the second evaluation dimension is an evaluation dimension created in combination with an actual application scenario; Constructing test questions for the target evaluation indicator set, the test questions are used to test the reasoning effect of the model on the second evaluation dimension, and each of the test questions corresponds to a preset correct answer; Obtaining target power data required for the test question; Processing the target power data using the model to output inference result data; The inference result data is evaluated by using the preset correct answer corresponding to the test question to output an evaluation result of the inference effect of the model.

2. The method according to claim 1, characterized in that The step of constructing a test topic for the target evaluation indicator set includes: In at least one second evaluation dimension corresponding to the target evaluation indicator set, a different score value is configured for each of the second evaluation dimensions, and the score value is used to measure the importance of the second evaluation dimension; According to a preset weight allocation method, the proportion of the number of questions allocated to the second evaluation dimension corresponding to different scoring values; According to the proportion of the number of questions, test questions are constructed on different second evaluation dimensions.

3. The method according to claim 2, characterized in that The step of evaluating the inference result data by using the preset correct answer corresponding to the test question to output an evaluation result of the inference effect of the model includes: For each of the test questions, obtaining the inference result corresponding to the test question from the inference result data; Compare the inference result with the preset correct answer corresponding to the test question, and if they are consistent, obtain the score corresponding to the test question according to the score value of the second evaluation dimension corresponding to the test question; if they are inconsistent, the test question is not scored; After comparing each of the test questions one by one in the inference result data, obtaining cumulative scores corresponding to the plurality of the test questions; By comparing the accumulated score with a preset comprehensive evaluation score range, the comprehensive evaluation result corresponding to the model is determined.

4. The method according to any one of claims 1 to 3, characterized in that After forming the target evaluation indicator set, the method further includes: In the target evaluation indicator set, determining the included target evaluation indicators and the first evaluation dimension corresponding to each of the target evaluation indicators; expanding the first semantic information on each of the first evaluation dimensions; Based on the first semantic information obtained on different first evaluation dimensions, combining with different preset application scenarios to construct a plurality of second semantic information; According to each of the second semantic information, construct a corresponding second evaluation dimension; The second evaluation dimension is used as the evaluation dimension for the target evaluation indicator set.

5. The method according to any one of claims 1 to 3, characterized in that The method further comprises: For a historical data source processed by the model, obtaining a first evaluation result of the historical source; For the design architecture of the model itself, obtaining a second evaluation result of the design architecture; For the historical application scenario corresponding to the model, obtaining a third evaluation result of the historical application scenario; The first evaluation result, the second evaluation result and the third evaluation result are integrated as reference evaluation factors for the reasoning ability of the model on power data.

6. A device for evaluating the effect of model reasoning on a power system, characterized in that: The device comprises: A first construction unit is used to construct a plurality of evaluation indicators for the model, each of the evaluation indicators corresponds to a first evaluation dimension, and there is no information overlap between the first evaluation dimensions corresponding to different evaluation indicators; the model is a large language model that performs reasoning operations on power data on the power system; A selection unit, configured to select at least one evaluation indicator from the multiple evaluation indicators to form a target evaluation indicator set, wherein the target evaluation indicator set corresponds to at least one second evaluation dimension, and the second evaluation dimension is an evaluation dimension created in combination with an actual application scenario; A second construction unit is used to construct a test question for the target evaluation indicator set, wherein the test question is used to test the reasoning effect of the model on the second evaluation dimension, and each of the test questions corresponds to a preset correct answer; A first acquisition unit, used for acquiring target power data required for the test question; A processing unit, used to process the target power data using the model to output inference result data; An evaluation unit is used to evaluate the inference result data by using the preset correct answer corresponding to the test question to output an evaluation result of the inference effect of the model.

7. The device according to claim 6, characterized in that The second building block comprises: A configuration module, configured to configure different scoring values ​​for each of the second evaluation dimensions corresponding to the target evaluation indicator set, wherein the scoring values ​​are used to measure the importance of the second evaluation dimension; An allocation module, configured to allocate a proportion of the number of questions to the second evaluation dimension corresponding to different scoring values ​​according to a preset weight allocation method; A construction module is used to construct test questions on different second evaluation dimensions according to the proportion of the number of questions.

8. The device according to claim 7, characterized in that The evaluation unit comprises: A first acquisition module, configured to acquire, for each of the test questions, an inference result corresponding to the test question from the inference result data; a comparison module, configured to compare the inference result with the preset correct answer corresponding to the test question, and if they are consistent, obtain the score corresponding to the test question according to the score value of the second evaluation dimension corresponding to the test question; if they are inconsistent, the test question is not scored; A second acquisition module, configured to obtain cumulative scores corresponding to a plurality of the test questions after comparing each of the test questions one by one in the inference result data; The determination module is used to determine the comprehensive evaluation result corresponding to the model by comparing the accumulated score with a preset comprehensive evaluation score range.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method for evaluating the effect of model reasoning on a power system according to any one of claims 1 to 5 is implemented.

10. An electronic device, characterized in that: include: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the computer program is executed by the processor, the method for evaluating the effect of model reasoning on a power system as described in any one of claims 1 to 5 is implemented.

Images