An AI large model-based energy agent testing method and a storage medium

Abstract

The application discloses an AI large model-based energy intelligent agent testing method and a storage medium, and relates to the cross field of energy digitization and large model automatic testing. The method constructs a layered, associated and versioned energy professional knowledge base, dynamically retrieves relevant knowledge units according to a testing task, injects five-layer standardized prompt words, generates test cases by an AI large model, and after multi-level constraint verification of knowledge consistency, business scope and parameter legality, inputs effective test cases into an energy intelligent agent by an automatic script and collects replies. According to the testing type, key evaluation elements are extracted, a combination evaluation mechanism of deterministic rule verification and semantic evaluation is used for judgment, and when the evaluation elements are missing, multiple rounds of follow-up questions are triggered. Finally, a test report is generated, and the prompt word local optimization and knowledge base incremental update are driven in reverse. The application improves the professionalism of test cases, the reliability of evaluation and the efficiency of defect positioning, and is suitable for multi-round interactive function and quality testing of energy intelligent agents.

Description

Technical Field

[0001] This invention relates to the field of energy digitization and software automation testing, specifically to a testing method and storage medium for energy intelligent agents based on a large AI model. Background Technology

[0002] Traditional energy agent testing relies on human testers manually designing test cases based on requirements documents using equivalence class partitioning and boundary value exhaustive search, which has the following inherent drawbacks: 1. Manual test cases are limited by the experience and energy of testers, resulting in incomplete coverage of business paths, blind spots in thinking, inability to generate test cases for massive combinations of industry scenarios, and low test coverage. 2. Traditional automated testing can only make binary assertions on the output results and can only verify whether the output is "right" or "wrong". It cannot judge the quality of the agent's response from the semantic and reasoning logic level and lacks deep semantic evaluation capabilities. 3. The energy industry is a vertically segmented field. General automated testing frameworks do not have built-in energy-related terminology and business rules, and the generated test questions are prone to being detached from actual operation and maintenance scenarios, resulting in poor test case authenticity. 4. The energy agent has multi-round interactive question-and-answer capabilities. Traditional single static test cases cannot simulate the real interactive process of maintenance personnel asking and following up questions continuously, making it difficult to verify the agent's continuous reasoning ability.

[0003] Existing automated testing solutions based on general large models also have technical shortcomings: First, general large models lack energy-specific knowledge constraints, which can easily lead to industry illusions and generate invalid test cases that are detached from energy business, interfering with test judgment; Second, they lack a standardized layered prompt word architecture, and prompt word iteration relies on repeated manual debugging, resulting in extremely high manpower and time costs for building the knowledge base and prompt words in the early stages; Third, the content generated by the large model is random, and the same input can produce different test outputs, making it difficult to reproduce defects, and there is no standardized multi-round follow-up questioning convergence mechanism, making it impossible to uniformly judge test results; Fourth, they do not design boundary out-of-bounds value test logic for energy equipment control scenarios, and cannot verify the intelligent reasoning ability of the agent to identify abnormal parameters and correct errors.

[0004] In summary, existing technologies lack automated testing solutions for energy intelligent agents that are adapted to vertical energy industries, support multi-round dynamic interaction, have industry knowledge constraints, and can automatically iterate and ask follow-up questions. Summary of the Invention

[0005] To address the shortcomings of existing technologies, the present invention aims to provide a testing method and storage medium for energy intelligent agents based on AI large models. This method can solve the problems of automatically, batch, and reliably completing functional testing, semantic quality assessment, and compliance verification of energy intelligent agents in multi-round interaction scenarios based on an energy professional knowledge base and a hierarchical standardized prompt word architecture. Furthermore, it can optimize prompt words and knowledge base through test results, forming a continuously iterative testing closed loop.

[0006] To achieve the above objectives, the present invention provides the following technical solution: A testing method for energy intelligent agents based on an AI large-scale model includes: S1, collecting energy business documents and structured business data and parsing them into knowledge units to construct an energy professional knowledge base; S2, layering role definition, function definition, task definition, behavior constraint, and output requirement layers to form five layers of standardized prompts; S3, binding the energy professional knowledge base and the five layers of standardized prompts to the AI ​​large-scale model; S4, writing an automated script to configure the AI ​​large-scale model interface key and the login, location, input, and output processes of the energy intelligent agent page; S5, performing intent parsing and entity parsing on the test task to obtain structured conditions, and retrieving knowledge units from the energy professional knowledge base to inject into the five layers of standardized prompts to form test prompts; S6, the AI ​​large-scale model generates test cases based on the test prompts, and sequentially performs knowledge consistency verification, business scope verification, and parameter legality verification; S7, running the automated script to process the test cases that pass the verification. The test questions in the test cases are used as the first round of questions input into the energy agent, and the returned response content is collected; S8, the evaluation indicators are determined according to the test type, and key evaluation elements are extracted from the response content. Elements that can be verified by the energy professional knowledge base are verified using deterministic rules, and the remaining elements are semantically evaluated to obtain the verification result and the evaluation result. If the two are inconsistent, they are marked as pending review; S9, when there are missing key evaluation elements and the number of follow-up questions has not reached the preset maximum value, follow-up question prompts are generated, input into the energy agent, and the response content is collected. Then, S8 is executed again; when all key evaluation elements are satisfied or the number of follow-up questions reaches the preset maximum value, the interaction is terminated and the judgment result is output; S10, the question content, response content, evaluation result and judgment result of each round are recorded, and a test report is generated; S11, S6 to S10 are executed in a loop through the combination of business scenario parameters. Based on the test report, the five-layer standardized prompts are locally optimized and the energy professional knowledge base is incrementally updated.

[0007] In this invention, preferably, the step of retrieving knowledge units from the energy expertise base includes: for each retrieved knowledge unit... Calculate the comprehensive retrieval score of the i-th knowledge unit relative to the current test task T using the following formula. And sorted from highest to lowest based on overall search score: , Where i is the sequence number of the knowledge unit, α is the preset weight coefficient of the semantic similarity component, β is the preset weight coefficient of the entity matching component, γ is the preset weight coefficient of the timeliness component, and δ is the preset weight coefficient of the priority component, and α+β+γ+δ=1. For semantic similarity components, For entity matching degree components, For time-sensitive components, As priority components, each component is normalized to the interval [0,1].

[0008] In this invention, the preferred element is the time-sensitive component. Let be the quantified value of the timeliness of the i-th knowledge unit, calculated using the following formula: , Where λ is the preset time decay coefficient. This is the current test execution time. The time of the most recent entry or update of the i-th knowledge unit. The range of its value is (0,1).

[0009] In this invention, preferably, in step S6, the knowledge consistency verification, business scope verification, and parameter validity verification include: Calculate the knowledge consistency score for test cases using the following formula: , Where c represents a test case, and Con(c) represents the knowledge consistency score of the test case. The entity set extracted from c includes projects, business scenarios, energy types, time dimensions, metrics, and equipment. In order to retrieve the corresponding set of entities in the energy knowledge base, To belong to and A collection of entities; The business scope verification is as follows: if the business scenario to which the test case belongs belongs to the set of business scenarios supported by the energy professional knowledge base, it is considered to pass; otherwise, it fails. The parameter validity is checked as follows: For the control parameter p in the test case, the test is considered passed if the following expression is true, otherwise it is considered failed: , Where p represents the control parameters to be sent to the device in the test case. This represents the lower limit of the safety threshold corresponding to p in the energy knowledge base. This represents the upper limit of the safety threshold corresponding to p in the energy expertise base; When a test case contains multiple control parameters, the above judgment is performed on each control parameter separately. The parameter validity check is passed when all of them pass. The knowledge consistency check is considered passed if the knowledge consistency score is not lower than the preset consistency threshold ε; otherwise, it fails. The value of ε is between 0.6 and 0.9. If any of the knowledge consistency check, business scope check, or parameter validity check fails, the test case will be marked as an invalid test case or a suspected hallucination test case.

[0010] In this invention, preferably, in S8, the evaluation indicators include: Evaluation criteria for knowledge-based question-and-answer questions: relevance of answers, logical consistency, and completeness of knowledge basis; Data query evaluation: indicator matching, data completeness, accuracy of the time dimension, and correctness of data source; Equipment control evaluation: equipment compatibility, legality of control parameters, completeness of execution result feedback, and satisfaction of safety constraints.

[0011] In this invention, preferably, in S8, the key evaluation elements are extracted as follows: according to the test type, a preset element extraction rule or element extraction prompt is called to extract the element name and element value corresponding to each evaluation indicator from the response content, and organize them into a structured key evaluation element set in key-value pair form; the preset evaluation element set corresponding to the evaluation indicator is the set of element names required by all evaluation indicators under the corresponding test type; in S9, the missing key evaluation elements are obtained by performing a difference operation between the key evaluation element set extracted in the current response and the preset evaluation element set corresponding to the evaluation indicator; the follow-up prompt is generated based on the original question content, the previous round of response content, the current test target, the missing key evaluation elements, and the knowledge units corresponding to the missing elements in the energy professional knowledge base, and uses the role definition layer, behavior constraint layer, and output requirement layer in the five-layer standardized prompt; the preset maximum number of follow-up rounds is 2 to 5 rounds.

[0012] In this invention, preferably, in S2, among the five layers of standardized prompts, the role definition layer is used to limit the AI ​​model's identity and professional competence boundaries as a senior energy management AI test engineer; the function definition layer is used to limit the business functions and test types of the tested energy intelligent agent; the task definition layer is used to clarify the current test cases, execution steps, and evaluation objectives; the behavior constraint layer is used to limit the knowledge source, business scope, language behavior, and security rules, and configure test target constraint prompts, which require the test questions generated by the AI ​​model to carry negative emotional words in order to test the energy intelligent agent's ability to identify sensitive words and respond in compliance with regulations; the output requirement layer is used to specify the output structure of test questions, follow-up questions, and test results.

[0013] In this invention, preferably, in S11, the local optimization is triggered when any of the following conditions occur: multiple consecutive test cases show the same type of error; the same type of business problem cannot be correctly identified by the energy intelligent agent; the test results are inconsistent with the expected results; the proportion of test cases that fail exceeds a preset threshold; the proportion of generated test cases with content not present in the energy professional knowledge base exceeds a preset threshold; or the key evaluation elements cannot be completed even after reaching a preset maximum number of follow-up questions. The local optimization includes: locating the corresponding level in the five-layer standardized prompt words according to the error type, and adding constraints, examples, or knowledge requirements at that level; wherein, a business scope error corresponds to the function definition layer, a lack of knowledge source corresponds to the knowledge source constraint in the behavior constraint layer, a follow-up question deviating from the original intent corresponds to the task definition layer, an incorrect answer format corresponds to the output requirement layer, and a language behavior error corresponds to the language behavior constraint in the behavior constraint layer.

[0014] In this invention, preferably, the energy professional knowledge base adopts a hierarchical classification and association organizational structure, including at least a project layer, a business layer, an equipment layer, an indicator layer, a time dimension layer, and a rule layer; the energy professional knowledge base constructed in S1 and the incrementally updated energy professional knowledge base in S11 are implemented as follows: when project configuration, equipment ledger, indicator configuration, or business rules change, the added, modified, and deleted data are obtained through the data synchronization interface, and the changed data is cleaned, structured, and version-marked before updating the energy professional knowledge base; in S5, the retrieval of knowledge units from the energy professional knowledge base is to dynamically retrieve corresponding knowledge based on the current project, business scenario, and task conditions, and prioritize the use of the latest valid version.

[0015] A computer-readable storage medium includes instructions that, when executed on a computer, cause the computer to perform the previously described method for testing energy agents based on a large AI model.

[0016] Compared with the prior art, the beneficial effects of the present invention are: This method constructs a hierarchical, interconnected, and versioned energy knowledge base and constrains the AI ​​large model from the source to only load knowledge units relevant to the current test task, effectively avoiding the industry illusion problem of general large models and improving the authenticity and professionalism of test cases. Through a multi-level constraint verification mechanism of "knowledge consistency verification + business scope verification + parameter legality verification," invalid test cases that are detached from energy business or exceed safety thresholds are filtered out, reducing the risk of misjudgment. A combined evaluation mechanism that combines deterministic rule verification and semantic evaluation leverages both the knowledge base's accurate verification capability of objective facts and retains the large model's semantic and logical reasoning and evaluation capabilities. When the conclusions of the two conflict, the result is marked as "pending verification," significantly reducing misjudgments caused by the evaluation model's own illusions. A multi-round dynamic adaptive test logic of "question-response-analysis-evaluation-follow-up questioning" is employed. The system simulates continuous interaction with real users to fully verify the continuous reasoning and information completion capabilities of the energy intelligence agent. It achieves standardized convergence of the testing process by pre-setting a maximum number of follow-up questions, reducing the reproducibility difficulty caused by the randomness of large models. By breaking down "good or bad" into quantifiable and verifiable evaluation indicators, it provides clear pass / fail / requirement criteria for all three scenarios: knowledge question answering, data querying, and equipment control. Through a five-layer standardized prompt word architecture and a local optimization mechanism based on error type, prompt word adjustments no longer rely on a complete rewrite but are precisely supplemented at the level where problems occur, improving iteration efficiency. Test results drive incremental updates to the knowledge base and local optimization of prompt words, forming a closed-loop testing system of "testing - discovering defects - locating root causes - optimizing knowledge and prompt words - regression verification," continuously improving the quality of the testing system itself. Attached Figure Description

[0017] Figure 1 This is a flowchart of the energy intelligent agent testing method based on the AI ​​large model of the present invention.

[0018] Figure 2 This is a schematic diagram of the structure of the five-layer standardized prompt words in the energy intelligent agent testing method based on the AI ​​large model of the present invention. Detailed Implementation

[0019] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0020] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains. The terminology used herein in the description of the invention is for the purpose of describing particular embodiments only and is not intended to be limiting of the invention. The term "and / or" as used herein includes any and all combinations of one or more of the associated listed items.

[0021] Example 1 This embodiment mainly describes a testing method for energy intelligent agents based on large AI models, such as... Figure 1 As shown, it includes the following steps: S1 collects energy business documents and structured business data and parses them into knowledge units to build an energy professional knowledge base.

[0022] In this step, energy business documents include product requirement specifications, operation manuals, maintenance manuals, as well as project data, time-based data, and formula data; structured business data includes project configurations, equipment ledgers, indicator configurations, and business rules. After collection, the source documents are first parsed to extract the main text, titles, tables, and key fields; then, knowledge units are segmented according to chapters, business objects, and semantic integrity to identify entities such as project names, equipment names, indicator names, business rules, and parameter ranges; structured data is directly mapped according to preset fields; finally, a unique identifier, data type, source document, update time, and version number are established for each knowledge unit and written into the energy professional knowledge base, realizing the structured accumulation and version management of energy knowledge.

[0023] The energy professional knowledge base adopts a hierarchical, categorized, and interconnected organizational structure, including at least a project layer, a business layer, an equipment layer, an indicator layer, a time dimension layer, and a rule layer. It further includes three sub-bases: a static business knowledge base, a target indicator base, and an equipment ledger base. The updating and dynamic retrieval of the energy professional knowledge base are implemented as follows: When project configurations, equipment ledgers, indicator configurations, or business rules change, the added, modified, and deleted data are obtained through a data synchronization interface. After cleaning, structuring, and version marking, the energy professional knowledge base is incrementally updated. This corresponds to the specific implementation methods for building the energy professional knowledge base in S1 and incrementally updating the energy professional knowledge base in S11. During test task execution, corresponding knowledge is dynamically retrieved based on the current project, business scenario, and task conditions, with priority given to calling the latest valid version. This corresponds to the specific implementation method for retrieving knowledge units from the energy professional knowledge base in S5.

[0024] S2 combines the role definition layer, function definition layer, task definition layer, behavior constraint layer, and output requirement layer into five layers of standardized prompts.

[0025] In this step, the five layers of content are pieced together layer by layer to form a complete set of prompt word templates, which are used to drive the AI ​​large model to generate test cases, generate follow-up questions, and output test results. The specific limitations and hierarchical positioning rules of each of the five layers are explained in detail in Example 4.

[0026] S3 binds an energy expertise base with five layers of standardized prompts to a large AI model.

[0027] This step involves configuring the basic operating parameters of the AI ​​large-scale model, including the model server address, call frequency limit, maximum generation length, and temperature parameters. By binding an energy expertise base and five layers of standardized prompts, the AI ​​large-scale model can directly reference energy expertise and be constrained by each layer when generating test cases, reducing the risk of generated content deviating from energy business.

[0028] Specifically, the binding includes two parts: knowledge base binding and prompt word binding. Knowledge base binding involves registering the retrieval interface address, access credentials, and retrieval parameters of the energy knowledge base into the AI ​​model's call configuration, enabling the AI ​​model to retrieve knowledge units in real-time according to structured conditions during the generation process. Prompt word binding involves saving five layers of standardized prompt words in the order of role definition, function definition, task definition, behavior constraint, and output requirement layers as a prompt word template file. Placeholders are reserved for knowledge injection positions in each layer of the template. When calling the AI ​​model, the knowledge units retrieved in real-time are first filled into the knowledge injection positions in the template, and then the completed prompt words are submitted to the AI ​​model as the input context. This achieves test case generation under knowledge constraints through retrieval-enhanced generation.

[0029] S4: Write automated scripts to configure the AI ​​large model interface key and the login, location, input, and output processes of the energy intelligent agent page.

[0030] In this step, the automated script is written using a scripting language combined with a page automation framework, such as Python combined with Playwright or Selenium. It is used to sequentially invoke knowledge retrieval, prompt word assembly, test case generation, energy agent interaction, result parsing, evaluation, and report generation according to the test task. The AI ​​large model interface key is configured in the script's environment variables or encrypted configuration file. The script requests and calls the AI ​​large model interface via Hypertext Transfer Protocol to complete test case generation and semantic evaluation.

[0031] The script pre-configures the Uniform Resource Locator (URL) for the tested energy agent's page, login account and password, input box element location expressions, send control element location expressions, and reply content container element location expressions. It executes in the following order: login, opening the dialog page, locating the input box, writing the question, triggering the send control, detecting changes in the reply content container according to a preset polling cycle, and capturing the complete reply text after the reply text stabilizes. For device control tests, the script also captures the control commands and execution result feedback returned by the energy agent.

[0032] S5 performs intent parsing and entity parsing on the test task to obtain structured conditions, and then retrieves knowledge units from the energy professional knowledge base to inject five layers of standardized prompt words to form test prompt words.

[0033] In this step, intent parsing is used to identify the test type of the test task, including knowledge-based question answering, data querying, or equipment control; entity parsing is used to extract structured conditions such as project, business scenario, energy type, time dimension, indicator type, and equipment type. During retrieval, not all knowledge is statically written into the prompt words, but rather dynamically filtered based on relevance. Only the knowledge units most relevant to the current test task are injected into the prompt word context, reducing the interference of irrelevant knowledge on the AI ​​large-scale model's inference and improving the professionalism and relevance of test cases. The calculation method for the retrieval score is explained in detail in Example 2.

[0034] Specifically, intent parsing is achieved by combining keyword matching and sentence template matching, or by using an AI model to categorize test tasks into one of three test types: knowledge question answering, data querying, or equipment control, based on preset classification prompts. Entity parsing is achieved by using named entity recognition, or by performing dictionary matching based on the project thesaurus, equipment thesaurus, indicator thesaurus, and rule thesaurus in the energy professional knowledge base. Fields such as project, business scenario, energy type, time dimension, indicator type, and equipment type are extracted from the test tasks, and the extracted results are organized into structured query conditions.

[0035] S6 uses an AI model to generate test cases based on test prompts, and then performs knowledge consistency verification, business scope verification, and parameter validity verification in sequence.

[0036] In this step, knowledge consistency verification is used to determine whether the projects, equipment, indicators, energy types, and business objects in the test cases can be found in the energy professional knowledge base; business scope verification is used to determine whether the test cases fall within the business scope supported by the tested energy intelligent agent; and parameter validity verification is used to determine whether the control parameters in the test cases conform to the preset business rules, parameter boundaries, and security constraints in the energy professional knowledge base. Through the above multi-level constraint verification, invalid test cases generated by the illusion of large AI models can be effectively filtered out. The specific verification method is explained in detail in Example 2.

[0037] In this embodiment, the test case includes at least the test case number, test type, project and business scenario, test question text, and expected key evaluation element set; wherein the test question text is used as the question content input to the energy intelligent agent, and the expected key evaluation element set is used to compare with the key evaluation elements actually extracted in the response content to determine the missing elements; the test case is passed between the various stages of the script in the form of structured data objects.

[0038] S7 runs an automated script that takes the test questions from the validated test cases as the first round of questions to be input into the energy agent, and collects the returned responses.

[0039] In this step, the first round of questions is automatically sent to the energy agent by an automated script. The sent content includes test questions and necessary contextual information. After receiving the questions, the energy agent returns text, data, or control responses. The automated script collects the response and passes it to subsequent steps for processing.

[0040] Specifically, after logging in, the automated script locates the energy agent's dialogue input box using an input box element positioning expression, writes the test question and necessary context into the input box, and triggers the send control. It then checks for text changes in the response content container according to a preset polling cycle. Once the text stops changing for two consecutive polling cycles, the script determines that the response rendering is complete and captures the complete response text. For device control tests, the script simultaneously captures the control commands, execution results, and safety prompts returned by the energy agent. The collected response content, along with the questions asked in this round, is stored in the test record for this round.

[0041] S8. Determine the evaluation indicators according to the test type, extract key evaluation elements from the response content, verify the elements that can be verified by the energy professional knowledge base using deterministic rules, and perform semantic evaluation on the remaining elements to obtain the verification results and evaluation results. If the two are inconsistent, they are marked as pending review.

[0042] In this step, evaluation indicators are categorized into three types based on test type: knowledge-based question-and-answer tests evaluate answer relevance, logical consistency, and completeness of knowledge basis; data query tests evaluate indicator matching, data completeness, accuracy in the time dimension, and correctness of data sources; and equipment control tests evaluate equipment matching, legality of control parameters, completeness of execution result feedback, and satisfaction of safety constraints. Deterministic rule verification applies to elements that can be directly verified by the energy professional knowledge base, such as equipment name, indicator name, data source, and parameter range; semantic evaluation applies to elements requiring reasoning and judgment, such as logical completeness, semantic rationality, and answer fit. When the conclusions of the two methods are inconsistent, the test case is marked as pending review to reduce misjudgments caused by the illusion of the evaluation model itself. It should be noted that although deterministic rule verification and semantic evaluation are performed on different types of evaluation elements, they are both overall conclusions drawn from the objective fact dimension and the semantic reasoning dimension, respectively, for the same response content. These are the deterministic rule verification result R and the semantic evaluation result M. Inconsistency between the two means that one of R and M is passed while the other is failed, indicating a conflict between the knowledge base fact comparison conclusion and the semantic reasoning conclusion. In this case, the correctness of the test case cannot be automatically determined, so it is marked as pending review and manually confirmed by the energy professional knowledge base. This does not refer to a comparison between two sets of evaluation elements. When there are no evaluation elements that can be verified by the energy professional knowledge base under a certain test type, R is defaulted to pass, and M is used as the final judgment result. Conversely, when there are no semantic evaluation elements, M is defaulted to pass, and R is used as the final judgment result. When both types of evaluation elements are empty, it is directly judged as a failure and the element is recorded as missing.

[0043] The key evaluation elements are extracted as follows: Pre-defined element extraction rules or element extraction prompts are invoked according to the test type to extract the element names and values ​​corresponding to each evaluation indicator from the response content, and organized into a structured set of key evaluation elements in key-value pairs. The element extraction rules are implemented using regular expressions and keyword template matching, suitable for structured elements such as device names, indicator names, values, and time. The element extraction prompts are executed by an AI large-scale model, suitable for elements requiring semantic understanding, and the extraction results are also output in key-value pair format of element names and values. The pre-defined set of evaluation elements corresponding to the evaluation indicator is the set of element names required by all evaluation indicators under the corresponding test type.

[0044] S9: When there are missing key evaluation elements and the number of follow-up questions has not reached the preset maximum value, generate follow-up question prompts, input them into the energy agent and collect the response content, then return to execute S8; when all key evaluation elements are satisfied or the number of follow-up questions has reached the preset maximum value, terminate the interaction and output the judgment result.

[0045] In this step, the missing key evaluation elements are obtained by performing a difference operation between the set of key evaluation elements extracted from the current response and the preset set of evaluation elements corresponding to the evaluation indicators. Follow-up prompts are generated based on the original question content, the previous round of response content, the current test objective, the missing key evaluation elements, and the knowledge units corresponding to the missing elements in the energy professional knowledge base. Through multiple rounds of follow-up questions, a scenario of continuous questioning by a real user to an energy intelligent agent can be simulated, testing its continuous reasoning and information completion capabilities.

[0046] The maximum number of follow-up questions is set according to the test type, ranging from 2 to 5 rounds, with 3 rounds being preferred. The interaction terminates after the maximum number of follow-up questions is reached to avoid the test process from failing to converge due to infinite follow-up questions. At the same time, it ensures that the number of interaction rounds is consistent when the same test task is executed multiple times, which facilitates the reproduction and comparison of test results.

[0047] S10 records the questions asked, responses, evaluation results, and judgment results for each round, and generates a test report.

[0048] In this step, the test report includes the questions asked, follow-up questions asked, responses received, evaluation results for each round, the final judgment, and any missing key evaluation elements. The test report serves both as a presentation of the test conclusions and as data support for subsequent prompt optimization and knowledge base updates.

[0049] S11 iterates through the combination of business scenario parameters and executes S6 to S10 in a loop. Based on the test report, it locally optimizes the five-layer standardized prompt words and incrementally updates the energy professional knowledge base.

[0050] In this step, the business scenario parameter combinations include enumerable parameters such as project, business scenario, energy type, time dimension, indicator type, and equipment type. Traversing these parameter combinations allows for the batch generation of test cases covering multiple scenarios. After testing, based on the common errors of the failed test cases and those awaiting review, the system reverse-engineers the corresponding level in the five-layer standardized prompt words for local optimization, or incrementally updates the missing knowledge content to the energy professional knowledge base, thus forming a closed-loop testing process of "testing - defect discovery - root cause location - knowledge and prompt word optimization - regression verification." The specific triggering conditions for local optimization are detailed in Example 4.

[0051] Specifically, the traversal method can be the full combination of the Cartesian product of each enumerable parameter, or it can be an orthogonal array or pairwise sampling, which controls the size of test cases while ensuring scenario coverage. During the traversal, S6 to S10 are executed once for each parameter combination, and the judgment results of each parameter combination are summarized in the test report.

[0052] Example 2 This embodiment mainly describes the process of dynamic knowledge retrieval and multi-level constraint verification.

[0053] Retrieving knowledge units from the energy expertise base in S5 includes: for each retrieved knowledge unit... Let i be the sequence number of the knowledge unit, and T be the current test task. The comprehensive retrieval score of the i-th knowledge unit relative to the current test task T is calculated using the following formula. And sorted from highest to lowest based on overall search score: (1) Wherein, α is the preset weight coefficient of the semantic similarity component, β is the preset weight coefficient of the entity matching component, γ is the preset weight coefficient of the timeliness component, and δ is the preset weight coefficient of the priority component. For semantic similarity components, For entity matching degree components, For time-sensitive components, The components are priority components, and each component is normalized to the interval [0,1]. All the weight coefficients mentioned above are non-negative and satisfy α+β+γ+δ=1.

[0054] In practical applications, the semantic similarity component is obtained by calculating the cosine similarity between the knowledge unit text vector and the test task query vector; the entity matching component is obtained by comparing the degree of overlap between the entities labeled in the knowledge unit and the entities extracted by the test task; the timeliness component is used to prioritize the most recently added or updated knowledge units; and the priority component is used to reflect the importance level of the knowledge unit in the business rules.

[0055] The calculation methods and value ranges of each component are as follows: Semantic similarity component Take the cosine similarity between the text vector of the i-th knowledge unit and the query vector of the current test task T, add 1 to the cosine similarity, take half of it, and map it to the interval [0,1]; entity matching degree component The ratio of the number of elements in the intersection of the entity set labeled by the i-th knowledge unit and the entity set extracted by the current test task T to the number of elements in the entity set extracted by the current test task T is taken, with a value range of [0,1]; Timeliness component Calculated using the above exponential decay formula, the value range is (0,1]; priority component The ratio of the importance level of the i-th knowledge unit as preset in the business rules to the highest importance level is taken, with a value range of [0,1].

[0056] The weighting coefficients α, β, γ, and δ are all non-negative and satisfy α + β + γ + δ = 1. Their values ​​are configured according to the test type: α = 0.45, β = 0.25, γ = 0.15, δ = 0.15 for knowledge-based question-and-answer tests; α = 0.35, β = 0.30, γ = 0.25, δ = 0.10 for data query tests; and α = 0.30, β = 0.25, γ = 0.10, δ = 0.35 for equipment control tests. The resulting comprehensive retrieval score is then calculated. The value range is [0,1], and the preset threshold θ is between 0.6 and 0.8, preferably 0.7. The above values ​​can be determined by those skilled in the art through conventional experiments based on the data update frequency and testing accuracy requirements of specific business operations, and do not constitute a limitation on the scope of protection of this invention.

[0057] Time-sensitive components Calculate using the following formula: (2) Where λ is the preset time decay coefficient. This is the current test execution time. The time of the most recent entry or update of the i-th knowledge unit. and All are calculated in days. minus , where is the time difference between the i-th knowledge unit and the current test execution time, exp[·] represents an exponential function with the natural constant e as the base, and · represents multiplication. The value range is (0,1). This formula indicates that the more recent a knowledge unit is, the higher its timeliness score. After the knowledge base is updated, the system prioritizes calling the latest valid version to avoid outdated knowledge interfering with test case generation and evaluation. Only when... The corresponding knowledge unit will only be injected with test prompt words if the threshold value is not lower than the preset threshold θ.

[0058] The time decay coefficient λ is set according to the knowledge update frequency: when the energy business rules are updated on a weekly basis, λ=0.05 is taken, and when the equipment ledger is updated on a daily basis, λ=0.2 is taken, so that the timeliness score of knowledge units that have not been updated for more than 30 days drops to below 0.05, and thus are effectively filtered by the threshold θ.

[0059] S6's multi-level constraint verification includes knowledge consistency verification, business scope verification, and parameter validity verification. Knowledge consistency verification measures the degree to which entities in a test case are supported within the energy knowledge base. The knowledge consistency score Con(c) for test case c is calculated using the following formula, where c is the test case to be verified: (3) Where c is the test case to be verified, and Con(c) is the knowledge consistency score of test case c. This is the set of entities extracted from c, including projects, business scenarios, energy types, time dimensions, metrics, and equipment. To provide a set of entities that can be retrieved from the energy knowledge base; To belong to and The set of entities constituted by Con(c). The closer Con(c) is to 1, the more the entities in the test case are supported by the energy expertise base, and the lower the probability of hallucination.

[0060] The rule for determining knowledge consistency is as follows: If Con(c) is not lower than the preset consistency threshold ε, the test is considered successful; otherwise, it fails. ε ranges from 0.6 to 0.9, with 0.8 being preferred. If the set is empty, Con(c) is directly determined to be zero and the test case is marked as invalid.

[0061] The business scope verification is as follows: if the business scenario to which c belongs belongs to the set of business scenarios supported by the energy professional knowledge base, it is considered to pass; otherwise, it fails.

[0062] The parameter validity check is as follows: for the control parameter p in c, it is considered valid if the following expression is true, otherwise it is considered invalid. (4) Where p represents the control parameters to be sent to the device in test case c. This represents the lower limit of the safety threshold corresponding to p in the energy knowledge base. This represents the upper limit of the safety threshold corresponding to p in the energy knowledge base; that is, the value of p is neither lower than the lower limit of the safety threshold nor higher than the upper limit of the safety threshold. By validating the parameter validity, out-of-bounds control commands in test cases can be prevented, reducing the safety risks to real equipment during the testing process.

[0063] When a test case involves multiple control parameters, the above judgment is performed on each control parameter separately. The parameter validity check passes if all judgments pass, and fails if any judgment fails. If the safety threshold for a certain control parameter is not included in the energy professional knowledge base, the parameter is marked as a parameter to be confirmed, the execution of the test case is paused, and a prompt is given to manually supplement the threshold, or the historical value range of the parameter is temporarily used to avoid misjudgment due to missing thresholds. Lower limit of safety threshold. and upper limit of safety threshold These parameters originate from the equipment rated parameters registered in the equipment ledger database, the process safety boundaries configured in the rule layer, and the industry standard limits recorded in the static business knowledge base.

[0064] If any of the knowledge consistency check, business scope check, or parameter validity check fails, the corresponding test case will be marked as an invalid test case or a suspected hallucination test case, and regeneration will be triggered. During regeneration, the missing knowledge constraints or information exceeding the security threshold corresponding to the test case will be fed back to the AI ​​big model to constrain the subsequent generated content.

[0065] Example 3 This embodiment mainly describes the combined evaluation and dynamic questioning process.

[0066] S8's evaluation metrics are divided into three categories based on the test type. Knowledge-based question-and-answer tests evaluate the relevance of answers, logical consistency, and completeness of knowledge base; data query tests evaluate matching accuracy, data completeness, accuracy in the time dimension, and correctness of data sources; and equipment control tests evaluate equipment matching accuracy, legality of control parameters, completeness of execution result feedback, and satisfaction of safety constraints.

[0067] The deterministic rule verification result R is obtained as follows: For evaluation elements that can be directly verified by the energy professional knowledge base, the value of the element in the response content is compared with the standard value in the energy professional knowledge base item by item. If they are consistent or fall within the allowable error range, the item is judged to pass. When all elements pass, R is passed. When any item fails, R is failed.

[0068] The semantic evaluation result M is obtained as follows: The semantic evaluation is performed by the evaluation model, which is implemented using the same large AI model as the generated test cases, or by an independent large evaluation model. During the evaluation, evaluation prompts are assembled according to the test type. The evaluation prompts include the original question content and response content of the tested energy intelligent agent, the semantic evaluation index and scoring rules corresponding to the current test type, and the evaluation model is required to output the score of each semantic evaluation index and the judgment conclusion of pass or fail for each item. The score of each semantic evaluation index is normalized to the interval [0,1]. When the score is not lower than the preset semantic threshold σ, the item is judged as pass. σ is 0.6 to 0.9, preferably 0.75. When all semantic evaluation indices corresponding to the test type are judged as pass, M is pass. When any item is judged as fail, M is fail.

[0069] The judgment rules for combined evaluation are as follows: when both the deterministic rule verification result R and the semantic evaluation result M are passed, the evaluation is considered passed; when both R and M are failed, the evaluation is considered failed; when R and M are inconsistent, the evaluation is considered pending review. The pending review status is used to alert testers to test cases where the rule and semantic conclusions conflict. Test cases judged to be pending review are recorded in the test report and added to the manual review queue. Testers manually confirm the results based on the energy professional knowledge base and then backfill the final conclusion. The backfilled conclusion serves as the basis for the next local optimization and knowledge base update. In the above judgment rules, the comparison object of R and M is two overall conclusions of the same response content, not a comparison between two sets of evaluation elements. R is derived from objective fact comparison, and M is derived from semantic reasoning evaluation. The two are independent and mutually corroborative; therefore, if one is passed and the other is failed, a conclusion conflict is constituted. To avoid ambiguity in judgments when a certain type of evaluation element is missing, if no evaluation element verifiable by the energy expertise base is available for the current test type, R is defaulted to pass and M is used as the final judgment result; if no semantic evaluation element is available, M is defaulted to pass and R is used as the final judgment result; if both types of evaluation elements are empty, it is directly judged as fail and the element is recorded as missing. This ensures that a definite conclusion of pass, fail, or pending review can be obtained for any test type.

[0070] Dynamic follow-up questions are triggered when there are missing key evaluation elements and the current follow-up questioning round has not reached the preset maximum follow-up questioning round. Missing key evaluation elements are obtained by performing a difference operation between the set of key evaluation elements extracted from the current response and the preset set of evaluation elements corresponding to the evaluation indicators. Follow-up question prompts are generated based on the original question content, the previous round's response content, the current test objective, the missing key evaluation elements, and the knowledge units corresponding to the missing elements in the energy professional knowledge base. After inputting the follow-up question prompts as the energy agent for the next round of questioning and collecting the new round's response content, the process returns to execution S8 to continue result analysis and combination evaluation until all key evaluation elements are satisfied or the preset maximum follow-up questioning round is reached; the preset maximum follow-up questioning round is 2 to 5 rounds, with 3 rounds being preferred. When generating follow-up question prompts, the role definition layer, behavior constraint layer, and output requirement layer of the five-layer standardized prompts are used, only replacing the test objective and missing element descriptions in the task definition layer to ensure consistency in expression style and constraints across multiple rounds of follow-up questioning, and to avoid deviation of the follow-up question content from the original test intent.

[0071] Example 4 This embodiment mainly describes the five-layer standardized prompt words, the local optimization process, and provides the implementation method of the computer-readable storage medium.

[0072] The five-layer standardized prompts are stacked sequentially in the order of role definition layer, function definition layer, task definition layer, behavior constraint layer, and output requirement layer, as shown in the structure below. Figure 2As shown. The role definition layer defines the identity and professional capabilities of the AI ​​large model as a senior energy management AI test engineer; the function definition layer defines the business functions and test types of the tested energy intelligent agent, including knowledge question answering, data querying, and equipment control; the task definition layer clarifies the current test cases, execution steps, and evaluation objectives; the behavior constraint layer defines the knowledge source, business scope, language behavior, and security rules; and the output requirement layer specifies the test questions, follow-up questions, and the output structure of the test results.

[0073] The behavioral constraint layer is also used to configure test target constraint prompts. For example, test questions generated by the AI ​​model can be required to contain negative sentiment words to test the energy agent's ability to identify sensitive words and respond in compliance with regulations. When responding, the energy agent should avoid using impolite or negative expressions and should not provide advice beyond the scope of business, thereby verifying the safety of its language behavior.

[0074] The local optimization of the five-layer standardized prompts in S11 is triggered when any of the following situations occur: multiple test cases show the same type of error; the same type of business problem cannot be correctly identified by the energy agent; the test results are inconsistent with the expected results; the proportion of test cases that fail exceeds the preset threshold; the proportion of generated test cases with content that is not in the energy professional knowledge base exceeds the preset threshold; or the key evaluation elements cannot be completed even after reaching the preset maximum number of follow-up questions.

[0075] Local optimization is achieved through a process of "error type identification → prompt word hierarchy location → local rule adjustment → regression verification," rather than a complete rewrite of the prompt words. Specifically, errors in business scope correspond to the function definition layer, missing knowledge sources correspond to knowledge source constraints in the behavior constraint layer, follow-up questions deviating from the original intent correspond to the task definition layer, incorrect answer format corresponds to the output requirement layer, and language behavior errors correspond to language behavior constraints in the behavior constraint layer. The system identifies specific problems based on test failure results and adds constraints, examples, or knowledge requirements only to the corresponding level. The relevant tests are then re-executed, and the test results before and after the adjustment are compared.

[0076] The specific values ​​in the above triggering conditions can be configured as needed. For example, the "multiple" in "multiple consecutive test cases with the same type of error" can be 3 to 5; the preset threshold for the proportion of test cases that fail can be 10% to 30%; the threshold for the proportion of test cases with non-existent content in the energy professional knowledge base can be 5% to 20%; during regression verification, all test cases that trigger local optimization, as well as test cases belonging to the same business scenario, are re-executed, and the changes in the proportion of test cases that fail and the proportion of test cases to be reviewed before and after the adjustment are compared. If the proportion does not decrease, the local optimization is rolled back and manual intervention is required.

[0077] This invention also provides a computer-readable storage medium storing a computer program. When executed by a processor, this computer program implements the various processes of the above-described embodiment of the energy intelligent agent testing method based on an AI large model, and achieves the same technical effect. The computer-readable storage medium can be of various types, such as read-only memory (ROM), random access memory (RAM), magnetic disk, or optical disk.

[0078] The above description is a detailed description of the preferred embodiments of the present invention. However, the embodiments are not intended to limit the scope of the patent application of the present invention. All equivalent changes or modifications made under the technical spirit of the present invention should fall within the patent scope covered by the present invention.

Claims

1. A testing method for energy intelligent agents based on AI large-scale models, characterized in that, include: S1: Collect energy business documents and structured business data and parse them into knowledge units to build an energy professional knowledge base; S2, by layering the role definition layer, function definition layer, task definition layer, behavior constraint layer, and output requirement layer, forms five layers of standardized prompt words; S3 binds the energy expertise base with five layers of standardized prompts to the AI ​​big model; S4, write automated scripts to configure the AI ​​large model interface key and the login, location, input and output processes of the energy intelligent agent page; S5 performs intent parsing and entity parsing on the test task to obtain structured conditions, and then retrieves knowledge units from the energy professional knowledge base to inject five layers of standardized prompt words to form test prompt words; S6 uses an AI model to generate test cases based on test prompts, and performs knowledge consistency verification, business scope verification, and parameter validity verification in sequence. S7 runs an automated script that takes the test questions from the verified test cases as the first round of questions to be input into the energy agent and collects the returned response content. S8. Determine the evaluation indicators according to the test type, extract key evaluation elements from the response content, verify the elements that can be verified by the energy professional knowledge base using deterministic rules, and perform semantic evaluation on the remaining elements to obtain the verification results and evaluation results. If the two are inconsistent, they are marked as pending review. S9: When there are missing key evaluation elements and the number of follow-up questions has not reached the preset maximum value, generate follow-up question prompts, input them into the energy agent and collect the response content, then return to execute S8; when all key evaluation elements are satisfied or the number of follow-up questions has reached the preset maximum value, terminate the interaction and output the judgment result. S10 records the questions asked, responses, evaluation results, and judgment results for each round, and generates a test report; S11 iterates through the combination of business scenario parameters and executes S6 to S10 in a loop. Based on the test report, it locally optimizes the five-layer standardized prompt words and incrementally updates the energy professional knowledge base.

2. The energy agent testing method based on AI large model according to claim 1, characterized in that, In S5, retrieving knowledge units from the energy expertise base includes: for each retrieved knowledge unit... Calculate the comprehensive retrieval score of the i-th knowledge unit relative to the current test task T using the following formula. And sorted from highest to lowest based on overall search score: ， Where i is the sequence number of the knowledge unit, α is the preset weight coefficient of the semantic similarity component, β is the preset weight coefficient of the entity matching component, γ is the preset weight coefficient of the timeliness component, and δ is the preset weight coefficient of the priority component, and α+β+γ+δ=1. For semantic similarity components, For entity matching degree components, For time-sensitive components, As priority components, each component is normalized to the interval [0,1].

3. The energy agent testing method based on AI large model according to claim 2, characterized in that, Time-sensitive components Let be the quantified value of the timeliness of the i-th knowledge unit, calculated using the following formula: ， Where λ is the preset time decay coefficient. This is the current test execution time. The time of the most recent entry or update of the i-th knowledge unit. The range of its value is (0,1).

4. The energy agent testing method based on AI large model according to claim 1, characterized in that, In S6, knowledge consistency verification, business scope verification, and parameter validity verification include: Calculate the knowledge consistency score for test cases using the following formula: ， Where c represents a test case, and Con(c) represents the knowledge consistency score of the test case. The entity set extracted from c includes projects, business scenarios, energy types, time dimensions, metrics, and equipment. In order to retrieve the corresponding set of entities in the energy knowledge base, To belong to and A collection of entities; The business scope verification is as follows: if the business scenario to which the test case belongs belongs to the set of business scenarios supported by the energy professional knowledge base, it is considered to pass; otherwise, it fails. The parameter validity is checked as follows: For the control parameter p in the test case, the test is considered passed if the following expression is true, otherwise it is considered failed: ， Where p represents the control parameters to be sent to the device in the test case. This represents the lower limit of the safety threshold corresponding to p in the energy knowledge base. This represents the upper limit of the safety threshold corresponding to p in the energy expertise base; When a test case contains multiple control parameters, the above judgment is performed on each control parameter separately. The parameter validity check is passed when all of them pass. The knowledge consistency check is considered passed if the knowledge consistency score is not lower than the preset consistency threshold ε; otherwise, it fails. The value of ε is between 0.6 and 0.

9. If any of the knowledge consistency check, business scope check, or parameter validity check fails, the test case will be marked as an invalid test case or a suspected hallucination test case.

5. The energy agent testing method based on AI large model according to claim 1, characterized in that, In S8, the evaluation indicators include: Evaluation criteria for knowledge-based question-and-answer questions: relevance of answers, logical consistency, and completeness of knowledge basis; Data query evaluation: indicator matching, data completeness, accuracy of the time dimension, and correctness of data source; Equipment control evaluation: equipment compatibility, legality of control parameters, completeness of execution result feedback, and satisfaction of safety constraints.

6. The energy agent testing method based on AI large model according to claim 1, characterized in that, In S8, the key evaluation elements are extracted as follows: according to the test type, the preset element extraction rules or element extraction prompts are called to extract the element names and element values ​​corresponding to each evaluation indicator from the response content, and organize them into a structured set of key evaluation elements in the form of key-value pairs; the preset evaluation element set corresponding to the evaluation indicator is the set of element names required by all evaluation indicators under the corresponding test type. In S9, the missing key evaluation elements are obtained by performing a difference operation between the set of key evaluation elements extracted from the current response and the set of preset evaluation elements corresponding to the evaluation indicators; the follow-up prompts are generated based on the original question content, the previous round of response content, the current test target, the missing key evaluation elements, and the knowledge units corresponding to the missing elements in the energy professional knowledge base, and the role definition layer, behavior constraint layer, and output requirement layer in the five-layer standardized prompts are used. The maximum number of follow-up questions is set to 2 to 5.

7. The energy agent testing method based on AI large model according to claim 1, characterized in that, In S2, among the five layers of standardized prompts, the role definition layer is used to limit the AI ​​model's identity and professional competence boundaries as a senior energy management AI test engineer; the function definition layer is used to limit the business functions and test types of the tested energy intelligent agent; the task definition layer is used to clarify the current test cases, execution steps, and evaluation objectives; the behavior constraint layer is used to limit the knowledge source, business scope, language behavior, and security rules, and configure test target constraint prompts. The test target constraint prompts require the test questions generated by the AI ​​model to carry negative emotional words in order to test the energy intelligent agent's ability to identify sensitive words and respond in compliance with regulations; the output requirement layer is used to specify the output structure of test questions, follow-up questions, and test results.

8. The energy agent testing method based on AI large model according to claim 1, characterized in that, In S11, the local optimization is triggered when any of the following situations occur: multiple test cases show the same type of error; the same type of business problem cannot be correctly identified by the energy intelligent agent; the test results are inconsistent with the expected results; the proportion of test cases that fail exceeds a preset threshold. The proportion of generated test cases containing content that is not present in the energy expertise base exceeds a preset threshold. Even after reaching the maximum number of follow-up questions set in advance, the key evaluation elements still could not be fully identified. The local optimization includes: locating the corresponding level in the five-layer standardized prompt words according to the error type, and adding constraints, examples or knowledge requirements at that level; wherein, business scope error corresponds to the function definition layer, knowledge source missing corresponds to knowledge source constraints in the behavior constraint layer, follow-up question deviates from the original intent corresponds to the task definition layer, answer format error corresponds to the output requirement layer, and language behavior error corresponds to language behavior constraints in the behavior constraint layer.

9. The energy agent testing method based on AI large model according to claim 1, characterized in that, The energy expertise base adopts a hierarchical classification and related organizational structure, including at least a project layer, a business layer, an equipment layer, an indicator layer, a time dimension layer, and a rule layer. The energy expertise base built in S1 and the incremental update energy expertise base in S11 are implemented as follows: When the project configuration, equipment ledger, indicator configuration or business rules change, the added, modified and deleted data are obtained through the data synchronization interface, and the changed data is cleaned, structured and version marked before updating the energy expertise base. In S5, the step of retrieving knowledge units from the energy professional knowledge base involves dynamically retrieving corresponding knowledge based on the current project, business scenario, and task conditions, and prioritizing the use of the latest valid version.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium includes instructions that, when executed on a computer, cause the computer to perform the energy agent testing method based on an AI large model as described in any one of claims 1 to 9.

Images