Robustness evaluation methods, devices, computer equipment, readable storage media, and program products for power system AI models

Abstract

This application relates to a method, apparatus, computer equipment, computer-readable storage medium, and computer program product for robustness evaluation of a power system AI model. It includes: constructing basic evaluation data based on original power system flow samples; inputting the basic evaluation data into the power system AI model to be evaluated to obtain model output; iteratively perturbing the basic evaluation data based on the model output and the power system AI model to be evaluated using a data perturbation strategy and an adaptive closed-loop perturbation strategy to obtain a perturbation output; and generating robustness evaluation results for the power system AI model to be evaluated based on the model output and the perturbation output. This method can adaptively iterate perturbations, thereby improving the accuracy of robustness evaluation.

Description

Technical Field

[0001] This application relates to the field of power system technology, and in particular to a robustness evaluation method, apparatus, computer equipment, computer-readable storage medium, and computer program product for power system AI models. Background Technology

[0002] With the development of digitalization, networking, and intelligentization technologies in power systems, artificial intelligence-based power flow classification technology has been widely applied. This technology can automatically learn flow characteristics and accurately identify business flow and abnormal flow. It features strong adaptability, high classification accuracy, and no need for manual rule definition, making it an important support tool for power system network security monitoring and protection. Before actually deploying an artificial intelligence model, it is usually necessary to verify the model's security and stability using traditional robustness testing methods.

[0003] Traditional technologies primarily evaluate models through methods such as testing with fixed test datasets, comparing against preset scenarios, and statistically analyzing the output results. However, current robustness evaluation methods suffer from simplistic perturbation approaches, making it difficult to simulate real-world perturbation behavior under the complex communication environments of power systems. Summary of the Invention

[0004] Therefore, it is necessary to provide a robustness evaluation method, apparatus, computer equipment, computer-readable storage medium, and computer program product for power system AI models that can adaptively iterate perturbations to improve the accuracy of robustness evaluation, in order to address the above-mentioned technical problems.

[0005] Firstly, this application provides a robustness evaluation method for an AI model of a power system, including:

[0006] Based on the original power system flow samples, basic evaluation data is constructed;

[0007] The basic evaluation data is input into the AI ​​model of the power system to be evaluated to obtain the model output;

[0008] Based on the model output and the power system AI model to be evaluated, the basic evaluation data is iteratively perturbed using a data perturbation strategy and an adaptive closed-loop perturbation strategy to obtain the perturbation output.

[0009] Based on the model output and the disturbance output, robustness evaluation results are generated for the power system AI model to be evaluated.

[0010] In one embodiment, based on the model output and the power system AI model to be evaluated, the basic evaluation data is iteratively perturbed using a data perturbation strategy and an adaptive closed-loop perturbation strategy to obtain a perturbation output, wherein one iteration includes:

[0011] Obtain the embedding space offset from the previous iteration; wherein the embedding space offset of the first iteration is a random initial value; based on the embedding space offset, determine the perturbation intensity of the current iteration through an adaptive closed-loop perturbation strategy; based on the perturbation intensity of the current iteration, perturb the perturbation dataset of the previous iteration through a data perturbation strategy to obtain the perturbation dataset of the current iteration; input the perturbation dataset of the current iteration into the power system AI model to be evaluated to obtain the perturbation result of the current iteration; based on the model output and the perturbation result, determine the embedding space offset of the current iteration; based on the model output and the perturbation result, determine whether a preset stopping condition has been met; if the preset stopping condition has not been met, start the next iteration perturbation process; if the preset stopping condition has been met, stop the iteration perturbation process and use the perturbation result of the current iteration as the perturbation output.

[0012] In one embodiment, the step of perturbing the perturbation dataset of the previous iteration based on the perturbation strength of the current iteration using a data perturbation strategy to obtain the perturbation dataset of the current iteration includes:

[0013] If the data perturbation strategy includes at least one of the semantic constraint perturbation strategy, the temporal perturbation strategy, and the joint behavior perturbation strategy, then based on the perturbation strength of the current iteration and the data perturbation strategy, the perturbation dataset of the previous iteration is perturbed sequentially in a preset order to obtain the perturbation dataset of the current iteration; if the data perturbation strategy includes any one of the semantic constraint perturbation strategy, the temporal perturbation strategy, and the joint behavior perturbation strategy, then based on the perturbation strength of the current iteration and the data perturbation strategy, the perturbation dataset of the previous iteration is perturbed to obtain the perturbation dataset of the current iteration.

[0014] In one embodiment, determining whether a preset stopping condition has been met based on the model output and the perturbation result includes:

[0015] Based on the model output and the perturbation result, the classification change result of the current iteration is determined; if the classification change result of the current iteration exceeds a preset classification threshold, the embedding space offset of the current iteration exceeds a preset instability value, or the number of perturbations in the current iteration is not less than the preset maximum number of perturbations, then the preset stopping condition is determined to have been met; if the classification change result of the current iteration does not exceed the preset classification threshold, the embedding space offset of the current iteration does not exceed the preset instability value, and the number of perturbations in the current iteration is less than the preset maximum number of perturbations, then the preset stopping condition is determined not to have been met.

[0016] In one embodiment, the robustness evaluation result of generating the power system AI model to be evaluated based on the model output and the disturbance output includes:

[0017] Obtain the total number of iterations and the disturbance intensity during the last iteration's disturbance processing; determine the model robustness score based on the preset instability value, the disturbance intensity, the model output, and the total number of iterations; determine the model performance index based on the model output and the disturbance output; generate a visualization result based on the model performance index; and determine the robustness evaluation result of the power system AI model to be evaluated based on the model robustness score and the visualization result.

[0018] In one embodiment, determining the model performance index based on the model output and the perturbation output includes:

[0019] Based on the classification results in the model output and the classification results in the perturbation output, the consistency of classification decisions is determined; based on the confidence results in the model output and the confidence results in the perturbation output, the smoothness of the predicted probability distribution is determined; based on the semantic features of the hidden layer corresponding to the model output and the semantic features of the hidden layer corresponding to the perturbation output, the stability coefficient is determined; based on the accuracy corresponding to the model output and the accuracy corresponding to the perturbation output, the robustness degradation index is determined; based on the accuracy corresponding to the model output, the accuracy corresponding to the perturbation output, the statistical error limit of the data perturbation strategy, and the number of data perturbation strategies, the sensitivity multidimensional comparative analysis is determined; the consistency of classification decisions, the smoothness of the predicted probability distribution, the stability coefficient, the robustness degradation index, and the sensitivity multidimensional comparative analysis constitute the model performance indicators.

[0020] Secondly, this application also provides a robustness evaluation device for an AI model of a power system, comprising:

[0021] The module is used to build basic evaluation data based on the original power system flow samples;

[0022] The input module is used to input the basic evaluation data into the power system AI model to be evaluated and obtain the model output;

[0023] The disturbance module is used to perform iterative disturbance processing on the basic evaluation data based on the model output and the power system AI model to be evaluated, through a data disturbance strategy and an adaptive closed-loop disturbance strategy, to obtain the disturbance output.

[0024] The evaluation module is used to generate robustness evaluation results for the power system AI model to be evaluated based on the model output and the disturbance output.

[0025] Thirdly, this application also provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to perform the following steps:

[0026] Based on the original power system flow samples, basic evaluation data is constructed;

[0027] The basic evaluation data is input into the AI ​​model of the power system to be evaluated to obtain the model output;

[0028] Based on the model output and the power system AI model to be evaluated, the basic evaluation data is iteratively perturbed using a data perturbation strategy and an adaptive closed-loop perturbation strategy to obtain the perturbation output.

[0029] Based on the model output and the disturbance output, robustness evaluation results are generated for the power system AI model to be evaluated.

[0030] Fourthly, this application also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, performs the following steps:

[0031] Based on the original power system flow samples, basic evaluation data is constructed;

[0032] The basic evaluation data is input into the AI ​​model of the power system to be evaluated to obtain the model output;

[0033] Based on the model output and the power system AI model to be evaluated, the basic evaluation data is iteratively perturbed using a data perturbation strategy and an adaptive closed-loop perturbation strategy to obtain the perturbation output.

[0034] Based on the model output and the disturbance output, robustness evaluation results are generated for the power system AI model to be evaluated.

[0035] Fifthly, this application also provides a computer program product, including a computer program that, when executed by a processor, performs the following steps:

[0036] Based on the original power system flow samples, basic evaluation data is constructed;

[0037] The basic evaluation data is input into the AI ​​model of the power system to be evaluated to obtain the model output;

[0038] Based on the model output and the power system AI model to be evaluated, the basic evaluation data is iteratively perturbed using a data perturbation strategy and an adaptive closed-loop perturbation strategy to obtain the perturbation output.

[0039] Based on the model output and the disturbance output, robustness evaluation results are generated for the power system AI model to be evaluated.

[0040] The aforementioned robustness evaluation method, apparatus, computer equipment, computer-readable storage medium, and computer program product for power system AI models construct basic evaluation data based on original power system flow samples; input the basic evaluation data into the power system AI model to be evaluated to obtain model output; based on the model output and the power system AI model to be evaluated, iteratively perturb the basic evaluation data using a data perturbation strategy and an adaptive closed-loop perturbation strategy to obtain perturbation output; and generate robustness evaluation results for the power system AI model to be evaluated based on the model output and the perturbation output. This application uses a unique data perturbation strategy to iteratively perturb the dataset, which can simulate real perturbation behavior under complex communication environments in power systems. Attached Figure Description

[0041] To more clearly illustrate the technical solutions in the embodiments of this application or related technologies, the drawings used in the description of the embodiments of this application or related technologies will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0042] Figure 1 This is a flowchart illustrating a robustness evaluation method for an AI model of a power system in one embodiment.

[0043] Figure 2 This is a schematic diagram of the iterative perturbation process in one embodiment;

[0044] Figure 3 This is a structural block diagram of a robustness evaluation device for an AI model of a power system in one embodiment;

[0045] Figure 4 This is an internal structural diagram of a computer device in one embodiment. Detailed Implementation

[0046] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0047] It should be noted that the terms "first," "second," etc., used in this application can be used to describe various elements, but these elements are not limited by these terms. These terms are only used to distinguish the first element from the second element. The terms "comprising" and "having," and any variations thereof, used in this application, are intended to cover non-exclusive inclusion. The term "multiple" used in this application refers to two or more. The term "and / or" used in this application refers to one of the embodiments, or any combination of multiple embodiments.

[0048] In one embodiment, such as Figure 1 As shown, a robustness evaluation method for an AI model of a power system is provided. This embodiment illustrates the method by applying it to a terminal. It is understood that this method can also be applied to a server, and further to a system including both a terminal and a server, and implemented through interaction between the terminal and the server. In this embodiment, the method includes the following steps:

[0049] Step 102: Construct basic evaluation data based on the original power system flow samples.

[0050] This involves collecting and constructing raw traffic samples that conform to the specific communication specifications and business logic of the power system. These samples cover typical business scenarios in the power production control area and the information management area, including but not limited to: dispatch control traffic (such as IEC 60870-5-104 (International Electrotechnical Commission 60870-5-104) telemetry communication protocol, GOOSE (Generic Object Oriented Substation Event) substation event messages oriented towards generic objects), real-time monitoring traffic (such as MMS (Manufacturing Message Specification) manufacturing message specification, SV (Sampled Value) sampled value messages), and production business management traffic. To achieve a deep analysis of the model's robustness, this invention, based on the extraction of conventional traffic features, focuses on constructing a business semantic association feature system, specifically including: Refined protocol statistical features: In addition to message length distribution, traffic mean, and variance, it also includes the distribution of specific fields based on the power protocol payload, such as the type identifier distribution of ASDU (Application Service Data Unit) and the frequency of COT (Cause of Transmission), used to accurately characterize the static fingerprint of power services. Strongly constrained timing features: In addition to the IAT (Inter-Arrival Time) packet interval time series and session duration, it also introduces periodic task step size features; for example, for sampled value packets (SV) or periodically uploaded measurements, its strict time-interval constraint characteristics are extracted as a benchmark for subsequent evaluation of the model's sensitivity to timing jitter. Business behavior logic characteristics: Deeply explore the business semantic logic behind traffic, mainly including command-response closed-loop characteristics, communication stage state machine characteristics, and message causal relationship characteristics. Among them, the command-response closed-loop characteristics are used to extract the logical pairing relationship between scheduling commands (such as remote control selection / execution) and return confirmation messages. The communication stage state machine characteristics are used to depict the dynamic state transition sequence of power equipment from connection establishment, initialization, data synchronization to periodic transmission. The message causal relationship characteristics are used to describe the causal chain reaction at the business level, such as the SOE (Sequence of Events) event sequence record traffic, which is generated after the protection action message is triggered.

[0051] Step 104: Input the basic evaluation data into the power system AI model to be evaluated to obtain the model output.

[0052] This involves acquiring a power system AI model that requires robustness testing, such as a traffic classification AI model, which is used to classify and identify power system communication traffic. The specific structure, input feature format, and training method of this model are not limited to ensure the universal adaptability of the evaluation method of this invention to different models.

[0053] Optionally, the model output includes: Decision-end results: Extracting the final classification label and Softmax normalized probability distribution from the model output to evaluate the stability and consistency of the model's external discrimination results under both original and subsequent perturbed traffic samples. Hidden layer semantic representation: Capturing high-dimensional feature activation vectors from the key intermediate hidden layers of the model to characterize the distribution differences between the original and subsequent perturbed traffic samples in the feature space, quantifying the alignment and offset of the model's internal semantic representation under complex perturbation conditions.

[0054] Step 106: Based on the model output and the power system AI model to be evaluated, the basic evaluation data is iteratively perturbed using a data perturbation strategy and an adaptive closed-loop perturbation strategy to obtain the perturbation output.

[0055] The process begins by perturbing the basic evaluation data, then inputting the perturbed data into the model to obtain the perturbed output. The output includes both the decision-making results and the semantic representation of the hidden layer.

[0056] Step 108: Based on the model output and the disturbance output, generate the robustness evaluation results of the power system AI model to be evaluated.

[0057] Optionally, robustness evaluation results may include robustness metrics and visualizations.

[0058] The aforementioned robustness evaluation method, apparatus, computer equipment, computer-readable storage medium, and computer program product for power system AI models construct basic evaluation data based on original power system flow samples; input the basic evaluation data into the power system AI model to be evaluated to obtain model output; based on the model output and the power system AI model to be evaluated, iteratively perturb the basic evaluation data using a data perturbation strategy and an adaptive closed-loop perturbation strategy to obtain perturbation output; and generate robustness evaluation results for the power system AI model to be evaluated based on the model output and the perturbation output. This application uses a unique data perturbation strategy to iteratively perturb the dataset, which can simulate real perturbation behavior under complex communication environments in power systems.

[0059] In one exemplary embodiment, such as Figure 2As shown, based on the model output and the power system AI model to be evaluated, the basic evaluation data is iteratively perturbed using a data perturbation strategy and an adaptive closed-loop perturbation strategy to obtain a perturbation output. One iteration includes:

[0060] Obtain the embedding space offset from the previous iteration; wherein the embedding space offset of the first iteration is a random initial value; based on the embedding space offset, determine the perturbation intensity of the current iteration through an adaptive closed-loop perturbation strategy; based on the perturbation intensity of the current iteration, perturb the perturbation dataset of the previous iteration through a data perturbation strategy to obtain the perturbation dataset of the current iteration; input the perturbation dataset of the current iteration into the power system AI model to be evaluated to obtain the perturbation result of the current iteration; based on the model output and the perturbation result, determine the embedding space offset of the current iteration; based on the model output and the perturbation result, determine whether a preset stopping condition has been met; if the preset stopping condition has not been met, start the next iteration perturbation process; if the preset stopping condition has been met, stop the iteration perturbation process and use the perturbation result of the current iteration as the perturbation output.

[0061] The adaptive closed-loop perturbation strategy establishes a feedback adjustment mechanism based on the model's internal representation. After inputting the perturbation flow sample into the model to be evaluated, the feature embedding information of the model's intermediate hidden layers is extracted in real time. The representation offset between the original flow sample and the perturbation flow sample in the feature space is calculated (this offset can be an embedding space offset, using Euclidean distance or cosine offset, etc.). This representation offset is used as a feedback signal to dynamically optimize the direction and intensity of subsequent perturbations, gradually guiding the model into the "semantic ambiguity region" or "feature distribution edge" region. Specifically, after generating the perturbation flow sample, the original flow sample and the perturbation flow sample are input into the model to be evaluated respectively. The feature embedding vector of the specified intermediate hidden layer of the model is extracted. The original flow sample embedding is denoted as... The perturbation flow sample embedding is represented as The feedback signal is calculated by taking the offset between the two elements in the embedding space, specifically expressed as follows:

[0062]

[0063] in, Indicates the first The offset is represented after the first perturbation. The intensity of the next perturbation is dynamically adjusted based on this offset, with the update rule as follows:

[0064]

[0065] in: This represents the current disturbance intensity; This is the feedback adjustment coefficient. When the offset is large, the disturbance intensity is increased; when the offset is small, the disturbance amplitude is maintained or decreased.

[0066] For example, first, the embedding space offset calculated in the previous round is obtained. If it is the first iteration, a random initial offset is used. Next, based on the magnitude of this offset, an adaptive closed-loop perturbation strategy automatically determines the appropriate perturbation strength for this round: a small offset indicates the model is relatively stable, so the perturbation is appropriately increased; a large offset indicates the model is already sensitive, so the perturbation strength is maintained or fine-tuned. After determining the perturbation strength for this round, the system uses a data perturbation strategy to further adjust the perturbation data obtained in the previous round, generating new perturbation data for this round. Then, this new perturbation data is input into the AI ​​model to be evaluated, obtaining the model's classification result under the perturbation data, which is the perturbation result for this round. Afterwards, the model output and the perturbation result obtained in this round are compared to calculate the embedding space offset reflecting the degree of change in the model's internal features. Finally, determine whether the stopping condition is met. If the stopping condition is not met, continue to the next round of iteration using the result of this round, and continuously strengthen the perturbation; if the stopping condition is met, stop the iteration and take the perturbation result obtained in the last round as the final perturbation output.

[0067] In this embodiment, a closed-loop feedback adjustment logic is constructed by embedding spatial offsets, and iterative data perturbation is carried out by dynamically adapting the perturbation parameters. This not only conforms to the actual operation scenario of power business to complete the model stability detection, but also effectively identifies metastable problems with abnormal internal model representation, thus comprehensively improving the accuracy and completeness of robustness evaluation.

[0068] In an exemplary embodiment, the step of perturbing the perturbation dataset of the previous iteration based on the perturbation strength of the current iteration using a data perturbation strategy to obtain the perturbation dataset of the current iteration includes:

[0069] If the data perturbation strategy includes at least one of the semantic constraint perturbation strategy, the temporal perturbation strategy, and the joint behavior perturbation strategy, then based on the perturbation strength of the current iteration and the data perturbation strategy, the perturbation dataset of the previous iteration is perturbed sequentially in a preset order to obtain the perturbation dataset of the current iteration; if the data perturbation strategy includes any one of the semantic constraint perturbation strategy, the temporal perturbation strategy, and the joint behavior perturbation strategy, then based on the perturbation strength of the current iteration and the data perturbation strategy, the perturbation dataset of the previous iteration is perturbed to obtain the perturbation dataset of the current iteration.

[0070] 1. Semantic Constraint Feature Perturbation Strategy: This strategy is based on Deep Packet Inspection (DPI) technology for power communication, constructing standardized semantic templates for power business behavior, and strictly adhering to power-specific communication specifications and business logic. During perturbation generation, it strictly ensures the TCP / IP verification legitimacy of traffic samples, the integrity of protocol field dependencies, and the validity of power application layer business logic. For example, it adheres to core business rules such as the fixed heartbeat cycle of Generic Object Oriented Substation Event (GOOSE) messages. Under these constraints, a limited perturbation function is used to make small, controllable, and refined adjustments to traffic statistical features. This strategy mainly simulates real-world power field conditions, reproducing the drift in traffic statistical features caused by factors such as slight network jitter, link aging, and background traffic interference. Relying on the concealed perturbation characteristics of "unchanged business semantics and slight feature shift," it can accurately evaluate the ability of power AI models to identify micro-perturbations in legitimate power business scenarios, effectively avoiding evaluation bias caused by invalid perturbation traffic samples. 2. Progressive Temporal Perturbation Strategy (Temporal Perturbation Strategy): This strategy employs a step-by-step gradient perturbation algorithm, with the time dimension as the core perturbation direction. It dynamically adjusts core temporal features of power flow, such as the inter-arrival time (IAT), session duration, and local packet timing arrangement. By setting the gradient perturbation coefficient ϵ, it generates multi-gradient, staged perturbation flow samples ranging from slight timing jitter to severe delay offset. Unlike traditional single-intensity temporal perturbation methods, this strategy uses a stepped, incremental perturbation intensity design, allowing the perturbation coefficient ϵ to gradually increase with each iteration. At each perturbation intensity level, it collects model inference results and feature state data, fully recording the trajectory of model performance changes. Based on multi-gradient perturbation data, a model performance degradation analysis system is constructed. This system can quantitatively analyze the evolution of the model's discriminative ability as the temporal perturbation intensity increases, accurately identify the model's deficiencies in learning time-dependent features, determine the robustness failure threshold of the model's temporal dimension, and achieve a refined evaluation of the model's temporal stability. 3. Multidimensional Coupled Joint Behavior Disturbance Strategy (Joint Behavior Disturbance Strategy): This strategy constructs a multidimensional coupled disturbance matrix to achieve synchronous and coordinated disturbance of flow statistics characteristics, time-series correlation characteristics, and legitimate power business behaviors. During the disturbance process, compliant redundant communication behaviors such as legitimate idle query messages and equipment status report messages are inserted between power dispatch command exchanges to simulate the nonlinear flow change characteristics caused by the superposition of multiple factors in a real complex power grid.Traditional single-dimensional disturbances can only cover local and simple operating conditions and cannot reproduce complex coupled interference scenarios in the power grid. This strategy constructs complex power network disturbance scenarios with high concealment and high realism through coupled composite disturbances with multiple features and behaviors. It fully exposes the hidden loopholes and identification defects of the model in learning the multi-dimensional feature association logic and processing complex coupled flow data. It effectively solves the technical problems of incomplete coverage of single-disturbance dimension evaluation scenarios and one-sided evaluation results, and comprehensively improves the authenticity and completeness of robustness evaluation.

[0071] For example, when performing data perturbation, the corresponding operation is executed according to the selected perturbation strategy type. When two or more perturbation methods are enabled simultaneously, the perturbation dataset generated in the previous round is perturbed one by one in strict order of semantic constraint perturbation, temporal perturbation, and joint behavior perturbation, combined with the perturbation strength determined in this round, to form the perturbation dataset of this round. If only a single perturbation strategy is selected, the perturbation dataset generated in the previous round is perturbed in combination with the perturbation strength determined in this round to form the perturbation dataset of this round.

[0072] In this embodiment, single or multiple disturbance strategies are flexibly selected according to a predetermined order, and the dataset is modified layer by layer in combination with the real-time determined disturbance intensity. This enables the simulation of real power grid disturbance scenarios from multiple dimensions such as business semantics, temporal characteristics, and linkage behavior, making the evaluation process fit the actual operating conditions and comprehensively testing the model's ability to identify different types of flow anomalies.

[0073] In an exemplary embodiment, determining whether a preset stopping condition has been met based on the model output and the perturbation result includes:

[0074] Based on the model output and the perturbation result, the classification change result of the current iteration is determined; if the classification change result of the current iteration exceeds a preset classification threshold, the embedding space offset of the current iteration exceeds a preset instability value, or the number of perturbations in the current iteration is not less than the preset maximum number of perturbations, then the preset stopping condition is determined to have been met; if the classification change result of the current iteration does not exceed the preset classification threshold, the embedding space offset of the current iteration does not exceed the preset instability value, and the number of perturbations in the current iteration is less than the preset maximum number of perturbations, then the preset stopping condition is determined not to have been met.

[0075] Among them, the classification change result exceeding the preset classification threshold means that the rate of change between the classification result after perturbation and the classification result of the original data exceeds the preset classification threshold.

[0076] For example, firstly, the model output is compared with the results of this round of perturbation to statistically determine the classification change results of this iteration. Then, three judgment indicators are checked. If any of the following occurs: the classification change result exceeds a set threshold, the feature embedding offset exceeds the instability threshold, or the number of iterations reaches the specified upper limit, the stopping requirements are met, and the current iteration process ends. If the classification change result and the feature embedding offset are both within the allowable range, and the number of iterations has not reached the maximum value, the stopping conditions are not triggered, and the next round of perturbation iteration operation continues.

[0077] In this embodiment, the iteration stopping condition is determined jointly by three dimensions: classification result bias, internal feature shift of the model, and maximum number of iterations, abandoning the traditional method of relying solely on classification results. This approach can accurately capture explicit defects such as external classification failures and identify implicit metastability issues such as normal model classification but unstable internal feature shifts. Furthermore, the upper limit on the number of iterations constrains the entire evaluation process, effectively avoiding invalid iterations. This multi-constraint determination mechanism makes the perturbation iteration process controllable, accurate, and comprehensive, enabling a complete and realistic exploration of the robustness boundaries of power system AI models, significantly improving the reliability and completeness of model stability evaluation.

[0078] In an exemplary embodiment, the robustness evaluation result of the power system AI model to be evaluated, based on the model output and the disturbance output, includes:

[0079] Obtain the total number of iterations and the disturbance intensity during the last iteration's disturbance processing; determine the model robustness score based on the preset instability value, the disturbance intensity, the model output, and the total number of iterations; determine the model performance index based on the model output and the disturbance output; generate a visualization result based on the model performance index; and determine the robustness evaluation result of the power system AI model to be evaluated based on the model robustness score and the visualization result.

[0080] Optionally, the robustness score integrates the model's classification accuracy on the original traffic samples and introduces an adaptive closed-loop perturbation to the point where the model collapses (the perturbation intensity during the last iteration of perturbation processing). And the maximum number of iterations (total number of iterations). Specifically:

[0081]

[0082] in, The final overall robustness score is normalized to a range of values. ; The classification accuracy of the model on the original undisturbed flow sample set, with a value range of [value missing]. ; The normalized limit perturbation threshold score is calculated as follows: ,in This represents the final perturbation intensity when the model triggers the termination condition. The maximum allowable disturbance limit boundary is preset for the system; The normalized iteration score is calculated as follows: ,in This represents the actual iteration number when the model triggers the termination condition. The maximum number of iterations is preset for the system. , , These are the weighting coefficients for the original accuracy, the extreme perturbation threshold, and the continuous adversarial rounds, respectively, and satisfy the following conditions: .

[0083] In addition, the visualization results include: 1. Performance change trends under different perturbation intensities: This result is specifically used to output the classification accuracy and decision confidence of the model as the perturbation intensity dynamically adjusts during the adaptive iteration process of perturbation-inference-feedback-reperturbation. The dynamic degradation trajectory, which increases with time, clearly maps the complete performance evolution range of the model from "steady state" to "collapse"; 2. Represents the results of stability analysis. : Utilizing the embedding space offset of each iteration 1. Output the cumulative offset path of key intermediate hidden layer feature vectors in the latent space in the form of multidimensional data or visualized trajectories; 2. Sensitivity analysis results (sensitivity multidimensional comparative analysis): By constructing a sensitivity matrix, compare and analyze the contribution weight of different strategies such as semantic constraint feature perturbation, progressive temporal perturbation, and multidimensional coupling joint behavior perturbation to the feature offset (embedding space offset) inside the model during the adaptive approximation of the model failure boundary; 3. Robustness degradation curve, used to reflect the curve of continuous change in model recognition performance as the perturbation intensifies and the iteration progresses.

[0084] For example, the total number of iterations and the disturbance intensity during the last iteration's disturbance processing are obtained; based on a preset instability value, the disturbance intensity, the model output, and the total number of iterations, a model robustness score is determined; based on the model output and the disturbance output, a model performance index is determined; based on the model performance index, a visualization result is generated, specifically including: performance change trends under different disturbance intensities, stability analysis results, sensitivity analysis results, and robustness degradation curves. Based on the model robustness score and the visualization result, the robustness evaluation result of the power system AI model to be evaluated is determined.

[0085] In this embodiment, a quantitative robustness scoring system is constructed by integrating the iterative limit perturbation intensity, effective iteration rounds, and original recognition accuracy, thereby achieving a quantitative and unified evaluation of the model's anti-perturbation capability. At the same time, by combining multi-dimensional model performance indicators, visualization results such as performance change trends, characterization stability analysis, perturbation sensitivity analysis, and robustness degradation curves are generated, accurately demonstrating the robustness results of the model.

[0086] In an exemplary embodiment, determining the model performance metric based on the model output and the perturbation output includes:

[0087] Based on the classification results in the model output and the classification results in the perturbation output, the consistency of classification decisions is determined; based on the confidence results in the model output and the confidence results in the perturbation output, the smoothness of the predicted probability distribution is determined; based on the semantic features of the hidden layer corresponding to the model output and the semantic features of the hidden layer corresponding to the perturbation output, the stability coefficient is determined; based on the accuracy corresponding to the model output and the accuracy corresponding to the perturbation output, the robustness degradation index is determined; based on the accuracy corresponding to the model output, the accuracy corresponding to the perturbation output, the statistical error limit of the data perturbation strategy, and the number of data perturbation strategies, the sensitivity multidimensional comparative analysis is determined; the consistency of classification decisions, the smoothness of the predicted probability distribution, the stability coefficient, the robustness degradation index, and the sensitivity multidimensional comparative analysis constitute the model performance indicators.

[0088] Among them, classification decision consistency measures the proportion of original flow samples and disturbed flow samples whose classification results are consistent. Predictive probability distribution smoothness quantifies the fluctuation of model confidence by calculating the KL divergence (Kullback-Leibler Divergence) or JS divergence of the model output vector before and after the disturbance. This metric aims to capture "metastable" samples that, although correctly classified, have experienced drastic fluctuations in confidence, providing a probabilistic basis for real-time early warning of power systems. Deep semantic representation stability coefficient (stability coefficient) characterizes the offset trajectory of the hidden layer feature space under disturbance, including hidden layer feature distance, representation offset, and representation consistency. For example, the representation stability coefficient is defined as:

[0089]

[0090] in: This represents the hidden layer features of the original traffic sample. This represents the hidden layer features of the perturbation flow sample. This introduces norm-normalized Euclidean or cosine distance. This metric assesses whether a perturbation has affected the model's core semantic representation by measuring the shift in activation values ​​of hidden layer neurons. Compared to simply observing the output, this metric can detect distortions in the model's internal logic earlier, achieving a more accurate characterization of the model's "failure latency." Robustness degradation metrics: used to analyze the degradation trend of model performance with increasing perturbation intensity. For example, the robustness degradation rate is defined as:

[0091]

[0092] in: The classification accuracy of the original traffic samples; The classification accuracy under perturbation conditions; This represents the change in disturbance intensity. This is the semantic consistency score for perturbed traffic samples. This metric not only reflects the rate of accuracy decline but also, by introducing semantic constraint weights, distinguishes between "performance degradation due to changes in business logic" and "collapse due to insufficient robustness of the model itself." Through analysis... The nonlinear changes (such as the presence of a sudden drop inflection point) are used to determine the model's safety margin threshold. Sensitivity multidimensional comparative analysis: Combining statistical inference and confidence interval analysis (e.g., 95% confidence level), the impact weights of different perturbation strategies (time-series perturbation vs. behavioral perturbation) on the above indicators are compared. By constructing a sensitivity matrix, key feature dimensions that lead to model performance degradation are identified, providing decision support for subsequent targeted model hardening (such as adversarial training and feature cleaning). For example, for a specific feature dimension... Sensitivity weights (such as those in the time series dimension) Defined as:

[0093]

[0094] in: For the first The statistical mean of model performance degradation under various perturbation strategies; This is the statistical error limit calculated at a 95% confidence level; To evaluate the total number of perturbation strategies used, this calculation method uses the lower bound of the confidence interval. As a deterministic degradation measure, it rigorously eliminates accidental noise interference caused by random delays or packet loss in power communication networks. This indicator not only quantifies the absolute destructive power of different disturbance dimensions but also ensures the statistical significance of the evaluation results. This is achieved by analyzing the various dimensions of the sensitivity matrix. If the weight of a certain dimension is abnormally high, the vulnerability of the model can be accurately located, thereby guiding the system to deploy the corresponding gateway at the front end of the model or to focus on generating extreme samples of that dimension.

[0095] For example, based on the classification results in the model output and the classification results in the perturbation output, the proportion of identical classification results is determined to obtain classification decision consistency; based on the confidence results in the model output and the confidence results in the perturbation output, the proportion of identical prediction results but confidence differences exceeding a preset gap is determined to obtain prediction probability distribution smoothness; based on the hidden layer semantic features corresponding to the model output and the hidden layer semantic features corresponding to the perturbation output, the proportion of embedding space offset exceeding a preset offset is determined to obtain a stability coefficient; based on the accuracy corresponding to the model output and the accuracy corresponding to the perturbation output, the degradation trend of model performance with increasing perturbation intensity is determined to obtain a robustness degradation index; based on the accuracy corresponding to the model output, the accuracy corresponding to the perturbation output, the statistical error limit of the data perturbation strategy, and the number of data perturbation strategies, the influence weight of different perturbation strategies on the preset confidence is determined, and a sensitivity multidimensional comparative analysis is performed; the classification decision consistency, the prediction probability distribution smoothness, the stability coefficient, the robustness degradation index, and the sensitivity multidimensional comparative analysis constitute the model performance index.

[0096] In this embodiment, a complete model performance index system is constructed from five dimensions: classification decision, probability distribution, hidden layer representation, performance degradation, and multi-dimensional sensitivity. This system can simultaneously take into account the model's external decision performance and internal semantic representation state, comprehensively depicting the dynamic performance change law of the power system AI model under multi-level and multi-type disturbances. It effectively makes up for the problem that a single accuracy evaluation dimension is one-sided and cannot reflect the implicit instability of the model, and greatly improves the refinement and comprehensive credibility of robustness evaluation.

[0097] In an exemplary embodiment, a robustness evaluation method for a power system AI model includes: constructing basic evaluation data based on original power system flow samples; inputting the basic evaluation data into the power system AI model to be evaluated to obtain model output; obtaining the embedding space offset of the previous iteration; wherein the embedding space offset of the first iteration is a random initial value; determining the perturbation intensity of the current iteration based on the embedding space offset using an adaptive closed-loop perturbation strategy; if the data perturbation strategy includes at least one of a semantic constraint perturbation strategy, a temporal perturbation strategy, and a joint behavior perturbation strategy, then based on the perturbation intensity of the current iteration and the data perturbation strategy, perturbing the perturbation dataset of the previous iteration in a preset order to obtain the perturbation dataset of the current iteration; if the data perturbation strategy includes any one of a semantic constraint perturbation strategy, a temporal perturbation strategy, and a joint behavior perturbation strategy, then based on the perturbation intensity of the current iteration and the data perturbation strategy, perturbing the perturbation dataset of the previous iteration to obtain the perturbation dataset of the current iteration. The disturbance dataset of the current iteration is input into the power system AI model to be evaluated to obtain the disturbance result of the current iteration. Based on the model output and the disturbance result, the embedding space offset of the current iteration is determined. Based on the model output and the disturbance result, the classification change result of the current iteration is determined. If the classification change result of the current iteration exceeds a preset classification threshold, the embedding space offset of the current iteration exceeds a preset instability value, or the number of disturbances in the current iteration is not less than a preset maximum number of disturbances, then it is determined that the preset stopping condition has been met. If the classification change result of the current iteration does not exceed the preset classification threshold, the embedding space offset of the current iteration does not exceed the preset instability value, and the number of disturbances in the current iteration is less than the preset maximum number of disturbances, then it is determined that the preset stopping condition has not been met. If the preset stopping condition has not been met, the next iteration disturbance process begins. If the preset stopping condition is met, the iteration disturbance process is stopped, and the disturbance result of the current iteration is used as the disturbance output.The process involves obtaining the total number of iterations and the perturbation intensity during the last iteration's perturbation processing; determining the model robustness score based on a preset instability value, the perturbation intensity, the model output, and the total number of iterations; determining classification decision consistency based on the classification results in the model output and the perturbation output; determining the predictive probability distribution smoothness based on the confidence results in the model output and the perturbation output; determining the stability coefficient based on the hidden layer semantic features corresponding to the model output and the hidden layer semantic features corresponding to the perturbation output; determining the robustness degradation index based on the accuracy corresponding to the model output and the accuracy corresponding to the perturbation output; and determining the sensitivity multidimensional comparative analysis based on the accuracy corresponding to the model output, the accuracy corresponding to the perturbation output, the statistical error limit of the data perturbation strategy, and the number of data perturbation strategies. The classification decision consistency, the predictive probability distribution smoothness, the stability coefficient, the robustness degradation index, and the sensitivity multidimensional comparative analysis constitute the model performance index. Based on the model performance index, a visualization result is generated; and based on the model robustness score and the visualization result, the robustness evaluation result of the power system AI model to be evaluated is determined.

[0098] It should be understood that although the steps in the flowcharts of the embodiments described above are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the embodiments described above may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages in other steps. It is understood that the steps in different embodiments can be freely combined as needed, and all non-contradictory solutions formed by such combinations are within the scope of protection of this application.

[0099] In one exemplary embodiment, such as Figure 3 As shown, a robustness evaluation device for an AI model of a power system is provided, comprising: a construction module 301, an input module 302, a disturbance module 303, and an evaluation module 304, wherein:

[0100] The module is used to build basic evaluation data based on the original power system flow samples;

[0101] The input module is used to input the basic evaluation data into the power system AI model to be evaluated and obtain the model output;

[0102] The disturbance module is used to perform iterative disturbance processing on the basic evaluation data based on the model output and the power system AI model to be evaluated, through a data disturbance strategy and an adaptive closed-loop disturbance strategy, to obtain the disturbance output.

[0103] The evaluation module is used to generate robustness evaluation results for the power system AI model to be evaluated based on the model output and the disturbance output.

[0104] In one embodiment, the disturbance module is further configured to:

[0105] Obtain the embedding space offset from the previous iteration; wherein the embedding space offset of the first iteration is a random initial value; based on the embedding space offset, determine the perturbation intensity of the current iteration through an adaptive closed-loop perturbation strategy; based on the perturbation intensity of the current iteration, perturb the perturbation dataset of the previous iteration through a data perturbation strategy to obtain the perturbation dataset of the current iteration; input the perturbation dataset of the current iteration into the power system AI model to be evaluated to obtain the perturbation result of the current iteration; based on the model output and the perturbation result, determine the embedding space offset of the current iteration; based on the model output and the perturbation result, determine whether a preset stopping condition has been met; if the preset stopping condition has not been met, start the next iteration perturbation process; if the preset stopping condition has been met, stop the iteration perturbation process and use the perturbation result of the current iteration as the perturbation output.

[0106] In one embodiment, the disturbance module is further configured to:

[0107] If the data perturbation strategy includes at least one of the semantic constraint perturbation strategy, the temporal perturbation strategy, and the joint behavior perturbation strategy, then based on the perturbation strength of the current iteration and the data perturbation strategy, the perturbation dataset of the previous iteration is perturbed sequentially in a preset order to obtain the perturbation dataset of the current iteration; if the data perturbation strategy includes any one of the semantic constraint perturbation strategy, the temporal perturbation strategy, and the joint behavior perturbation strategy, then based on the perturbation strength of the current iteration and the data perturbation strategy, the perturbation dataset of the previous iteration is perturbed to obtain the perturbation dataset of the current iteration.

[0108] In one embodiment, the disturbance module is further configured to:

[0109] Based on the model output and the perturbation result, the classification change result of the current iteration is determined; if the classification change result of the current iteration exceeds a preset classification threshold, the embedding space offset of the current iteration exceeds a preset instability value, or the number of perturbations in the current iteration is not less than the preset maximum number of perturbations, then the preset stopping condition is determined to have been met; if the classification change result of the current iteration does not exceed the preset classification threshold, the embedding space offset of the current iteration does not exceed the preset instability value, and the number of perturbations in the current iteration is less than the preset maximum number of perturbations, then the preset stopping condition is determined not to have been met.

[0110] In one embodiment, the evaluation module is further configured to:

[0111] Obtain the total number of iterations and the disturbance intensity during the last iteration's disturbance processing; determine the model robustness score based on the preset instability value, the disturbance intensity, the model output, and the total number of iterations; determine the model performance index based on the model output and the disturbance output; generate a visualization result based on the model performance index; and determine the robustness evaluation result of the power system AI model to be evaluated based on the model robustness score and the visualization result.

[0112] In one embodiment, the evaluation module is further configured to:

[0113] Based on the classification results in the model output and the classification results in the perturbation output, the consistency of classification decisions is determined; based on the confidence results in the model output and the confidence results in the perturbation output, the smoothness of the predicted probability distribution is determined; based on the semantic features of the hidden layer corresponding to the model output and the semantic features of the hidden layer corresponding to the perturbation output, the stability coefficient is determined; based on the accuracy corresponding to the model output and the accuracy corresponding to the perturbation output, the robustness degradation index is determined; based on the accuracy corresponding to the model output, the accuracy corresponding to the perturbation output, the statistical error limit of the data perturbation strategy, and the number of data perturbation strategies, the sensitivity multidimensional comparative analysis is determined; the consistency of classification decisions, the smoothness of the predicted probability distribution, the stability coefficient, the robustness degradation index, and the sensitivity multidimensional comparative analysis constitute the model performance indicators.

[0114] Each module in the robustness evaluation device for the aforementioned power system AI model can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device, or stored in the memory of a computer device as software, so that the processor can call and execute the operations corresponding to each module.

[0115] In one exemplary embodiment, a computer device is provided, which may be a server, and its internal structure diagram may be as follows: Figure 4As shown, this computer device includes a processor, memory, input / output (I / O) interfaces, and a communication interface. The processor, memory, and I / O interfaces are connected via a system bus, and the communication interface is also connected to the system bus via the I / O interfaces. The processor provides computational and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system, computer programs, and a database. The internal memory provides the environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The database stores power system flow samples. The I / O interfaces are used for information exchange between the processor and external devices. The communication interface is used for communication with external terminals via a network connection. When the computer program is executed by the processor, it implements a robustness evaluation method for a power system AI model.

[0116] Those skilled in the art will understand that Figure 4 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0117] In one exemplary embodiment, a computer device is provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to perform the following steps:

[0118] Based on the original power system flow samples, basic evaluation data is constructed;

[0119] The basic evaluation data is input into the AI ​​model of the power system to be evaluated to obtain the model output;

[0120] Based on the model output and the power system AI model to be evaluated, the basic evaluation data is iteratively perturbed using a data perturbation strategy and an adaptive closed-loop perturbation strategy to obtain the perturbation output.

[0121] Based on the model output and the disturbance output, robustness evaluation results are generated for the power system AI model to be evaluated.

[0122] In one embodiment, the processor, when executing a computer program, also performs the following steps:

[0123] Obtain the embedding space offset from the previous iteration; wherein the embedding space offset of the first iteration is a random initial value; based on the embedding space offset, determine the perturbation intensity of the current iteration through an adaptive closed-loop perturbation strategy; based on the perturbation intensity of the current iteration, perturb the perturbation dataset of the previous iteration through a data perturbation strategy to obtain the perturbation dataset of the current iteration; input the perturbation dataset of the current iteration into the power system AI model to be evaluated to obtain the perturbation result of the current iteration; based on the model output and the perturbation result, determine the embedding space offset of the current iteration; based on the model output and the perturbation result, determine whether a preset stopping condition has been met; if the preset stopping condition has not been met, start the next iteration perturbation process; if the preset stopping condition has been met, stop the iteration perturbation process and use the perturbation result of the current iteration as the perturbation output.

[0124] In one embodiment, the processor, when executing a computer program, also performs the following steps:

[0125] If the data perturbation strategy includes at least one of the semantic constraint perturbation strategy, the temporal perturbation strategy, and the joint behavior perturbation strategy, then based on the perturbation strength of the current iteration and the data perturbation strategy, the perturbation dataset of the previous iteration is perturbed sequentially in a preset order to obtain the perturbation dataset of the current iteration; if the data perturbation strategy includes any one of the semantic constraint perturbation strategy, the temporal perturbation strategy, and the joint behavior perturbation strategy, then based on the perturbation strength of the current iteration and the data perturbation strategy, the perturbation dataset of the previous iteration is perturbed to obtain the perturbation dataset of the current iteration.

[0126] In one embodiment, the processor, when executing a computer program, also performs the following steps:

[0127] Based on the model output and the perturbation result, the classification change result of the current iteration is determined; if the classification change result of the current iteration exceeds a preset classification threshold, the embedding space offset of the current iteration exceeds a preset instability value, or the number of perturbations in the current iteration is not less than the preset maximum number of perturbations, then the preset stopping condition is determined to have been met; if the classification change result of the current iteration does not exceed the preset classification threshold, the embedding space offset of the current iteration does not exceed the preset instability value, and the number of perturbations in the current iteration is less than the preset maximum number of perturbations, then the preset stopping condition is determined not to have been met.

[0128] In one embodiment, the processor, when executing a computer program, also performs the following steps:

[0129] Obtain the total number of iterations and the disturbance intensity during the last iteration's disturbance processing; determine the model robustness score based on the preset instability value, the disturbance intensity, the model output, and the total number of iterations; determine the model performance index based on the model output and the disturbance output; generate a visualization result based on the model performance index; and determine the robustness evaluation result of the power system AI model to be evaluated based on the model robustness score and the visualization result.

[0130] In one embodiment, the processor, when executing a computer program, also performs the following steps:

[0131] Based on the classification results in the model output and the classification results in the perturbation output, the consistency of classification decisions is determined; based on the confidence results in the model output and the confidence results in the perturbation output, the smoothness of the predicted probability distribution is determined; based on the semantic features of the hidden layer corresponding to the model output and the semantic features of the hidden layer corresponding to the perturbation output, the stability coefficient is determined; based on the accuracy corresponding to the model output and the accuracy corresponding to the perturbation output, the robustness degradation index is determined; based on the accuracy corresponding to the model output, the accuracy corresponding to the perturbation output, the statistical error limit of the data perturbation strategy, and the number of data perturbation strategies, the sensitivity multidimensional comparative analysis is determined; the consistency of classification decisions, the smoothness of the predicted probability distribution, the stability coefficient, the robustness degradation index, and the sensitivity multidimensional comparative analysis constitute the model performance indicators.

[0132] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon, the computer program performing the following steps when executed by a processor:

[0133] Based on the original power system flow samples, basic evaluation data is constructed;

[0134] The basic evaluation data is input into the AI ​​model of the power system to be evaluated to obtain the model output;

[0135] Based on the model output and the power system AI model to be evaluated, the basic evaluation data is iteratively perturbed using a data perturbation strategy and an adaptive closed-loop perturbation strategy to obtain the perturbation output.

[0136] Based on the model output and the disturbance output, robustness evaluation results are generated for the power system AI model to be evaluated.

[0137] In one embodiment, when the computer program is executed by a processor, it also performs the following steps:

[0138] Obtain the embedding space offset from the previous iteration; wherein the embedding space offset of the first iteration is a random initial value; based on the embedding space offset, determine the perturbation intensity of the current iteration through an adaptive closed-loop perturbation strategy; based on the perturbation intensity of the current iteration, perturb the perturbation dataset of the previous iteration through a data perturbation strategy to obtain the perturbation dataset of the current iteration; input the perturbation dataset of the current iteration into the power system AI model to be evaluated to obtain the perturbation result of the current iteration; based on the model output and the perturbation result, determine the embedding space offset of the current iteration; based on the model output and the perturbation result, determine whether a preset stopping condition has been met; if the preset stopping condition has not been met, start the next iteration perturbation process; if the preset stopping condition has been met, stop the iteration perturbation process and use the perturbation result of the current iteration as the perturbation output.

[0139] In one embodiment, when the computer program is executed by a processor, it also performs the following steps:

[0140] If the data perturbation strategy includes at least one of the semantic constraint perturbation strategy, the temporal perturbation strategy, and the joint behavior perturbation strategy, then based on the perturbation strength of the current iteration and the data perturbation strategy, the perturbation dataset of the previous iteration is perturbed sequentially in a preset order to obtain the perturbation dataset of the current iteration; if the data perturbation strategy includes any one of the semantic constraint perturbation strategy, the temporal perturbation strategy, and the joint behavior perturbation strategy, then based on the perturbation strength of the current iteration and the data perturbation strategy, the perturbation dataset of the previous iteration is perturbed to obtain the perturbation dataset of the current iteration.

[0141] In one embodiment, when the computer program is executed by a processor, it also performs the following steps:

[0142] Based on the model output and the perturbation result, the classification change result of the current iteration is determined; if the classification change result of the current iteration exceeds a preset classification threshold, the embedding space offset of the current iteration exceeds a preset instability value, or the number of perturbations in the current iteration is not less than the preset maximum number of perturbations, then the preset stopping condition is determined to have been met; if the classification change result of the current iteration does not exceed the preset classification threshold, the embedding space offset of the current iteration does not exceed the preset instability value, and the number of perturbations in the current iteration is less than the preset maximum number of perturbations, then the preset stopping condition is determined not to have been met.

[0143] In one embodiment, when the computer program is executed by a processor, it also performs the following steps:

[0144] Obtain the total number of iterations and the disturbance intensity during the last iteration's disturbance processing; determine the model robustness score based on the preset instability value, the disturbance intensity, the model output, and the total number of iterations; determine the model performance index based on the model output and the disturbance output; generate a visualization result based on the model performance index; and determine the robustness evaluation result of the power system AI model to be evaluated based on the model robustness score and the visualization result.

[0145] In one embodiment, when the computer program is executed by a processor, it also performs the following steps:

[0146] Based on the classification results in the model output and the classification results in the perturbation output, the consistency of classification decisions is determined; based on the confidence results in the model output and the confidence results in the perturbation output, the smoothness of the predicted probability distribution is determined; based on the semantic features of the hidden layer corresponding to the model output and the semantic features of the hidden layer corresponding to the perturbation output, the stability coefficient is determined; based on the accuracy corresponding to the model output and the accuracy corresponding to the perturbation output, the robustness degradation index is determined; based on the accuracy corresponding to the model output, the accuracy corresponding to the perturbation output, the statistical error limit of the data perturbation strategy, and the number of data perturbation strategies, the sensitivity multidimensional comparative analysis is determined; the consistency of classification decisions, the smoothness of the predicted probability distribution, the stability coefficient, the robustness degradation index, and the sensitivity multidimensional comparative analysis constitute the model performance indicators.

[0147] In one embodiment, a computer program product is provided, including a computer program that, when executed by a processor, performs the following steps:

[0148] Based on the original power system flow samples, basic evaluation data is constructed;

[0149] The basic evaluation data is input into the AI ​​model of the power system to be evaluated to obtain the model output;

[0150] Based on the model output and the power system AI model to be evaluated, the basic evaluation data is iteratively perturbed using a data perturbation strategy and an adaptive closed-loop perturbation strategy to obtain the perturbation output.

[0151] Based on the model output and the disturbance output, robustness evaluation results are generated for the power system AI model to be evaluated.

[0152] In one embodiment, when the computer program is executed by a processor, it also performs the following steps:

[0153] Obtain the embedding space offset from the previous iteration; wherein the embedding space offset of the first iteration is a random initial value; based on the embedding space offset, determine the perturbation intensity of the current iteration through an adaptive closed-loop perturbation strategy; based on the perturbation intensity of the current iteration, perturb the perturbation dataset of the previous iteration through a data perturbation strategy to obtain the perturbation dataset of the current iteration; input the perturbation dataset of the current iteration into the power system AI model to be evaluated to obtain the perturbation result of the current iteration; based on the model output and the perturbation result, determine the embedding space offset of the current iteration; based on the model output and the perturbation result, determine whether a preset stopping condition has been met; if the preset stopping condition has not been met, start the next iteration perturbation process; if the preset stopping condition has been met, stop the iteration perturbation process and use the perturbation result of the current iteration as the perturbation output.

[0154] In one embodiment, when the computer program is executed by a processor, it also performs the following steps:

[0155] If the data perturbation strategy includes at least one of the semantic constraint perturbation strategy, the temporal perturbation strategy, and the joint behavior perturbation strategy, then based on the perturbation strength of the current iteration and the data perturbation strategy, the perturbation dataset of the previous iteration is perturbed sequentially in a preset order to obtain the perturbation dataset of the current iteration; if the data perturbation strategy includes any one of the semantic constraint perturbation strategy, the temporal perturbation strategy, and the joint behavior perturbation strategy, then based on the perturbation strength of the current iteration and the data perturbation strategy, the perturbation dataset of the previous iteration is perturbed to obtain the perturbation dataset of the current iteration.

[0156] In one embodiment, when the computer program is executed by a processor, it also performs the following steps:

[0157] Based on the model output and the perturbation result, the classification change result of the current iteration is determined; if the classification change result of the current iteration exceeds a preset classification threshold, the embedding space offset of the current iteration exceeds a preset instability value, or the number of perturbations in the current iteration is not less than the preset maximum number of perturbations, then the preset stopping condition is determined to have been met; if the classification change result of the current iteration does not exceed the preset classification threshold, the embedding space offset of the current iteration does not exceed the preset instability value, and the number of perturbations in the current iteration is less than the preset maximum number of perturbations, then the preset stopping condition is determined not to have been met.

[0158] In one embodiment, when the computer program is executed by a processor, it also performs the following steps:

[0159] Obtain the total number of iterations and the disturbance intensity during the last iteration's disturbance processing; determine the model robustness score based on the preset instability value, the disturbance intensity, the model output, and the total number of iterations; determine the model performance index based on the model output and the disturbance output; generate a visualization result based on the model performance index; and determine the robustness evaluation result of the power system AI model to be evaluated based on the model robustness score and the visualization result.

[0160] In one embodiment, when the computer program is executed by a processor, it also performs the following steps:

[0161] Based on the classification results in the model output and the classification results in the perturbation output, the consistency of classification decisions is determined; based on the confidence results in the model output and the confidence results in the perturbation output, the smoothness of the predicted probability distribution is determined; based on the semantic features of the hidden layer corresponding to the model output and the semantic features of the hidden layer corresponding to the perturbation output, the stability coefficient is determined; based on the accuracy corresponding to the model output and the accuracy corresponding to the perturbation output, the robustness degradation index is determined; based on the accuracy corresponding to the model output, the accuracy corresponding to the perturbation output, the statistical error limit of the data perturbation strategy, and the number of data perturbation strategies, the sensitivity multidimensional comparative analysis is determined; the consistency of classification decisions, the smoothness of the predicted probability distribution, the stability coefficient, the robustness degradation index, and the sensitivity multidimensional comparative analysis constitute the model performance indicators.

[0162] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile memory and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM). The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, artificial intelligence (AI) processors, etc., and are not limited to these.

[0163] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this application.

[0164] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of this patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.

Claims

1. A robustness evaluation method for an AI model of a power system, characterized in that, The method includes: Based on the original power system flow samples, basic evaluation data is constructed; The basic evaluation data is input into the AI ​​model of the power system to be evaluated to obtain the model output; Based on the model output and the AI ​​model of the power system to be evaluated, the basic evaluation data is iteratively perturbed using a data perturbation strategy and an adaptive closed-loop perturbation strategy to obtain the perturbation output. Based on the model output and the disturbance output, robustness evaluation results are generated for the power system AI model to be evaluated.

2. The method according to claim 1, characterized in that, The AI ​​model of the power system to be evaluated, based on the model output, iteratively perturbs the basic evaluation data through a data perturbation strategy and an adaptive closed-loop perturbation strategy to obtain the perturbation output: Obtain the embedding space offset from the previous iteration; where the embedding space offset from the first iteration is a random initial value. Based on the embedded space offset, the perturbation intensity for the current iteration is determined by an adaptive closed-loop perturbation strategy; Based on the perturbation intensity of the current iteration, the perturbation dataset of the previous iteration is perturbed using a data perturbation strategy to obtain the perturbation dataset of the current iteration. Input the disturbance dataset of the current iteration into the power system AI model to be evaluated to obtain the disturbance result of the current iteration; Based on the model output and the perturbation result, determine the embedding space offset for the current iteration; Based on the model output and the perturbation result, it is determined whether the preset stopping condition has been met; if the preset stopping condition has not been met, the next iteration perturbation process begins; if the preset stopping condition has been met, the iteration perturbation process is stopped, and the perturbation result of the current iteration is used as the perturbation output.

3. The method according to claim 2, characterized in that, The process of perturbing the perturbation dataset of the previous iteration based on the perturbation strength of the current iteration using a data perturbation strategy to obtain the perturbation dataset of the current iteration includes: If the data perturbation strategy includes at least one of the semantic constraint perturbation strategy, the temporal perturbation strategy, and the joint behavior perturbation strategy, then based on the perturbation strength of the current iteration and the data perturbation strategy, the perturbation dataset of the previous iteration is perturbed in a preset order to obtain the perturbation dataset of the current iteration. If the data perturbation strategy includes any one of the semantic constraint perturbation strategy, temporal perturbation strategy, and joint behavior perturbation strategy, then based on the perturbation strength of the current iteration and the data perturbation strategy, the perturbation dataset of the previous iteration is perturbed to obtain the perturbation dataset of the current iteration.

4. The method according to claim 2, characterized in that, The step of determining whether a preset stopping condition has been met based on the model output and the perturbation result includes: Based on the model output and the perturbation result, the classification change result for the current iteration is determined; If the classification change result of the current iteration exceeds the preset classification threshold, the embedding space offset of the current iteration exceeds the preset instability value, or the number of disturbances in the current iteration is not less than the preset maximum number of disturbances, then it is determined that the preset stopping condition has been met. If the classification change result of the current iteration does not exceed the preset classification threshold, the embedding space offset of the current iteration does not exceed the preset instability value, and the number of disturbances in the current iteration is less than the preset maximum number of disturbances, then it is determined that the preset stopping condition has not been met.

5. The method according to claim 1, characterized in that, The robustness evaluation results of the power system AI model to be evaluated, generated based on the model output and the disturbance output, include: Obtain the total number of iterations and the perturbation intensity during the last iteration's perturbation processing; The model robustness score is determined based on the preset instability value, the disturbance intensity, the model output, and the total number of iterations. Based on the model output and the perturbation output, the model performance index is determined; Based on the model performance metrics, generate visualization results; Based on the model robustness score and the visualization results, the robustness evaluation result of the power system AI model to be evaluated is determined.

6. The method according to claim 5, characterized in that, The process of determining model performance metrics based on the model output and the perturbation output includes: Based on the classification results in the model output and the classification results in the perturbation output, the consistency of classification decisions is determined; The smoothness of the predicted probability distribution is determined based on the confidence results in the model output and the confidence results in the perturbation output. The stability coefficients are determined based on the hidden layer semantic features corresponding to the model output and the hidden layer semantic features corresponding to the perturbation output. Based on the accuracy corresponding to the model output and the accuracy corresponding to the perturbation output, the robustness degradation index is determined. Based on the accuracy corresponding to the model output, the accuracy corresponding to the perturbation output, the statistical error limit of the data perturbation strategy, and the number of data perturbation strategies, a sensitivity multidimensional comparative analysis is determined; the classification decision consistency, the predictive probability distribution smoothness, the stability coefficient, the robustness degradation index, and the sensitivity multidimensional comparative analysis constitute the model performance indicators.

7. A robustness evaluation device for an AI model of a power system, characterized in that, The device includes: The module is used to build basic evaluation data based on the original power system flow samples; The input module is used to input the basic evaluation data into the power system AI model to be evaluated and obtain the model output; The disturbance module is used to perform iterative disturbance processing on the basic evaluation data based on the model output and the power system AI model to be evaluated, through a data disturbance strategy and an adaptive closed-loop disturbance strategy, to obtain the disturbance output. The evaluation module is used to generate robustness evaluation results for the power system AI model to be evaluated based on the model output and the disturbance output.

8. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 6.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 6.

10. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 6.

Images