An algorithm model evaluation method and system

Abstract

The application relates to the technical field of artificial intelligence, and discloses an algorithm model evaluation method and system. The method comprises the following steps: an evaluation strategy meta-learning framework selects one evaluation action from an activated behavior exploration strategy and an activated cognitive diagnosis strategy according to evaluation history; when the activated behavior exploration strategy is activated, a fragile dimension is identified according to a current state vector to generate a disturbance input; when the activated cognitive diagnosis strategy is activated, a previous error evaluation instance is analyzed to locate a defect and generate a diagnostic probe input; the input is applied to a model under test, a multi-dimensional state vector monitoring module collects data to generate a new state vector; the framework calculates a reward according to the new vector, and the action, the vector and the reward are added to the history to form a decision feedback closed loop. Through the construction of a self-adaptive decision and feedback mechanism, the evaluation process can be highly automated and intelligentized, and the depth, breadth and efficiency of the evaluation of the algorithm model are improved.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology, specifically to an algorithm model evaluation method and system. Background Technology

[0002] With the rapid development of artificial intelligence technology, various algorithm models are widely used in different fields. Ensuring the performance, robustness, and reliability of these models before actual deployment is crucial. Existing evaluation methods for algorithm models generally rely on manually designed test datasets or pre-set test cases. This approach typically requires significant expert knowledge and human resources, and is inefficient, failing to comprehensively cover all potential behavioral boundaries of the model in complex real-world scenarios. The dynamic nature of model behavior and the complexity of its inherent decision-making mechanisms make static testing-based evaluations ineffective in discovering potential vulnerabilities or unexpected behaviors of the model.

[0003] Furthermore, when models malfunction, existing methods often only point out the errors but struggle to pinpoint the underlying causes of the flaws, making the diagnostic process time-consuming and inefficient. Therefore, automating, intelligentizing, and adapting algorithm model evaluation, and improving its depth and efficiency, is a significant technical challenge facing the field of artificial intelligence.

[0004] Therefore, this invention proposes an algorithm model evaluation method and system to address the shortcomings of existing technologies. Summary of the Invention

[0005] To address the shortcomings of existing technologies, this invention provides an algorithm model evaluation method and system, which solves the problems of evaluation methods relying on manual design, having a single evaluation dimension, and lacking dynamic feedback.

[0006] To achieve the above objectives, the present invention provides the following technical solution: an algorithm model evaluation method, comprising the following steps:

[0007] S1. Based on the evaluation history, which includes historical state vectors and historical evaluation actions, the evaluation strategy meta-learning framework selects an evaluation action to generate evaluation input. The evaluation action is one of the activation behavior exploration strategy or the activation cognitive diagnosis strategy.

[0008] S2. When the evaluation action is to activate the behavior exploration strategy, the behavior exploration strategy module identifies the vulnerable dimension based on the current state vector and generates a perturbation input aimed at probing the boundary of the model behavior as the evaluation input.

[0009] S3. When the evaluation action is to activate the cognitive diagnostic strategy, the cognitive diagnostic strategy module analyzes the previous evaluation instances that caused the model output error to locate the defect and generate targeted diagnostic probe inputs as evaluation inputs.

[0010] S4. Apply the evaluation input to the algorithm model under test deployed in a resource-isolated operating environment;

[0011] S5. During the process of the tested algorithm model executing the evaluation input, the multi-dimensional state vector monitoring module collects and processes the original index data of the tested algorithm model to generate a new state vector characterizing the comprehensive operating state of the tested algorithm model.

[0012] S6. The evaluation strategy meta-learning framework calculates an immediate reward based on the new state vector, and adds the evaluation action, the new state vector, and the immediate reward of this evaluation to the evaluation history.

[0013] This invention also provides an algorithm model evaluation system, comprising: an evaluation strategy meta-learning framework for selecting an evaluation action based on evaluation history and calculating an immediate reward based on a new state vector to update the evaluation history; a behavior exploration strategy module for generating a perturbation input based on the current state vector upon receiving an instruction to activate the behavior exploration strategy; a cognitive diagnosis strategy module for parsing previously erroneous evaluation instances to generate diagnostic probe inputs upon receiving an instruction to activate the cognitive diagnosis strategy; a dynamic evaluation execution engine for providing a resource-isolated operating environment for the algorithm model under test and receiving and executing the perturbation inputs or diagnostic probe inputs; and a multi-dimensional state vector monitoring module for collecting data during the execution of inputs by the algorithm model under test and generating new state vectors for use by the evaluation strategy meta-learning framework.

[0014] This invention provides a method and system for evaluating algorithm models. It has the following beneficial effects:

[0015] This invention introduces an evaluation strategy meta-learning framework, which combines evaluation history, state vectors, and immediate rewards to construct a decision feedback closed loop. This framework can autonomously learn and select the optimal evaluation action based on historical evaluation data, replacing the manual process of test case design and process planning that relies on expert experience in traditional evaluation. This allows the entire evaluation process to be dynamically adjusted according to the real-time feedback of the model under test, thereby achieving a higher level of automation and intelligence.

[0016] This invention employs two complementary evaluation strategies—behavioral exploration and cognitive diagnosis—to achieve both a broad exploration of the model's capability boundaries and a deep analysis of known deficiencies. On one hand, the behavioral exploration strategy, by identifying vulnerable dimensions and generating perturbation inputs, proactively detects edge scenarios that are difficult for the model to cover in conventional testing. On the other hand, the cognitive diagnosis strategy, by constructing explanatory causal graphs to locate defects, enables in-depth analysis of the root causes of errors. Combined with multi-dimensional vector monitoring of the model's operational state, this results in more comprehensive and insightful evaluation conclusions.

[0017] The evaluation input generation process of this invention has a clear goal orientation, rather than random or generalized testing. Whether it is a perturbation input generated to detect specific vulnerable dimensions or a diagnostic probe input generated to verify specific defect nodes, each evaluation action focuses on discovering or locating specific problems. This goal-driven mechanism avoids redundant testing on dimensions where the model performs well, and concentrates computational resources on the most valuable evaluation directions, thereby improving the efficiency of discovering and locating model defects. Attached Figure Description

[0018] Figure 1 This is a logical architecture diagram of the algorithm model evaluation system based on cognitive mechanism and meta-learning self-evolution of the present invention;

[0019] Figure 2 This is a flowchart of the algorithm model evaluation method based on cognitive mechanism and meta-learning self-evolution of the present invention.

[0020] Among them, 100 is the dynamic evaluation execution engine; 200 is the multi-dimensional state vector monitoring module; 300 is the evaluation strategy meta-learning framework; 400 is the behavior exploration strategy module; and 500 is the cognitive diagnosis strategy module. Detailed Implementation

[0021] The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0022] See attached document Figure 1 The algorithm model evaluation system provided by this invention is deployed in one or more server computing devices, which include processors, memory, and network interfaces; logically, this system may include:

[0023] The dynamic evaluation execution engine 100 provides a resource-isolated runtime environment to deploy and execute the algorithm model under test; and receives evaluation input instructions from the behavior exploration strategy module 400 or the cognitive diagnosis strategy module 500 and applies them to the algorithm model under test.

[0024] A multi-dimensional state vector monitoring module 200 is connected to the dynamic evaluation execution engine 100. This multi-dimensional state vector monitoring module 200 is used to collect raw indicator data covering multiple dimensions of the model during the execution of evaluation input instructions by the algorithm model under test. The collected dimensions include, but are not limited to, the model's effectiveness, performance, resources, security, and reliability. After acquiring the raw indicator data, the multi-dimensional state vector monitoring module 200 processes it into a state vector, which is used to characterize the comprehensive operating state of the algorithm model under test at that moment.

[0025] The evaluation strategy meta-learning framework 300, which serves as the system's decision-making unit, receives a sequence of historical state vectors. Based on the sequence of historical state vectors and the evaluation strategies executed in the past, the evaluation strategy meta-learning framework 300 uses a built-in meta-learning algorithm to generate a decision instruction to activate the corresponding behavior exploration strategy module 400 or cognitive diagnosis strategy module 500.

[0026] A behavior exploration strategy module 400 is connected to an evaluation strategy meta-learning framework 300. The behavior exploration strategy module 400 is activated after receiving a decision instruction indicating the selection of a behavior exploration strategy. The behavior exploration strategy module 400 identifies behavioral vulnerabilities based on the current state vector provided by the multi-dimensional state vector monitoring module 200 and generates perturbation input accordingly. The perturbation input is then sent to the dynamic evaluation execution engine 100.

[0027] The cognitive diagnostic strategy module 500 is connected to the evaluation strategy meta-learning framework 300. The cognitive diagnostic strategy module 500 is activated after receiving a decision instruction indicating the selection of a cognitive diagnostic strategy. The cognitive diagnostic strategy module 500 acquires the evaluation content that caused the model's previous output error, parses the content to locate the logical defects or factual errors, generates diagnostic probe inputs, and sends them to the dynamic evaluation execution engine 100 for execution.

[0028] In one specific implementation, the implementation method of the multidimensional state vector monitoring module 200 is described in detail. The function of the multidimensional state vector monitoring module 200 is to transform the multidimensional and heterogeneous running performance of the algorithm model under test within an evaluation time step into a standardized and unified mathematical representation, namely, a state vector. This state vector provides structured environmental state information for the decision-making of the evaluation strategy meta-learning framework 300.

[0029] The core task of the multidimensional state vector monitoring module 200 is to define and construct the state vector, as well as the subsequent index acquisition and quantification processing; state vector Defined as a real vector, which at evaluation time step The instance is denoted as The specific composition is as follows: ;in: The effect component is used to measure the performance of the tested algorithm model at time t. The quality of the output content; this component can be calculated from a set of effect-based raw metrics, including but not limited to: semantic similarity score with the reference answer, accuracy score returned by the factual verification interface, or scores of automated evaluation metrics (such as ROUGE, CodeBLEU) for specific tasks (such as text summarization, code generation). The performance component is used to measure the performance of the algorithm model under test at time t. Efficiency in responding to evaluation inputs; this component can be calculated from a set of raw performance metrics, including but not limited to: end-to-end response latency per request, request throughput per unit time (QPS), or first token generation time (TTFT) when generating streaming output. Resource components are used to measure the performance of the tested algorithm model at time 100000. The computing resources consumed during runtime; this component can be calculated from a set of resource-based raw metrics, including but not limited to: peak memory usage of the graphics processing unit (GPU), GPU core utilization, or CPU utilization. The safety component is used to measure the performance of the algorithm model under test at time t. The ability to resist malicious or improper input; this component can be calculated from a set of security-related raw metrics, including but not limited to: the rejection rate against a pre-defined malicious instruction library, or the security risk score given by an independent AI content security classifier on the model output; The reliability component is used to measure the reliability of the algorithm model under test at time t. The stability and consistency of the output results; this component can be calculated from a set of reliability-related primitive indicators, including but not limited to: the semantic consistency of the model output when multiple semantically equivalent but different inputs are received; or the degree of fluctuation in the output results caused by changes in system load under the same input.

[0030] To generate the aforementioned state vector, the multidimensional state vector monitoring module 200 executes a standardized process that includes indicator acquisition and quantification, which may include the following steps:

[0031] S201: Metric Collection; In the dynamic evaluation execution engine 100, a set of software probes are deployed for the runtime environment of the algorithm model under test. These probes act as data collection agents, responsible for capturing raw metrics for each dimension. For example, the performance probe calculates latency by recording the timestamps of request arrival and response return at the network interface level; the resource probe obtains memory usage and utilization by periodically calling underlying hardware monitoring interfaces (such as the NVML library for NVIDIA GPUs); the performance and security probes send the output of the model under test to external, independent evaluation services or content security classification services via API calls to obtain scores. This step is performed for each dimension. (in Generate an original set of indicators. .

[0032] S202: Quantification; This step uses a preset scoring function. The original index set Convert to a range scalar value of the interval This value is a component of the state vector; its general form is: ;

[0033] Scoring function It is not a single form, but rather configured with multiple specific implementation methods based on the characteristics of different indicators;

[0034] One implementation is a linear normalization function, suitable for handling continuous numerical indicators; for indicators where higher values ​​are better (e.g., accuracy). Its scoring function can be specifically implemented as follows: ;in, and These are the upper and lower limits of the preset normal operating range for the indicator; for indicators where lower values ​​are better (e.g., latency), these are the upper and lower limits respectively. Its scoring function can be specifically implemented as follows: ;

[0035] Another implementation approach is a threshold-based multi-stage step function, suitable for handling rating indicators with clearly defined good and bad ranges. For example, for a security risk score... Its scoring function Specifically, it can be implemented as follows: ;in, and These are pre-set safety and warning risk thresholds.

[0036] Another implementation method is aggregate functions, which are suitable for the original indicator set. This includes scenarios involving multiple data points. For example, in a concurrent test, the performance probe collects a set of latency data. At this point, the scoring function Specifically, this can be achieved by first calculating the 95th percentile delay of the set. Then As input, the aforementioned linear normalization function is applied for processing.

[0037] In one specific implementation, the internal implementation of the evaluation strategy meta-learning framework 300 is described in detail. The core function of this framework is to realize the autonomous selection and optimization of the evaluation strategy. To achieve this function, the present invention formally models the evaluation strategy selection problem as a Markov-Decision-Process (MDP). MDP provides a mathematical framework for sequential decision-making in uncertain environments, enabling the system to learn a decision strategy that maximizes long-term evaluation benefits based on historical evaluation data.

[0038] A Markov decision process consists of a quintuple. The definitions, in which the specific technical connotations of each element are as follows:

[0039] state space A state in the state space This represents the time step in the evaluation. At that time, all environmental information observable by the system. In this invention, a state... It's not just the state vector at the current moment. Instead, it includes complete historical information from the start of the evaluation to the current moment; specifically, the state. It is defined as a sequence consisting of a historical state vector and historical actions: ;in, In time step The state vector generated by the multi-dimensional state vector monitoring module 200 In time step The evaluation action selected by this framework uses historical sequences as states, enabling the decision-making process to utilize information about the model's behavioral evolution trajectory, rather than relying solely on instantaneous snapshots, thus making more forward-looking judgments. In a specific implementation, to process this serialized state input, the evaluation policy meta-learning framework 300 can internally employ a recurrent neural network (RNN) or a Transformer encoder structure to process the variable-length historical sequence. It is encoded as a fixed-dimensional state representation vector.

[0040] Action space The action space is the set of evaluation strategies that the evaluation strategy meta-learning framework 300 can choose at each decision time; in this invention, this set is finite, specifically including: ;in, The representative selects and activates the behavior exploration strategy module 400 to perform an evaluation action aimed at exploring the boundaries of the model's behavior; The representative selects and activates the cognitive diagnostic strategy module 500 to perform an assessment action aimed at dissecting cognitive deficiencies within the model.

[0041] State transition probability State transition probability function Defined in a given history In this case, perform the evaluation action. Afterwards, the system transitioned to a new historical state. The probability of transition; in the complex scenario of algorithm model evaluation, this transition probability is implicit and unknown; it is determined by the complex internal state of the algorithm model under test and the evaluation actions. The specific input generated, as well as random factors in the evaluation system environment, jointly determine the state transition probability. Since the state transition probability is unknown, this invention uses a model-free reinforcement learning algorithm to solve the MDP.

[0042] reward function The reward function provides a scalar feedback signal to the evaluation policy meta-learning framework 300 to quantify the immediate value of an evaluation action; when performing an action... and make the system state from Transferred to Then, the system will calculate a reward value. The reward value is designed to guide the system to learn a strategy that can efficiently discover unknown defects and behavioral boundaries in the model.

[0043] Discount factor The discount factor is a hyperparameter used to adjust the relative importance of short-term and long-term rewards; the total reward of a test series is defined as the sum of the discounts on all future rewards; when When the value is close to 0, the system tends to choose actions that immediately generate high rewards; when When the value is close to 1, the system will pay more attention to long-term cumulative rewards and exhibit more forward-looking exploratory behavior.

[0044] In one specific implementation, the reward function in the evaluation strategy meta-learning framework 300 is described in detail. The reward function provides key feedback signals for the evaluation strategy meta-learning framework 300, and its value quantifies the effectiveness of each evaluation action to guide the meta-learning algorithm to optimize the evaluation strategy.

[0045] During the evaluation time step Perform an evaluation action Then, the system will calculate an instant reward value. The reward value is a weighted sum of an exploration reward and an influence reward, calculated using the following formula: ;in: It is a time step Total instant rewards; In time step Calculated exploration rewards; In time step Calculated impact reward; and These are preset weighting coefficients, both of which are non-negative real numbers, used to adjust the relative importance of exploration rewards and influence rewards in the total reward.

[0046] Explore Rewards Used to quantize the newly generated state vector The novelty level in the historical state space guides the system to explore unvisited state regions; its value is equal to the minimum distance between the current state vector and all previous state vectors in the evaluation history; a higher exploration reward indicates that the current evaluation action has discovered a completely new model behavior pattern; its calculation formula is: ;in: The multidimensional state vector monitoring module 200 monitors the current time step. The generated state vector; Stored in evaluation history In the previous time step The generated state vector; Represents the Euclidean distance (L2 norm); if the state vector for A dimensional vector with components of . Then the two vectors and The Euclidean distance between them is calculated as follows: ;in, For vectors The One portion, For vectors The Each component.

[0047] Impact on rewards This reward is used to quantify the magnitude of the perturbation caused by the latest evaluation action to the state of the algorithm under test. This reward value is directly related to significant changes in the model state; a higher impact reward usually means that the evaluation action touched a vulnerability in the model, causing a drastic change in its performance, effectiveness, or other dimensions. It is calculated as the Euclidean distance between the current state vector and the previous state vector. ;in, It was at the previous time step The generated state vector.

[0048] In a specific implementation, the total reward The calculation process may include the following sub-steps:

[0049] S301: Computational Exploration Rewards Evaluation strategy meta-learning framework 300 obtains the current state vector and from its internally stored evaluation history Retrieve all historical state vectors System iterative calculation With each historical state vector The Euclidean distance is calculated, and the minimum of all calculated distances is used as the exploration reward. The value;

[0050] S302: Calculate the impact on rewards Evaluation strategy meta-learning framework 300 obtains the current state vector The state vector from the previous time step The Euclidean distance between the two is directly calculated, and the result is used as the factor affecting the reward. The value;

[0051] S303: Calculate the weighted total reward The system uses preset weighting coefficients. and The exploration reward and influence reward calculated in the preceding steps are weighted and summed to obtain the final total immediate reward. To achieve adaptive emphasis of the evaluation strategy at different stages, there are various specific implementations for determining the weight coefficients.

[0052] In one implementation, the weighting coefficient can be set to a fixed value throughout the entire evaluation process; for example, setting... and To explore and influence in a balanced way.

[0053] In a more preferred embodiment, the weighting coefficients are dynamically adjusted to guide the evaluation process smoothly from the initial broad exploration to the later in-depth mining; this dynamic adjustment mechanism can be specifically implemented in one or a combination of the following executable methods: Implementation Method 1: Dynamic adjustment based on a preset scheduling function. This method uses a scheduling function that adjusts the weighting coefficients based on the evaluation time step. The weights are calculated as deterministic functions of the independent variables; a maximum number of evaluation steps is preset before the evaluation begins. ;; at each evaluation time step (in ), weight and The following linear scheduling function can be used for calculation: ;

[0054] ;in: and These are the initial values ​​for exploring the weights and influencing the weights (for example, they can be set to 1.0 and 0.1 respectively). and These are the final values ​​for exploring the weights and influencing the weights (for example, they can be set to 0.1 and 1.0 respectively).

[0055] Implementation Method Two: Adaptive Adjustment Based on Evaluation Status Feedback. This method adjusts the weights based on real-time feedback during the evaluation process, allowing them to adapt to changes in the evaluation progress. It may include the following sub-steps:

[0056] The system maintains a size of A sliding window is used to store and calculate the most recent The average exploration reward at each time step, denoted as . The system presets an exploration saturation threshold. .

[0057] At each time step After completion, the system compares and :like Below the threshold This indicates that recent evaluations have failed to effectively identify new states, and the exploration is approaching saturation; the system's performance has also decreased. And improve The operation aims to encourage deeper impact attacks on existing areas;

[0058] like Above the threshold This indicates that recent explorations have been very successful; the system has maintained or improved. The weights are adjusted to continue encouraging exploration of the unknown state space; a specific adjustment rule can be implemented through a proportional controller: ;in, It is the learning rate that controls the adjustment rate. The function is used to limit the weight values ​​to a preset minimum. and maximum Within the range; It can be set accordingly To keep the total weight sum to 1.

[0059] In one specific implementation, the evaluation strategy meta-learning framework 300 is used to implement the meta-policy. This paper provides a detailed explanation of the neural network (hereinafter referred to as the meta-policy network) and its update mechanism; the function of the meta-policy network is to process the sequential evaluation history as input. Mapped to an action space The probability distribution on the output is used to determine the decision to select a specific evaluation strategy.

[0060] The input to the meta-policy network is the evaluation history. Since the input is a sequence of variable length, the meta-policy network uses a neural network structure that can handle serialized data.

[0061] One specific implementation is that the meta-policy network adopts a recurrent neural network (RNN) structure, such as a long short-term memory (LSTM) network or a gated recurrent unit (GRU) network; in this implementation, at each historical time step , the state vector With action A vectorized representation (e.g., one-hot encoding) is concatenated and used as input to the LSTM network; the LSTM network updates its hidden state through its internal gating mechanism. After processing the entire history sequence, the hidden state of the last time step. Used throughout the entire evaluation history A fixed-dimensional representation vector.

[0062] Another specific implementation is that the meta-policy network adopts a Transformer encoder structure; in this implementation, each state-action pair in the historical sequence is encoded. The sequence is transformed into an input embedding vector, and a positional encoding is appended to it. The entire embedding sequence is then fed into the Transformer encoder. Its internal self-attention mechanism can compute the dependency between any two positions in the sequence, thus more effectively capturing long-distance dependencies between historical events. Finally, an aggregated vector output by the Transformer encoder (e.g., the output vector corresponding to a specific classification label [CLS]) is taken as the evaluation history. The representation vector;

[0063] After obtaining the historical representation vector, this vector is fed into one or more fully connected layers and finally through a Softmax activation function; the output of the Softmax function is a probability distribution whose dimension is the same as the action space. The values ​​are of equal magnitude, and each value in the distribution corresponds to the probability of selecting the corresponding evaluation action, i.e. The complete set of trainable parameters of the meta-policy network is denoted as . ;

[0064] Meta-policy network parameters The update employs a policy-gradient reinforcement learning algorithm; this algorithm adjusts parameters... This increases the probability of selecting actions that can bring higher cumulative returns; its parameter update formula is: ;in: These are the trainable parameters of the meta-policy network; It is the learning rate, a positive scalar hyperparameter used to control the step size of parameter updates; In time step The actual sampling and execution of evaluation actions; In time step Evaluation history; It is based on the current meta-policy network, given a history In this case, choose an action The probability of; It is the logarithm of the action selection probability with respect to the network parameters. The gradient indicates the direction of parameter adjustment to increase future selection actions. The possibility; From time step The initial cumulative discount reward is used to scale the gradient direction.

[0065] parameter The update process can be specifically implemented as an iterative training process including the following sub-steps:

[0066] S311: Collect evaluation trajectories; the system executes a complete evaluation sequence (also known as an episode), which can have a preset maximum step size. Or it may be determined by other termination conditions; during this process, the system collects and stores a complete trajectory containing state, action, and reward: ;

[0067] S312: Calculate the cumulative discount reward; after a test sequence ends, the system calculates the cumulative discount reward for each time step in the trajectory. Calculate the corresponding cumulative discount reward from back to front. The calculation formula is as follows: ;in, It is a discount factor; the calculation is from Start by iterating backwards to... .

[0068] S313: Calculate the gradient and update the parameters; for each time step in the trajectory. The system calculates the objective function term. Subsequently, the objective function with respect to the network parameters is calculated using automatic differentiation (backpropagation). gradient The gradients of all time steps in a complete sequence or a batch of sequences can be summed or averaged to obtain the final update gradient. Finally, according to the aforementioned parameter update formula, an optimizer (such as the Adam optimizer) is used to update the parameters. Update.

[0069] In one specific implementation, the implementation method of the behavior exploration strategy module 400 is described in detail. After receiving the activation instruction issued by the evaluation strategy meta-learning framework 300, the core function of the behavior exploration strategy module 400 is to generate a perturbation input that can effectively detect the behavioral boundary or unknown vulnerability of the algorithm model under test. The whole process can be decomposed into two main stages: vulnerability dimension identification and guided perturbation generation.

[0070] The behavior exploration strategy module 400 first performs vulnerability dimension identification, the purpose of which is to identify the current state vector provided by the multi-dimensional state vector monitoring module 200. In the process, it automatically identifies one or more of the most vulnerable components that are worth attacking;

[0071] One specific identification method is to use minimum component value localization, which aims to locate the dimension where the model is currently performing the worst; it finds the state vector by calculation. The index corresponding to the smallest component in the median : ;in, It is a state vector The Each component; the choice of this dimension for attack is intended to exacerbate the weaknesses already exhibited by the model.

[0072] Another specific recognition method is to use maximum state change localization; this method aims to locate the dimension that responds most drastically to the previous evaluation action; it calculates the current state vector. Compared with the state vector of the previous time step Find the absolute value of the change in each component and locate the index corresponding to the component with the largest change. : ;

[0073] Choosing this dimension for attack aims to deeply explore newly discovered sensitive points; the behavior exploration strategy module 400 can be configured to use any of the above methods, or a combination of both, to determine the final attack target dimension.

[0074] After identifying the target's vulnerable components Subsequently, the behavior exploration strategy module 400 enters the guided perturbation generation stage; the goal of this stage is to generate a perturbation input. This ensures that the input, when applied to the algorithm model under test, can minimize the target component. The future observations; this process is achieved through a controlled generative model.

[0075] The generative model can be a pre-trained language model that has been fine-tuned specifically for this task, denoted as... To achieve guidance, relying solely on the generative model's own free generation is insufficient; this invention introduces a proxy model. The surrogate model is a lightweight, differentiable neural network trained to simulate the joint behavior of the algorithm model under test and the multidimensional state vector monitoring module 200; that is, the surrogate model. The input is a candidate perturbation input. Its output is the state vector Predicted value ;

[0076] The specific implementation process of guided perturbation generation may include the following sub-steps:

[0077] S401: Target Setting. Based on the results of the aforementioned vulnerability dimension identification phase, determine the index of the target component to be minimized. ;

[0078] S402: Iterative Generation and Guidance. Generative Model Perturbation input is generated iteratively (token-by-token). When generating each token, in addition to its own language model probability, a guiding term is introduced; this guiding term is implemented through a proxy model. The calculation yields a specific guiding loss for a partially generated sequence. The loss is equal to the target component value predicted by the surrogate model: ;in, It is a proxy model Output predicted state vector In the model, the component values ​​correspond to the vulnerability dimension; subsequently, the guiding loss is calculated with respect to the generative model via automatic differentiation. The gradient of intrinsic parameters or activation values ​​(e.g., the gradient of the logits layer before generating the next token); this gradient indicates how to adjust the generation process to make the predictions... The value decreases; this gradient is used to adjust the output probability distribution of the generative model at the current step, thereby guiding it to generate a token that is more likely to result in the vulnerability of the target.

[0079] S403: Perturbation Output; When the generation process meets the termination condition (e.g., reaching a preset length or generating an end marker), the complete text sequence is used as the final perturbation input. Output and transmit to the dynamic evaluation execution engine 100;

[0080] In another alternative implementation, a non-gradient-based guided approach can be used; for example, Monte Carlo search can be employed to generate the model. First, multiple candidate perturbation inputs are generated independently; then, each candidate input is fed into the surrogate model. The evaluation process yields the predicted state vectors for each component; finally, the component that best predicts the target component is selected. The candidate input with the smallest value is used as the final perturbation input. .

[0081] In one specific implementation, the implementation method of the cognitive diagnostic strategy module 500 is described in detail. When the cognitive diagnostic strategy module 500 is activated, it receives the evaluation input that previously caused the model output error and the corresponding error text output. The primary task of the cognitive diagnostic strategy module 500 is to perform deep analysis on the error text output, transforming its inherent unstructured reasoning process into a structured, machine-readable, explanatory causal graph. ;in, It is a collection of nodes in the graph, where each node represents an independent semantic unit, such as a factual proposition, an entity concept, or an intermediate conclusion. It is the set of edges in the graph. Each edge is a directed edge that represents the reasoning relationship between nodes, such as causal relationship, support relationship or dependency relationship.

[0082] To achieve the conversion from text to graph, the cognitive diagnostic strategy module 500 executes a pipeline process containing multiple natural language processing steps; this process may specifically include the following sub-steps:

[0083] S501: Text preprocessing; The cognitive diagnosis strategy module 500 first preprocesses the received raw text output, including sentence segmentation, that is, dividing the complete text paragraph into independent sentence units as the basic unit for subsequent processing.

[0084] S502: Semantic Unit Nodeization; The cognitive diagnosis strategy module 500 uses the natural language processing technology stack to parse each sentence, extract the semantic units that form the basis of reasoning, and construct them as nodes;

[0085] One specific implementation involves this step further comprising two parallel processing steps:

[0086] First, Named Entity Recognition (NER) technology is executed to identify and extract proper nouns from the text, such as names of people, places, organizations, dates, and proper terms. Each identified named entity with independent meaning is constructed as an entity node and stored in a node set. ;

[0087] Secondly, the Open-Information-Extraction (OpenIE) technique is used to extract factual triples (subject, predicate, object) from sentences; each triple represents an independent proposition or assertion; for example, the triple (A, cause, B) is extracted from the sentence A leads to B; each extracted triple is constructed as a proposition node and stored in a node set. .

[0088] S503: Alignment of reference relationships between nodes; After independent semantic units are extracted as nodes, the cognitive diagnostic strategy module 500 performs coreference-resolution processing; This step aims to identify and associate different expressions in the text that point to the same entity or concept, such as pronouns (e.g., it, he), attributive descriptions (e.g., this company, this technology), etc.; Through coreference-resolution, multiple nodes that refer to the same object are merged or equivalent relationships are established, ensuring the connectivity and accuracy of the causal graph subsequently constructed.

[0089] S504: Construction of reasoning relationship edges; After the nodes are accurately defined and aligned, the module establishes directed edges between the nodes to represent their reasoning relationships;

[0090] One specific implementation is to use a rule-based matching method based on explicit connectors. The module has a built-in dictionary of connectors containing logical relationships such as cause and effect, condition, and example (e.g., words such as because, therefore, thus, if...then, for example, etc.). By scanning the text, these connectors are matched, and a directed edge of the corresponding type is established between the nodes corresponding to the semantic units connected by the connectors. For example, for the text "Because proposition P, therefore conclusion Q", a causal relationship edge from P to Q is established between the node representing P and the node representing Q.

[0091] Another specific implementation method is to use implicit relation inference based on syntactic structure; the module can use dependency parsing technology to parse the grammatical structure within and between sentences; by analyzing the subordinate, modifying or parallel relations in the syntax tree, it can infer the support, explanation or sequence relations between nodes, and can establish inference edges even in the absence of explicit connectors.

[0092] S505: Cause-effect graph assembly; After completing the above steps, the cognitive diagnostic strategy module 500 will collect all identified and processed nodes. Set of edges related to reasoning By integrating these elements, a complete, directed, and structured explanatory causal graph is ultimately constructed. This diagram visually illustrates the internal logical path followed by the tested algorithm model when generating its output.

[0093] In one specific implementation, after the cognitive diagnostic strategy module 500 has constructed the explanatory causal graph... Next, we will explain in detail how to locate defective nodes in the diagram. The purpose of this step is to automatically identify the root causes of errors in the overall output from the structured reasoning path, namely one or more erroneous factual propositions or illogical reasoning steps. These identified nodes are collectively referred to as defective nodes. ;

[0094] The process of locating defective nodes can include two parallel technical approaches: external factual verification and internal logical consistency checks.

[0095] When performing external factual verification, the system cross-references factual proposition nodes in the causal graph with one or more external trusted knowledge sources; this process may include the following sub-steps:

[0096] S511: Node content querying; Cognitive diagnostic strategy module 500 traversals of cause-effect graph The system generates propositional nodes; for each propositional node (e.g., a node represented as a triple (subject, predicate, object)), the system converts it into one or more questions that conform to natural language usage. For example, for the node (Albert Einstein, born in, Ulm, Germany), the query "Where was Albert Einstein born?" can be generated.

[0097] S512: Knowledge Source Inquiry; The system sends the generated query to one or more pre-configured external knowledge source interfaces; External knowledge sources may include: structured knowledge graphs, such as Wikidata; large-scale text knowledge bases accessible via API, such as Wikipedia; domain-specific professional databases, such as financial information databases used in financial evaluations; or independent third-party large language models configured for high-precision fact-checking.

[0098] S513: Comparison and Marking; The system receives the return results from the knowledge source and compares them with the content of the original proposition node; if the return results are consistent with the node content or can corroborate each other, the node is marked as "verified"; if the return results clearly contradict the node content, the node is marked as a "defective node" and added to the defective node set; if the knowledge source cannot provide relevant information, the node is marked as "unverifiable";

[0099] When performing internal logical consistency checks, the system does not rely on external knowledge, but instead analyzes the cause-effect graph. Logical fallacies are detected by examining the node's own topological structure and the semantic relationships between nodes. This process may include the following checking methods:

[0100] One method of checking is cyclical reasoning, where the system is in a directed graph. The loop detection algorithm is executed. If there exists a path in the graph that starts from node A, goes through one or more directed edges to reach node B, and at the same time there exists another path that can return from node B to node A, then a loop is considered to exist between node A and node B. In this case, all nodes that constitute the loop are marked as "defective nodes".

[0101] Another method is contradictory premise detection. The system checks whether there is a conclusion node C in the graph that is supported by two or more premise nodes (e.g., A and B) (i.e., there are edges A to C and B to C). The system extracts the proposition content represented by premise nodes A and B and inputs it into a pre-trained Natural-Language-Inference (NLI) model. The NLI model outputs the relationship between the two propositions, which is implied, neutral, or contradictory. If the relationship output by the NLI model is contradictory, it proves that the reasoning process is based on contradictory premises, and the system then marks both premise nodes A and B as defective nodes.

[0102] Another method of checking is isolated argument detection; the system traverses all non-root nodes in the graph (i.e., nodes with a length that is not zero); for any non-root node, the system checks all its parent nodes (i.e., the premise nodes that directly point to the node); if all the parent nodes of the node have been marked as "defective node" or "unverifiable", it means that the conclusion of the node lacks valid support, and the node itself is also marked as "defective node".

[0103] In one specific implementation, the cognitive diagnostic strategy module 500, upon successfully locating one or more defective nodes... Then, how to generate diagnostic probes? This step is designed to create a highly targeted new assessment input, not for broad exploration, but to validate, probe, or stress-test specific cognitive deficits that have been identified.

[0104] The probe is generated by a diagnostic probe generation function. Implementation: This function integrates the original context information that caused the error with the located defect information to generate the final probe text. It is represented as follows: ;in: It is the generated diagnostic probe, a text string; It is the original evaluation input that caused the model output error at the previous time step; It is a set of one or more defective nodes located in the previous analysis step; It is a diagnostic type that is automatically determined based on the type of defective node.

[0105] The generation of diagnostic probes is mainly achieved through a template-filling method; the cognitive diagnostic strategy module 500 has a pre-set diagnostic template library, which contains a series of text templates with placeholders; each template corresponds to a specific diagnostic intent;

[0106] The specific implementation of this generation method may include the following sub-steps:

[0107] S521: Determine the diagnosis type; the system first analyzes the defective node. The reason for the labeling is used to determine the diagnosis type. In one scenario, if a defective node is flagged due to a conflict with external knowledge sources (factual error), then the diagnosis type... It is identified as a fact-finding type. Another scenario is that if the defective node is marked for participating in circular reasoning or reasoning based on contradictory premises (logical error), then the diagnostic type is... It was identified as a logical scrutinizing type.

[0108] S522: Select a diagnostic template; the system will then select the diagnostic template based on the diagnostic type determined in the previous step. Select one or more suitable templates from the diagnostic template library; a template is a text string containing one or more placeholders; examples of placeholders include: {N_content}: used to fill in the specific proposition content represented by the defect node; {O_context}: used to fill in the context fragment of the original output text containing the defect node; {P_context}: used to fill in the original evaluation input that caused the error. The core content; {N_conflict_info}: used to populate information obtained from external knowledge sources that conflicts with the content of defective nodes.

[0109] S523: Fill the template and generate probes; the system extracts relevant information from the evaluation history and defect analysis results, and uses this information to replace the placeholders in the template to generate the final probe text. ;

[0110] Below are specific implementation examples for two diagnostic types:

[0111] For fact-finding diagnostics, when a defect node... When a statement is marked as factually incorrect, the system can choose one of the following templates: Template 1: "In your previous answer regarding {P_context}, you mentioned {N_content}; what is the source of this information?" Template 2: "According to my materials, the description of {N_content} seems to contradict {N_conflict_info}; could you reassess your conclusion based on this information?" For logically critical diagnostics, when two nodes are detected... and When constructing a circular argument, the system can select the following template:

[0112] Template: "In your argument, you use {N_content1} as the basis for {N_content2}, but at the same time it seems that you use {N_content2} as a premise for {N_content1}; this constitutes a circular argument. Please clarify the logical relationship between the two."

[0113] In one specific implementation, the implementation method of the dynamic evaluation execution engine 100 is described in detail. This engine performs the function of the execution layer in the entire evaluation system, providing a stable, isolated, and observable operating environment for the evaluation of the algorithm model under test;

[0114] The dynamic evaluation execution engine 100 provides a resource-isolated sandbox environment for the algorithm model under test. One specific implementation method is to use containerization technology to build the sandbox environment. In this implementation, the algorithm model under test and all its software dependencies, runtime environment configurations, etc., are pre-packaged into a standardized container image. When an evaluation task is started, the dynamic evaluation execution engine 100 instantiates an independent container based on this image. This container provides process, file system, and network isolation at the operating system level, and its available computing resources (such as the number of CPU cores, memory limit, and GPU memory allocation) can be precisely limited. This method ensures that each evaluation is carried out in a clean and consistent environment, eliminating interference from external applications or residual states from previous evaluations, thereby ensuring the reproducibility of the evaluation results.

[0115] The dynamic evaluation execution engine 100 is configured with a standardized input interface for receiving evaluation instructions generated by the behavior exploration strategy module 400 or the cognitive diagnosis strategy module 500, i.e., evaluation input. The input interface can be implemented as a RESTful API endpoint or a message queue consumer; the precise execution process of the evaluation command may include the following sub-steps:

[0116] S601: Receive evaluation instructions; The dynamic evaluation execution engine 100 listens for incoming evaluation instructions through its input interface, which contain the evaluation inputs to be executed. ;

[0117] S602: Command forwarding and execution; the engine will evaluate the input. Forward to the tested algorithm model service running in the sandbox container;

[0118] S603: Response capture; the engine processes the evaluation input in the algorithm model under test. During this period, all of its generated output content is captured completely. This includes the final result as well as any intermediate streaming outputs;

[0119] S604: Result returned; the engine will capture the output. The output can be returned to the caller (e.g., to the cognitive diagnostic strategy module 500 for subsequent analysis) or broadcast to the system's internal message bus for use by other modules that require the output.

[0120] The dynamic evaluation execution engine 100 and the multi-dimensional state vector monitoring module 200 work together to achieve non-intrusive data acquisition of the algorithm model under test. This collaborative working mechanism ensures that the monitoring behavior itself does not significantly affect the running performance of the algorithm model under test. In a specific implementation, when the dynamic evaluation execution engine 100 creates a sandbox container, it deploys various monitoring probes of the multi-dimensional state vector monitoring module 200 into the environment.

[0121] For the collection of resource metrics, the dynamic evaluation execution engine 100 is configured with resource probes to directly call the monitoring APIs provided by the container runtime (such as the Docker engine). Through this API, the probes can obtain data such as CPU usage and memory consumption of the internal processes from outside the container without inserting any monitoring logic into the code of the algorithm model under test.

[0122] For the collection of performance metrics, the dynamic evaluation execution engine 100 serves as the traffic entry point for all evaluation inputs. Upon receiving evaluation instructions, it... Time stamp of the moment record After capturing the complete response The time record timestamp; by calculating This allows for accurate determination of end-to-end latency; this process is completely transparent to the algorithm model under test.

[0123] See attached document Figure 2 The evaluation method provided by this invention can be executed in the aforementioned evaluation system. Specifically, the method includes the following steps: In the initialization phase, the method performs the following operations:

[0124] S701: Load the algorithm model under test; the dynamic evaluation execution engine 100 obtains the container image of the algorithm model under test, and instantiates a resource-isolated sandbox runtime environment accordingly, deploys and starts the algorithm model service under test in the environment;

[0125] S702: Initialize the meta-policy network; evaluate the policy meta-learning framework 300 for its internal representation of the meta-policy. Initialize the neural network by setting all its trainable parameters. Set to a set of random values ​​or pre-trained values;

[0126] S703: Perform baseline testing; the system inputs a set of pre-set standardized evaluation datasets covering various typical scenarios into the algorithm model under test; during this baseline testing process, the multi-dimensional state vector monitoring module 200 continuously collects various raw indicators, and calculates and generates an initial state vector after the test. This vector represents the initial synthesis state of the model under normal tasks;

[0127] S704: Initialize evaluation history; the system will initialize the initial state vector. As the starting point of the evaluation, it is stored in the evaluation history sequence;

[0128] After the initialization phase is complete, the method enters an iterative evaluation loop; at each time step of the loop... In the process, the sub-steps of decision-making, execution, perception and learning are executed sequentially until the termination condition is met;

[0129] S705: Decision Sub-step; Evaluation Strategy Meta-Learning Framework 300 obtains the current time step. Complete review history to date This historical sequence is input into the meta-policy network. The meta-policy network encodes the historical sequence and outputs a sequence in the action space. The system samples based on this probability distribution and selects an evaluation action. This action To activate either the Behavioral Exploration Strategy Module 400 or the Cognitive Diagnosis Strategy Module 500;

[0130] S706: Execute sub-step; the system executes the action selected in S705. Perform scheduling; if If the action is to activate the behavior exploration strategy, then the behavior exploration strategy module 400 is activated; the behavior exploration strategy module 400 identifies the vulnerable dimension based on the current state vector and generates a perturbation input aimed at attacking that dimension. ;like If the action is to activate the cognitive diagnostic strategy, then the cognitive diagnostic strategy module 500 is activated; this module 500 acquires the evaluation instance that previously caused the error, and through causal chain tracing and defect localization, generates a probe input with a clear diagnostic intent. .

[0131] S707: Generated Evaluation Input The input is sent to the dynamic evaluation execution engine 100; the engine precisely applies this input to the tested algorithm model running in the sandbox and fully captures the model's output. .

[0132] S708: Perception and Learning Sub-step; During and after the execution of S707, the multi-dimensional state vector monitoring module 200 collects the model's raw indicators in dimensions such as effectiveness, performance, resources, security, and reliability in real time, quantifies them, and generates new state vectors. ;

[0133] S709: Evaluation policy meta-learning framework 300 receives the newly generated state vector. And based on the previously calculated reward function formula, combined with Based on historical status information, the immediate reward for the current evaluation action is calculated. ;

[0134] S710: The system will generate a triplet (action) from this evaluation. New state vector ,award The collected trajectory data is appended to the evaluation history sequence for subsequent decision-making and learning. After one or more complete evaluation cycles (rounds), the system uses the collected trajectory data to refine the meta-policy network according to the previous policy gradient algorithm. parameters Update;

[0135] The iterative evaluation loop terminates when preset termination conditions are met. These termination conditions include, but are not limited to: the total number of evaluation time steps reaching a preset upper limit; or the total runtime and total computational resource consumption of the evaluation system reaching a preset budget; or the total reward value calculated by the system within multiple consecutive time steps. All data points were below a preset convergence threshold, indicating that the system was unlikely to discover any new defects that could cause significant changes in the model state. After the evaluation was terminated, the system automatically integrated all the data collected throughout the evaluation process and generated a comprehensive evaluation report.

[0136] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.

Claims

1. A method for evaluating algorithm models, characterized in that, Includes the following steps: S1. Based on the evaluation history, which includes historical state vectors and historical evaluation actions, the evaluation strategy meta-learning framework selects an evaluation action to generate evaluation input. The evaluation action is one of the activation behavior exploration strategy or the activation cognitive diagnosis strategy. S2. When the evaluation action is to activate the behavior exploration strategy, the behavior exploration strategy module identifies the vulnerable dimension based on the current state vector and generates a perturbation input aimed at probing the boundary of the model behavior as the evaluation input. S3. When the evaluation action is to activate the cognitive diagnostic strategy, the cognitive diagnostic strategy module analyzes the previous evaluation instances that caused the model output error to locate the defect and generate targeted diagnostic probe inputs as evaluation inputs. in; Step S3 specifically includes: The model output errors are parsed to construct an explanatory causal graph containing semantic unit nodes and reasoning relation edges; In the explanatory cause-effect graph, one or more defective nodes are located by performing external factual verification and internal logical consistency checks; Determine the diagnosis type based on the type of defective node; Select a text template with placeholders from the preset diagnostic template library according to the diagnostic type; Using the original evaluation input that caused the error, the content of the defect node, and related contextual information, the placeholders in the text template are filled to generate diagnostic probe input; S4. Apply the evaluation input to the algorithm model under test deployed in a resource-isolated operating environment; S5. During the process of the tested algorithm model executing the evaluation input, the multi-dimensional state vector monitoring module collects and processes the original index data of the tested algorithm model to generate a new state vector characterizing the comprehensive operating state of the tested algorithm model. S6. The evaluation strategy meta-learning framework calculates an immediate reward based on the new state vector, and adds the evaluation action, the new state vector, and the immediate reward of this evaluation to the evaluation history.

2. The algorithm model evaluation method according to claim 1, characterized in that, The step of selecting an evaluation action by the evaluation strategy meta-learning framework to generate evaluation input includes: The evaluation strategy meta-learning framework has a built-in meta-policy network, which receives the evaluation history as input and outputs the probability distribution in the action space. The evaluation strategy meta-learning framework samples based on probability distributions to select evaluation actions.

3. The algorithm model evaluation method according to claim 1, characterized in that, The instant reward consists of an exploration reward and an influence reward, which are weighted and summed using a weighting coefficient.

4. The algorithm model evaluation method according to claim 3, characterized in that, The weighting coefficient is dynamically adjusted based on a preset scheduling function with the evaluation time step as the independent variable.

5. The algorithm model evaluation method according to claim 3, characterized in that, The steps for adaptively adjusting the weighting coefficients based on evaluation status feedback include: Calculate the average value of exploration rewards over the most recent time steps, compare the average value with a preset exploration saturation threshold, and adjust the weighting coefficients based on the comparison results.

6. The algorithm model evaluation method according to claim 1, characterized in that, The state vector includes at least two components selected from the following: The effect component is used to measure the quality of the output content of the algorithm model under test. The performance component is used to measure the efficiency of the algorithm model under test in responding to the evaluation input. Resource component, used to measure the computing resources consumed by the tested algorithm model during runtime; The security component measures the ability of the algorithm model under test to resist malicious or improper input. The reliability component is used to measure the stability and consistency of the output results of the algorithm model under test.

7. The algorithm model evaluation method according to claim 1, characterized in that, The step of identifying the vulnerable dimension by the behavior exploration strategy module based on the current state vector includes: The vulnerability dimension can be determined by using minimum component value positioning to find the dimension corresponding to the smallest component in the current state vector; or by using maximum state change positioning to calculate the absolute value of the change of each component between the current state vector and the previous state vector, and find the dimension corresponding to the component with the largest change.

8. An algorithm model evaluation system, applied to the method described in any one of claims 1-7, characterized in that, The system includes: The evaluation strategy meta-learning framework is used to select an evaluation action based on the evaluation history and calculate an immediate reward based on the new state vector to update the evaluation history. The behavior exploration strategy module is used to generate perturbation input based on the current state vector when it receives an instruction to activate the behavior exploration strategy. The cognitive diagnostic strategy module is used to parse previously erroneous evaluation instances to generate diagnostic probe inputs when an instruction to activate the cognitive diagnostic strategy is received. The dynamic evaluation execution engine provides a resource-isolated runtime environment for the algorithm model under test and receives and executes the perturbation input or diagnostic probe input. The multidimensional state vector monitoring module is used to collect data during the execution of input by the algorithm model under test and generate new state vectors for use by the evaluation strategy meta-learning framework.

Images