Method, system, device, product and storage medium for assessing violations of artificial intelligence models

Abstract

The application provides a method, system, device, product and storage medium for evaluating violation of an artificial intelligence model, which comprises: querying event information related to a violation event of the artificial intelligence model from a monitoring system of the artificial intelligence model based on identification of the violation event; and obtaining monitoring data corresponding to a plurality of evaluation dimensions from the monitoring system based on the event information, and evaluating the violation cause of the artificial intelligence model in the corresponding evaluation dimension by using the obtained monitoring data. The plurality of evaluation dimensions comprise a training sample dimension, an inference step dimension, a neuron dimension, a safety constraint mechanism dimension, a violation risk evolution dimension in different training stages, and a plurality of violation risk dimensions of neurons in different training stages. The above method can deeply and comprehensively analyze the violation cause of the artificial intelligence model from multiple dimensions, thereby providing effective help for model governance and improving the model governance effect.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and in particular to a method, system, device, product, and storage medium for evaluating violations of an artificial intelligence model. Background Technology

[0002] With the widespread deployment of large language models in scenarios such as intelligent customer service, content generation, code assistance, and financial analysis, their security and compliance governance has become a core challenge for the industry. In actual operation, large language models may generate unauthorized conclusions, bypass sensitive instructions, and generate high-risk content, among other violations. The causes of these violations are complex, and only by deeply and comprehensively locating the root causes can effective repair and governance of large language models be achieved.

[0003] Governance solutions often employ a combination of multiple technical approaches. For example, an auditing system based on logs and event streams can be used to collect input and output logs during model runtime and combine them with a rule engine for violation detection; or a content security classifier can be deployed at the model output to annotate and alert the output results in real time; some solutions also use red team testing to periodically assess the security of the model through adversarial prompts and use the test pass rate as a decision-making reference.

[0004] However, in practice, the aforementioned technologies typically only perform detection from a single or superficial dimension. For example, log- and classifier-based solutions can only record and determine violations at the input / output level or classify output content, but cannot delve into the model's internal attribution analysis; red team testing is mostly intermittent overall evaluation, making it difficult to support end-to-end tracing of single risk events.

[0005] The inherent limitations of the aforementioned technologies have resulted in a lack of transparency, comprehensiveness, and depth in identifying the root causes of violations during the assessment and analysis of AI model violations, thus hindering effective assistance in the governance of AI models. Summary of the Invention

[0006] Based on the aforementioned technical issues, this application provides a method, system, device, product, and storage medium for evaluating violations of artificial intelligence models. This method can analyze the reasons for violations of artificial intelligence models in a deep and comprehensive manner from multiple dimensions, thereby providing effective assistance for model governance and improving the effectiveness of model governance.

[0007] The first aspect of this application provides a method for evaluating violations of an artificial intelligence model, including: The identification of violations based on artificial intelligence models involves querying event information related to the violations from the monitoring system of the artificial intelligence models; the monitoring system is used to monitor the inference and training processes of the artificial intelligence models. Based on the event information, monitoring data corresponding to multiple evaluation dimensions are obtained from the monitoring system, and the obtained monitoring data is used to evaluate the reasons for violations of the artificial intelligence model for the corresponding evaluation dimensions, thereby determining the evaluation results of the reasons for violations corresponding to each of the multiple evaluation dimensions. The multiple evaluation dimensions include multiple dimensions such as training sample dimension, inference step dimension, neuron dimension, security constraint mechanism dimension, violation risk evolution dimension at different training stages, and violation risk dimension of neurons at different training stages.

[0008] The second aspect of this application provides a violation assessment system for an artificial intelligence model, comprising: A monitoring system is used to monitor the operation and / or training process of an artificial intelligence model and to store the monitoring data. An evaluation system is used to perform the violation evaluation method of the artificial intelligence model described above by interacting with the monitoring system through data.

[0009] A third aspect of this application provides an electronic device, including a memory and a processor; The memory is connected to the processor and is used to store programs; The processor is used to implement the aforementioned violation assessment method for the artificial intelligence model by running the program in the memory.

[0010] The fourth aspect of this application provides a computer program product, including computer program instructions, which, when executed by a processor, cause the processor to perform the violation assessment method of the aforementioned artificial intelligence model.

[0011] The fifth aspect of this application provides a storage medium storing a computer program, which, when executed by a processor, causes the processor to perform the violation assessment method of the artificial intelligence model described above.

[0012] The violation assessment method for artificial intelligence models provided in this application first obtains unified event information from a monitoring system based on the identifier of the violation event. Then, it utilizes monitoring data from multiple dimensions obtained from the monitoring system, including training sample dimensions, inference step dimensions, neuron dimensions, safety constraint mechanism dimensions, violation risk evolution dimensions at different training stages, and neuron violation risk dimensions at different training stages, to assess the corresponding dimensions of violation causes in the artificial intelligence model. This method uses the violation event identifier as a unified entry point, integrating and processing source data from multiple dimensions such as the data layer, model layer, neuron layer, loop layer, and temporal layer, to establish an end-to-end, multi-layered causal attribution chain for violation events, from external violation phenomena to the internal mechanisms of the model. This achieves comprehensive and transparent root cause tracing for a single violation event. This solution solves the technical problem of existing technologies that can only detect violations from a single or superficial dimension, failing to delve into the multi-layered mechanisms within the model for attribution, resulting in opaque root cause localization of violation risks and the inability to establish an end-to-end causal chain. Therefore, it provides effective assistance for model governance and helps improve model governance effectiveness. Attached Figure Description

[0013] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of this application. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.

[0014] Figure 1 This is a schematic diagram of a violation assessment system for an artificial intelligence model provided in an embodiment of this application.

[0015] Figure 2 This is a flowchart illustrating a violation assessment method for an artificial intelligence model provided in an embodiment of this application.

[0016] Figure 3 This is a flowchart illustrating another violation assessment method for an artificial intelligence model provided in an embodiment of this application.

[0017] Figure 4 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0018] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0019] In the field of tracing the causes of violations in artificial intelligence models, to detect and alert on violations that occur during model operation, relevant technologies typically employ monitoring methods based on output logs or content classifiers. Specifically, these solutions deploy content security classifiers or rule engines at the model's output end to perform real-time judgments on each output, and combine this with a log system to record request and response event streams. When violating content is detected, an alert is triggered. The initial design goal is to quickly intercept perceptible violations, providing basic compliance monitoring capabilities for the model at a relatively low implementation cost.

[0020] However, when it's necessary to pinpoint the root cause of a specific violation and then use the results to repair the model and quantify the effects, the aforementioned solutions reveal a conflict between detection efficiency and depth of investigation. In pursuit of rapid judgment, these solutions limit data collection and analysis to the input / output level or the final classification score, failing to delve into the model's internal computation and processing. For example, when faced with complaints about the model outputting an unauthorized conclusion, relying on the output classifier only tells us that the output is marked as a violation, but it cannot explain which batch of training data, which inference logic step, which neuron, or which safety constraint loop led to the violation. This makes subsequent targeted repairs lack a basis, and it's difficult to establish a unified quantitative baseline for comparing the effects before and after the remediation.

[0021] The root cause of the above problems lies in the lack of a multi-level monitoring data collection mechanism that collaboratively collects and integrates the external operational phenomena and internal computing mechanisms of the model, and performs multi-level root cause analysis of violations from the phenomena to the internal mechanisms based on the multi-level monitoring data. Therefore, it is impossible to support end-to-end transparent root cause tracing of a violation event.

[0022] To address the aforementioned technical issues, this application proposes a different technical approach. Its core concept lies in: based on the identification of violation events, acquiring monitoring data from a monitoring system corresponding to multiple dimensions, including training sample dimensions, inference step dimensions, neuron dimensions, security constraint mechanism dimensions, violation risk evolution dimensions at different training stages, and neuron violation risk dimensions at different training stages. This data is then used to evaluate the violation causes of the artificial intelligence model across these dimensions, thereby achieving multi-level root cause analysis of violations from surface phenomena to internal mechanisms. Furthermore, after obtaining the violation cause evaluation results, the repaired model undergoes performance evaluation covering multiple indicator dimensions, including output quality, version stability, adversarial robustness, training process health, and neuron risk. The technical solution provided in this application solves the problem in related technologies where the single information dimension prevents in-depth root cause tracing and quantification of governance effects, achieving multi-level attribution from external violation phenomena to the model's internal mechanisms and measurable comparison of governance effects.

[0023] Figure 1 This is a schematic diagram of an exemplary implementation environment of the technical solution provided in the embodiments of this application.

[0024] The implementation environment is a violation assessment system based on an artificial intelligence model; see [link / reference]. Figure 1 As shown, the system includes a monitoring system 100 and an evaluation system 200. The monitoring system 100 and the evaluation system 200 establish a communication connection through a network to achieve data exchange.

[0025] The monitoring system 100 is used to monitor the operation and training process of the artificial intelligence model and store monitoring data. Specifically, it can be a system or platform for continuously or periodically observing, recording, analyzing, and storing relevant monitoring data on the operation and / or training process of the artificial intelligence model. For example, it may include, but is not limited to: a model evaluation platform, a training phase evaluation snapshot database, a model version registry, an inference service performance monitoring log system, an adversarial evaluation platform, or a combination thereof.

[0026] Specifically, the monitoring system 100 is configured with multiple data sources, such as a work order database, a session log system, a training data traceability management system, a model observability analysis service, a training phase evaluation snapshot database, and a model evaluation platform. These data sources record the status information of the artificial intelligence model at different stages and dimensions. The monitoring system 100 provides multiple data query interfaces, each corresponding to one or a type of monitoring data acquisition method, and responds to query requests from external systems through standardized communication protocols.

[0027] The assessment system 200 is the main body responsible for performing assessments of the causes of violations and quantitative evaluations of governance effectiveness. Internally, the assessment system 200 includes several modules for data acquisition, data verification, risk attribution analysis, quantitative governance assessment, and audit report generation. Logically, these modules form a complete processing pipeline from receiving tasks to outputting assessment results.

[0028] In the data flow, the assessment system 200 first generates query instructions for the monitoring system 100 based on violation event identifiers obtained from the upstream risk detection system or manual input. Upon receiving these query instructions, the monitoring system 100 retrieves corresponding data from its managed data sources and returns a structured response message containing event information, monitoring data, and indicator data to the assessment system 200. After acquiring this data, the assessment system 200 performs multi-dimensional analysis and generates results according to predetermined processing logic.

[0029] This implementation environment provides the data foundation and computational framework for subsequent specific assessment steps, enabling the root cause tracing of violations and the evaluation of governance effectiveness by artificial intelligence models to be achieved within a unified data acquisition and processing pipeline.

[0030] Based on the aforementioned violation assessment system using an artificial intelligence model, this application provides a violation assessment method using an artificial intelligence model, which can be executed by the assessment system 200 within the aforementioned violation assessment system.

[0031] See Figure 2 As shown in the embodiments of this application, the method for evaluating violations of an artificial intelligence model includes: S101. Identification of violations based on artificial intelligence models: Querying event information related to violations from the monitoring system of artificial intelligence models.

[0032] The aforementioned violations refer to any output or behavioral events generated by the artificial intelligence model during its operation that are determined by the upstream risk detection system to be inconsistent with the preset security or compliance policies.

[0033] The identifier for a violation event, also known as the risk event ID, is a globally unique sequence of characters or numbers used to uniquely locate a specific violation event during cross-system data interaction. This identifier can be automatically assigned by the upstream risk detection system when a violation is detected for the first time, and its format can be defined according to requirements, such as a string like "RE-20250228-001".

[0034] There are several ways for the assessment system 200 to obtain the identifier of the violation event. For example, after receiving an upstream alarm, the auditor or security engineer can obtain the identifier from the alarm notification or work order system and manually fill it in through the configuration input interface of the assessment system 200. Alternatively, when the assessment system 200 is integrated with the upstream alarm system, the alarm message automatically carries the identifier and triggers the root cause analysis task of the violation.

[0035] The aforementioned event information refers to any initial information related to a violation that has occurred, used to describe the context of the event or identify its source. For example, it may include, but is not limited to, at least one of the following: work order identifier, session identifier, session anomaly score, violation type, or a combination thereof.

[0036] The evaluation system 200 uses the identifier of the violation event as the search condition to query the violation event record table of the monitoring system 100 to see if the violation event exists. If it exists, it reads the violation type of the violation event (such as bypassing sensitive instructions, unauthorized tool calls, high-risk content generation, etc.) from the violation event record table. Then, using the identifier of the violation event as the search condition, it queries the work order database, session log database, and log anomaly scoring database of the monitoring system 100 to find the work order identifier, session identifier, and session anomaly score related to the violation event, thereby obtaining the event information related to the violation event.

[0037] In another embodiment, step S101 may specifically involve: identifying the violation event based on the artificial intelligence model, and obtaining event information related to the violation event from the monitoring system 100 through the ninth data query interface set in the monitoring system 100. The event information may include, but is not limited to, at least one of the following: work order identifier, session identifier, session anomaly score, and violation type.

[0038] Specifically, the assessment system 200 uses the identifier of the violation event as the primary key to initiate a query request to the ninth data query interface of the monitoring system 100, namely the signal / collect endpoint. After receiving the risk event ID from the assessment system 200, the signal / collect endpoint first verifies whether the event exists in the risk event record table and reads the violation type; then it queries the work order database, the session log database, and the log anomaly scoring database respectively; finally, it merges the fields returned by each upstream system into a structured JSON object according to a unified framework and returns it as the event information.

[0039] Event information can include data from multiple sources and dimensions. For example, it can include work order identifiers, session identifiers, log anomaly scores, violation types, etc.

[0040] The ticket identifier (ticket_signal) can be a ticket number automatically generated by the enterprise's ticket management system when it receives a user complaint or internal security alarm, such as "Ticket-FB-998".

[0041] The session identifier (session_signal) can be a unique ID assigned to each user session by the model dialogue management platform, such as "S-12345". The evaluation system 200 can use this identifier to associate it with the specific dialogue record that triggered the violation.

[0042] The session anomaly score (log_anomaly) is a numerical score given in real time by the log anomaly detection service to the request and response logs for the current session. The value ranges from 0 to 1, and the higher the score, the higher the degree of anomaly.

[0043] The violation type (trigger) can be a category label assigned by the upstream content security classifier when a violation is detected for the first time, such as "sensitive instruction bypass", "unauthorized tool call", "high-risk content generation", etc.

[0044] The above-mentioned event information is aggregated within the signal / collect endpoint using the violation event ID, and the output is a violation event instance uniquely identified by the violation event ID, which serves as the unified context input for subsequent evaluation steps.

[0045] The event information obtained in step S101 provides accurate and comprehensive initial input for all subsequent evaluation dimensions based on event information. Event information links the external manifestations of an event with the multi-source clues required for internal tracing, enabling previously isolated work orders, sessions, and log data to collaboratively support in-depth violation attribution analysis.

[0046] S102. Based on event information, obtain monitoring data corresponding to multiple evaluation dimensions from the monitoring system, and use the obtained monitoring data to evaluate the reasons for violations in the corresponding evaluation dimensions of the artificial intelligence model, and determine the evaluation results of the reasons for violations corresponding to each of the multiple evaluation dimensions.

[0047] In the embodiments of this application, the term "evaluation dimension" refers to an independent analytical perspective that analyzes the causes of violations by artificial intelligence models from different technical levels or analytical angles. Each evaluation dimension corresponds to a specific set of monitoring data and a specific set of evaluation logic, used to reveal the causes of violations in that dimension.

[0048] Specifically, the multiple evaluation dimensions include training sample dimension, inference step dimension, neuron dimension, security constraint mechanism dimension, violation risk evolution dimension at different training stages, and multiple dimensions of violation risk of neurons at different training stages.

[0049] Specifically, the analysis can be conducted through the following dimensions: Training sample dimension: whether the training samples of the AI ​​model cause the AI ​​model to trigger a violation event; Inference step dimension: which inference step(s) of the AI ​​model drifts, causing the model to trigger a violation event; Neuron dimension: which neuron(s) of the AI ​​model are abnormal, causing the model to trigger a violation event; Security constraint mechanism dimension: which part of the AI ​​model's security constraint mechanism malfunctions, causing the model to trigger a violation event; Violation risk evolution dimension at different training stages: the evolution process of violation risk of the AI ​​model at different training stages, thus determining at which training stage the AI ​​model's compliance deteriorates; and Neuron violation risk dimension at different training stages: at which training stage do the AI ​​model's neurons malfunction, causing the violation risk.

[0050] By using these six dimensions—data source layer, model reasoning logic layer, minimum computing unit layer, security function structure layer, training timeline layer, and spatiotemporal intersection layer—a three-dimensional and multi-layered root cause tracing of a single violation event can be performed.

[0051] The aforementioned monitoring data refers to raw data that reflects the attributes or state of an artificial intelligence model on a specific evaluation dimension. This data is obtained from the monitoring system 100 or other data sources and is used as input for subsequent violation cause assessment. For example, it may include, but is not limited to: the vector representation of the violation output in the intermediate layers of the model and its gradient with respect to the loss function, the hidden state vector and attention weight matrix of each inference step, the baseline activation mean and risk attribution score of each neuron, safety constraint component information, the harmful output rate of each training stage, or a combination thereof.

[0052] As a specific implementation method, monitoring data can be acquired through multiple standardized data query interfaces predefined in the monitoring system 100. These interfaces are standardized HTTP API interfaces deployed in the backend service layer or evaluation platform service layer of the monitoring system 100, responsible for responding to data query requests from the evaluation system 200 on demand. The evaluation system 200 can construct query requests for different interfaces based on various parameters in the event information obtained in step S101, such as risk event ID, model name, and version number, and acquire monitoring data corresponding to each dimension in parallel or serially.

[0053] For each evaluation dimension, a corresponding violation reason evaluation logic is set. By analyzing or evaluating the monitoring data corresponding to the evaluation dimension according to the evaluation logic, the cause of the model violation event can be found from the evaluation dimension.

[0054] For example, for the training sample dimension, the evaluation system 200 can obtain the vector representation of the violation output in the intermediate layer of the artificial intelligence model and the gradient relative to the security loss function or violation category score through the sixth data query interface based on the event information. Based on this, it can retrieve the training sample that contributes the most to the violation event from the training sample dataset and determine the dataset batch identifier, sample-level identifier and contamination score of the sample.

[0055] Regarding the reasoning step dimension, the evaluation system 200 can obtain the reasoning text, hidden state vector, and attention weight matrix generated by the artificial intelligence model for each reasoning step in the violation event through the seventh data query interface, and determine the violation reasoning step and its attribution score based on these reasoning elements.

[0056] For the neuron dimension, the evaluation system 200 can obtain the performance evaluation results of each neuron of the artificial intelligence model (at least one of baseline activation mean, risk attribution score, and stability at different training stages, and must include risk attribution score) through the fifth data query interface, and determine the neuron with the highest risk attribution score accordingly.

[0057] Regarding the security constraint mechanism dimension, the evaluation system 200 can obtain information on the security constraint components within the artificial intelligence model through the eighth data query interface, model the security constraint mechanism as a directed weighted computation loop, determine the weight of each connecting edge through path repair experiments, and determine the connecting edge with the largest weight as the centralized path point for risk propagation.

[0058] For the evolution dimension of violation risk in different training stages, the evaluation system 200 can obtain the stage performance evaluation results of the artificial intelligence model in each training stage through the fourth data query interface, including at least one of harmful output rate, neuronal instability and path drift score, and determine the training stage with the highest violation risk.

[0059] For the violation risk dimension of neurons in different training stages, the evaluation system 200 can comprehensively use the above-mentioned fourth data query interface and fifth data query interface to obtain the neuron with the highest risk attribution score and its stability in each training stage, as well as the harmful output rate in each training stage, and calculate and determine the violation risk score of the neuron in each training stage to achieve root cause localization in the time-series dimension.

[0060] More details on the specific implementation methods of the above-mentioned evaluation dimensions will be elaborated in the following examples, and will not be discussed here.

[0061] The aforementioned steps S101 and S102 together constitute an end-to-end, multi-layered causal attribution chain from external violation phenomena to the internal mechanisms of the model. Step S101 uses the identification of the violation event as a unified entry point, connecting signal sources scattered across multiple upstream systems and generating a complete event context. Step S102, based on this, projects the analytical perspective simultaneously onto the data layer, model inference layer, neuron layer, security loop layer, and training time sequence layer, acquiring and fusing monitoring data from multiple dimensions and performing corresponding analyses. This multi-dimensional collaborative evaluation method changes the limitation of related technologies that can only detect from a single or superficial dimension. It allows the location of the root cause of risk to advance from "a violation occurred" to the depth of "which specific training sample, in which inference step, through which neuron's abnormal activation, along which loop path, and ultimately in which training stage the violation occurred," solving the technical problems of opaque attribution of violation risks in existing artificial intelligence models and the inability to establish an end-to-end causal chain.

[0062] Figure 3This is a schematic flowchart illustrating a violation assessment method for an artificial intelligence model provided in another embodiment of this application. Figure 2 Based on the illustrated embodiment, the method of this embodiment may further include a step of quantitatively evaluating the remediation and treatment effects: S103. Based on the evaluation results of the reasons for violations corresponding to multiple evaluation dimensions, repair the artificial intelligence model.

[0063] Specifically, the evaluation system 200, after completing the execution... Figure 2 Following the analysis of the violations in the illustrated method, the root causes of the violations have been pinpointed to the training samples, inference steps, high-risk neurons, loop propagation paths, and critical training phases. Based on these detailed attribution conclusions, the development team can take targeted remedial measures outside the evaluation system 200. These measures could include cleaning or removing contaminated training data, fine-tuning the model to correct biases in specific inference chains, strengthening weak links in safety constraint loops, or optimizing reinforcement learning strategies for specific training phases. Alternatively, these remedial measures can be implemented through the evaluation system 200 under the developers' control, or the evaluation system can execute them autonomously.

[0064] S104. Obtain indicator data corresponding to multiple indicator dimensions from the monitoring system, and use the obtained indicator data to perform performance evaluation on the repaired artificial intelligence model for the corresponding indicator dimensions, so as to obtain the performance evaluation results corresponding to each of the multiple indicator dimensions.

[0065] In this embodiment, the indicator dimension refers to the evaluation perspective used from different technical angles to measure whether the remediated model meets the expected security and quality requirements during the quantitative evaluation phase of governance effectiveness. Multiple indicator dimensions may include output quality, version stability, adversarial robustness, training process health, and neuron risk. These five indicator dimensions comprehensively evaluate the remediation effect from aspects such as the compliance of model output content, the security of version iteration, the ability to defend against malicious attacks, risk control throughout the training lifecycle, and the residual risk of internal micro-computing units.

[0066] Metric data generally refers to data that reflects the performance of a repaired AI model on specific metric dimensions. This data is obtained from the monitoring system and used as input for subsequent performance evaluation. For example, it may include, but is not limited to: harmful output rate and policy violation rate before and after model repair, inference latency and security regression risk of each historical version, test pass rate in multiple adversarial scenarios, maximum harmful output rate of each training stage, risk attribution score for each neuron, or a combination thereof.

[0067] As a specific implementation method, the evaluation system 200 can obtain the indicator data corresponding to each indicator dimension through different data query interfaces set in the monitoring system 100.

[0068] For example, regarding the output quality dimension, the evaluation system 200 can obtain output quality evaluation data before and after model repair through the first data query interface set in the monitoring system 100. This evaluation data includes at least one of harmful output rate, policy violation rate, robustness score, and audit coverage, and necessarily includes harmful output rate; and determines the output quality improvement effect of the repaired model based on the data before and after repair.

[0069] Regarding the version stability dimension, the evaluation system 200 can obtain the performance evaluation data of each version of the model through the second data query interface. This evaluation data includes at least one of harmful output rate, inference latency, usability score and security regression risk, and necessarily includes security regression risk; and determines the security regression risk of the repaired model based on the data of each version.

[0070] Regarding the adversarial robustness dimension, the evaluation system 200 can obtain the adversarial test results of the repaired model in multiple adversarial scenarios such as jailbreak prompts, multi-round inducement, tool injection, adversarial rewriting, and extreme scenarios through a third data query interface. These test results include at least one of pass rate and boundary stability score, and pass rate must be included. Based on these results, the average pass rate of the repaired model in multiple adversarial scenarios is determined.

[0071] Regarding the training process health dimension, the evaluation system 200 can obtain the stage performance evaluation results of the repaired model at each training stage through the fourth data query interface, including at least one of harmful output rate, neuronal instability and path drift score, and necessarily including harmful output rate; and determine the maximum value of harmful output rate of the repaired model at each training stage based on these results.

[0072] For the neuron risk dimension, the evaluation system 200 can obtain at least one of the following for each neuron in the repaired model through the fifth data query interface: baseline activation mean, risk attribution score, and stability at different training stages, and must include the performance evaluation results of the risk attribution score; and based on these results, determine the neuron with the highest risk attribution score in the repaired model and its corresponding risk attribution score.

[0073] More detailed implementation methods for performance evaluation of the above-mentioned metrics will be described in the following examples.

[0074] Through the aforementioned steps S103 and S104, this embodiment of the application technically integrates the two stages of "problem localization" and "effect verification." First, based on the multi-dimensional violation cause assessment results obtained in step S102, the model is targeted for repair. Then, based on the unified data benchmark provided by the monitoring system 100, the repaired model undergoes a multi-dimensional performance evaluation corresponding to the cause assessment dimensions. This closed-loop design of "attribution-guided repair and quantitative verification of repair" solves the problem of difficulty in quantifying and comparing the effects of repair measures in related technologies. It allows for measurable and objective comparative analysis of governance effects from multiple technical perspectives, such as output quality, version stability, adversarial robustness, training process health, and neuron risk.

[0075] To further address the issue that decisions made in multi-dimensional governance quantitative assessments are easily subject to single-indicator game-theories due to the reliance on a single decision-making basis, this application also provides a preferred scheme for multi-rule combined gating.

[0076] In this preferred scheme, when multiple indicator dimensions specifically include all five dimensions such as output quality, version stability, adversarial robustness, training process health, and neuron risk, the evaluation system 200, after obtaining the performance evaluation results corresponding to each indicator dimension, further executes combined gating logic: the artificial intelligence model is determined to be successfully repaired only if the performance evaluation results corresponding to each indicator dimension meet the performance requirements corresponding to that dimension.

[0077] Specifically, the evaluation system 200 has five pre-defined independent judgment rules corresponding to the five indicator dimensions. For example, rule R1 requires that the harmful output rate (after remediation) in the output quality dimension not exceed the user-configurable `max_harmful_rate` threshold; rule R2 requires that the maximum security regression risk of all historical versions in the version stability dimension not exceed the `max_regression_risk` threshold; rule R3 requires that the average pass rate of multiple adversarial scenarios in the adversarial robustness dimension not be lower than the `min_avg_stress_pass` threshold; rule R4 requires that the peak of the harmful output rate in each training stage in the training process health dimension not exceed the `max_peak_stage_risk` threshold; and rule R5 requires that the risk attribution score of the neuron with the highest risk attribution score in the neuron risk dimension not exceed the `max_top_neuron_risk` threshold. All five threshold parameters can be flexibly configured by the user according to actual compliance requirements and input through the evaluation system 200.

[0078] The evaluation system 200 executes logic and operations based on the judgment results of each rule. When all five rules are judged as passed, the gating state is PASS, confirming successful model repair; if any rule fails, the gating state is BLOCK, and the model repair is considered incomplete. The gating score can be defined as the ratio of the number of passed rules to the total number of rules.

[0079] The technical effectiveness of this combined gating mechanism lies in the fact that the data sources for the five rules cover five mutually orthogonal data dimensions: output quality, version stability, adversarial robustness, training process health, and neuron risk. Any attempt to "gamble" on the release decision through over-optimization of a single-dimensional metric will fail to simultaneously ensure that the rules in the other four dimensions meet the pass conditions. Therefore, this mechanism technically ensures the robustness and reliability of the release decision, preventing the risk of masking model defects in other key safety dimensions due to single-metric optimization.

[0080] During data acquisition and processing, the evaluation system 200 relies on multiple external data query interfaces provided by the monitoring system 100, and may encounter unstable factors such as network anomalies and data format errors during actual operation. To ensure the basic availability of the evaluation process under these circumstances without sacrificing audit traceability, this application embodiment further provides a degradeable data acquisition and verification scheme.

[0081] Specifically, in step S102 or step S104, after obtaining monitoring data or indicator data from the monitoring system 100, the scheme adds a data verification and downgrade processing step before using these data for analysis and evaluation.

[0082] First, the data validation module of the evaluation system 200 performs pattern validation on each piece of monitoring data or indicator data to determine whether it conforms to the corresponding preset data specifications. Validation may include checking the existence of top-level necessary fields, data type of fields, integrity of nested structures, and array non-empty checks. For example, for data returned by the signal / collect endpoint, the validation module checks whether it necessarily contains necessary fields such as ticket_signal, session_signal, log_anomaly, and trigger, and confirms whether the log_anomaly field is numeric.

[0083] Secondly, when it is determined that the source data for any evaluation dimension does not conform to the corresponding data specifications, and the system has allowed fallback, the data acquisition module of the evaluation system 200 will deterministically generate a structurally compliant simulated monitoring data based on the data query interface name used to obtain the source data and the request parameters of this data request, and will inject this simulated data as the monitoring data for that evaluation dimension into the subsequent analysis process. Simultaneously, the evaluation system 200 records the mock_fallback flag and the specific reason for the pattern error in the source mapping table for auditing purposes.

[0084] If the actual API request itself encounters an anomaly, such as a connection timeout or network error, and fallback is enabled, the handling logic is similar: the system switches to simulated data and records the `mock_fallback` and the exception type name. If fallback is not enabled, regardless of whether pattern validation fails or the request is abnormal, the system directly throws a runtime error, preventing subsequent processes from executing to ensure the accuracy of the analysis conclusions.

[0085] This solution ensures that the assessment process will not be interrupted when the data source is unreliable, thus guaranteeing system availability. At the same time, by recording the actual source of each piece of data in detail (real interface, simulated data and the reasons for its degradation), it provides transparent and traceable clues for subsequent audit reports, thereby solving the problem of balancing high availability of the assessment system with data authenticity.

[0086] After completing the above-mentioned multi-dimensional assessment of the causes of violations and the quantitative assessment of governance effectiveness, in order to solve the problems of audit reports relying on manual writing, low efficiency and incomplete evidence chains in related technologies, this application also provides a preferred solution for automatic generation of audit reports.

[0087] In this preferred embodiment, the evaluation system 200 integrates the structured and unstructured data generated throughout the evaluation process and automatically populates it into a predefined audit report template. Specifically, the evaluation system 200 generates a model audit report containing a complete chain of evidence based on the event summary information of the violation event, the monitoring data corresponding to each of the multiple evaluation dimensions, the verification results of these monitoring data, the evaluation results of the reasons for the violation corresponding to each of the multiple evaluation dimensions, and the performance evaluation results corresponding to each of the multiple indicator dimensions.

[0088] The assessment system 200 can automatically extract key data from multiple analytical steps in the risk attribution tracing process to construct a structured evidence package. Each piece of evidence can include a unique evidence identifier (evidence_id), a data source endpoint path (path), a key data extract (excerpt), and a data actual source marker (source: api / mock_fallback), forming a standardized evidence chain that is both machine-readable and human-auditable. These evidence items, along with the verification results of monitoring data and the assessment conclusions of various dimensions, are automatically populated by a templated report rendering engine into a standard report template that includes sections such as a risk event overview, risk cause tracing conclusions, quantitative assessment of governance effectiveness, an audit evidence package reference table, release gate and interface verification results, and compliance conclusions and recommendations. Finally, the system can simultaneously export audit report files, evidence package files, and gate decision record files, achieving fully automated generation from data analysis to audit conclusions, ensuring the structured, standardized, and traceable nature of audit documents.

[0089] To further elaborate on how to assess the reasons for violations in the training sample dimension, this application provides a reverse data tracing scheme based on influence functions and source maps.

[0090] In this embodiment, the step of "obtaining monitoring data corresponding to the training sample dimension from the monitoring system based on event information, and using the monitoring data corresponding to the training sample dimension to evaluate the reasons for violations in the artificial intelligence model, and determining the evaluation results of the reasons for violations corresponding to the training sample dimension" is specifically defined as the following three-stage process: In the first phase, the evaluation system 200, based on the event information of the violation event, obtains the process data of the violation event from the monitoring system 100 through the sixth data query interface (i.e., the trace / data-root-cause endpoint) set in the monitoring system 100. This process data includes the vector representation of the violation output in the specified intermediate layer of the artificial intelligence model, and the gradient of the violation output relative to the security loss function or violation category score. Specifically, based on the risk event ID in the event information, the evaluation system 200 reads the violation input text, violation output text, and the corresponding model version number associated with the event, and calls the model's feature extraction interface to perform one forward and backward propagation calculation to capture the aforementioned vector representation and gradient information.

[0091] In the second stage, the evaluation system 200 uses the gradient information obtained in the first stage to retrieve the target training sample that contributes the most to the violation event from the training sample dataset. The core principle of the retrieval is to use the influence function method to estimate the direction of the update influence of each training sample on the model parameters during model training, and compare it with the gradient direction corresponding to the current violation output. If the parameter update direction caused by a training sample is highly consistent with the gradient direction of the violation output, it indicates that the training sample has greatly promoted the model to learn the behavior of generating such violation output. Based on this, the evaluation system 200 calculates the contribution of each candidate sample to the violation event and selects the sample with the highest contribution as the target training sample.

[0092] In the third stage, the evaluation system 200 determines the data information of the target training sample. This data information includes the dataset batch identifier (dataset_batch), the sample-level identifier (sample_id), and the contamination score (contamination_score). Specifically, after retrieving the target training sample, the evaluation system 200 can further query the training data source management system to map the sample's primary key to its corresponding dataset batch, collection source, and other metadata, ultimately determining the dataset batch (dataset_batch) and the sample primary key (sample_id). The contamination score can be directly determined based on the normalized result of the contribution in the second stage, with a value ranging from 0 to 1. A higher value indicates a stronger causal relationship between the training sample and the violating output.

[0093] This approach allows for the direct tracing of the causes of violations to the original data source that led the model to learn the violation, enabling precise localization from risk events to specific training samples and solving the problem of being unable to trace the root cause of training data contamination.

[0094] To further elaborate on how to evaluate the reasons for violations at the inference step dimension, this application provides an inference chain analysis scheme based on security constraint decay detection and attention attribution.

[0095] In this embodiment, the evaluation of violation reasons corresponding to the inference step dimension is specifically limited to the following process: First, based on event information, the evaluation system 200 obtains the inference elements of each inference step of the artificial intelligence model in the violation event through the seventh data query interface (i.e., the trace / model-root-cause endpoint) set in the monitoring system 100. These inference elements include the inference text generated for each inference step, the hidden state vector, and the corresponding attention weight matrix.

[0096] Specifically, the evaluation system 200 uses the original violation input associated with the event information, the model name, and the version number as query parameters to call the observable mode of the model's inference engine. In this mode, the model performs inference normally, but at each reasoning step, it additionally outputs and records the generated intermediate token sequence, hidden state vector, and attention weight matrix, forming a complete replay record of the violation inference.

[0097] Next, the evaluation system 200 determines the degree of compliance of each inference step with the security policy based on the inference elements of each inference step. Specifically, the evaluation system 200 can use a pre-trained policy probe classifier to process the hidden state vector of each inference step and output a policy compliance score between 0 and 1. The higher the score, the more the inference state of that step conforms to the predetermined security policy.

[0098] Then, the evaluation system 200 scans the compliance score sequence of all reasoning steps and identifies the reasoning step with the highest compliance score from those reasoning steps with compliance scores below a set compliance threshold, and identifies it as the violating reasoning step. The actual meaning of this step is to find the precise transition point where the security constraint jumps from "compliance" to "non-compliance", that is, the starting step (reasoning_step) where the security constraint begins to decay.

[0099] With this approach, the evaluation system 200 can accurately capture and locate at which specific step in the inference chain the security constraints begin to fail, thus solving the problem of being unable to attribute the internal reasoning process of the model.

[0100] After identifying the erroneous reasoning step, in order to further refine the attribution, explain which specific semantic unit in the input triggered the security decay of the step, and quantify the severity of the reasoning deviation, another embodiment of this application also provides an attention token tracking and comprehensive attribution scoring scheme.

[0101] After identifying the violation reasoning step using the aforementioned scheme, the evaluation system 200 further analyzes the attention weight matrix of that reasoning step. The evaluation system 200 uses an integral gradient method to perform gradient attribution on the input token sequence, calculating the causal contribution of each input semantic unit to the event that causes the security policy compliance of that reasoning step to fall below the threshold (i.e., the constraint decay event). The input semantic unit with the highest attribution score is the violating semantic unit (attention_token), representing the specific input word or phrase that most strongly drives the weakening of security constraints.

[0102] Next, the evaluation system 200 calculates a comprehensive attribution score. The calculation involves a weighted average of two factors. The first factor is the decrease in security policy compliance due to the violation reasoning step; specifically, it refers to the decrease in compliance of the violation reasoning step relative to the lowest compliant reasoning step. The lowest compliant reasoning step is the one with the lowest compliance among those reasoning steps whose compliance is not lower than the compliance threshold. This decrease is normalized. The second factor is the normalized result of the integral gradient attribution value (contribution) of the semantic unit that induced the violation, determined in the previous steps. The weighted average of these two normalized factors yields the comprehensive attribution score (attribution_score) for the violation caused by that reasoning step, ranging from 0 to 1, reflecting the severity of the model's deviation at the inference level.

[0103] This scheme advances the granularity of attribution from the reasoning step level to the input token level, not only locating "which step" went wrong, but also explaining "which word" triggered the error, and quantifying the degree of impact through a comprehensive score.

[0104] To further elaborate on how to assess the causes of violations at the neuron level, this application provides a neuron-level risk attribution scheme based on activation repair and cross-stage analysis.

[0105] In this embodiment, the evaluation of violation causes corresponding to the neuron dimension is specifically defined as the following process: First, the evaluation system 200, based on event information, obtains the performance evaluation results for each neuron of the artificial intelligence model through the fifth data query interface (i.e., the trace / neuron-analysis endpoint) set in the monitoring system 100. This acquisition process and the specific data content can be used for... Figure 2 The violation reason assessment in step S102 shown can also be used in the same way. Figure 3 The step S104 shown describes the performance evaluation of the repaired model in terms of neuron risk metrics. This performance evaluation result includes at least one of the following: baseline activation mean, risk attribution score, and stability at different training stages, and necessarily includes the risk attribution score.

[0106] Among them, the baseline activation mean is the average activation value of neurons when performing forward propagation on a normal, non-violation-triggered control sample set, representing the baseline level under normal operating conditions.

[0107] The risk attribution score is a quantitative indicator that measures the causal contribution of a neuron to a specific illegal output. It is obtained by performing an activation patching experiment on each target neuron sequentially for the input that triggers the violation. This involves artificially "patching" the neuron's activation value to its baseline mean activation value, thereby masking the neuron's specific response to the current illegal input, and observing the magnitude of the decrease in the model's illegal output probability. A larger decrease indicates a higher causal contribution of the neuron to the illegal output. After normalization, this is recorded as the neuron's risk attribution score, ranging from 0 to 1.

[0108] The stability (stage_consistency) at different training stages is obtained by calculating the inverse of the normalized variance of the risk attribution score of the same neuron at multiple training stages such as SFT-early stage, SFT-mid stage, RLHF-early stage, and RLHF-convergence stage. The value ranges from 0 to 1. The higher the value, the more stable and persistent the risk characteristics of the neuron are at different training stages.

[0109] After obtaining the three indicators mentioned above for all target neurons, the evaluation system 200 sorts them in descending order based on the risk attribution score and identifies the neuron with the highest risk attribution score and its corresponding risk attribution score. This high-risk neuron is considered the key node leading to the violation at the neuron level. This approach advances the accuracy of risk localization from the model layer or inference chain layer to the level of individual neurons that form the basis of model computation, thereby further improving the granularity of violation root cause analysis.

[0110] To further elaborate on how to assess the causes of violations at the level of security constraint mechanisms, this application provides a scheme for neural circuit modeling and risk amplification node localization.

[0111] In this embodiment, the assessment of violation causes corresponding to the security constraint mechanism dimension is specifically defined as the following process: First, the evaluation system 200 obtains information on the safety constraint components within the artificial intelligence model by using the eighth data query interface (i.e., the trace / circuit-path endpoint) set in the monitoring system 100, based on event information.

[0112] Next, based on the acquired security constraint component information, the evaluation system 200 abstracts and constructs a directed security constraint path graph from the discrete functional components related to security constraints within the model. This path can include multiple path nodes connected in sequence: input prompt node, policy token clustering node, role boundary neuron node, tool permission loop node, and security header node, ultimately pointing to the risk output node. Adjacent path nodes are connected by directed edges, representing the direction of information flow and processing.

[0113] Then, the evaluation system 200 quantitatively determines the weight of each connecting edge in the safety constraint path by performing a path patching experiment. Specifically, for any pair of adjacent nodes A to B, the experiment replaces the input of node B with the activation representation passed from A to B under normal, non-violation input conditions, thereby blocking the information transmission of the current violation input on this specific path, and observes the change in the probability of the model's final violation output. The larger the change, the more critical the role of the directed edge (A→B) in propagating violation risk information. After normalizing this change, the weight of the connecting edge (edge_weight), ranging from 0 to 1, is obtained.

[0114] Finally, the evaluation system 200 scans the weights of all connected edges and determines the connected edge with the largest weight value as the central path point for propagating violation risk information in the security constraint path.

[0115] This scheme structures the security constraint mechanism within the model into a weighted directed graph that can be quantified and analyzed. Based on the above analysis process, the propagation path and weak points of violation risks in complex network structures can be clearly and quantitatively revealed.

[0116] To further elaborate on how to assess the causes of violations based on the evolution of violation risks at different training stages, this application provides a time-series analysis scheme based on risk auditing during the training stage.

[0117] In this embodiment, the evaluation of violation causes corresponding to the violation risk evolution dimension at different training stages is specifically defined as the following process: First, based on event information, the evaluation system 200 obtains the stage performance evaluation results of the artificial intelligence model at each training stage from the monitoring system 100 through the fourth data query interface (i.e., the trace / train-stage-risk endpoint) set in the monitoring system 100. This evaluation result includes at least one of harmful output rate, neuronal instability, and path drift score.

[0118] These training phases can span the entire training lifecycle of the model, including, for example, the early stage of supervised fine-tuning (SFT-early stage), the middle stage of supervised fine-tuning (SFT-middle stage), the early stage of reinforcement learning based on human feedback (RLHF-early stage), and the convergence stage of reinforcement learning based on human feedback (RLHF-convergence stage). Performance evaluation results for each phase can be obtained by querying a database of periodically saved phase evaluation snapshots during model training. For example, the harmful output rate is the proportion of outputs judged as violations when running inference on a harmful content benchmark set during the checkpoint of this phase; neuron instability is the mean inter-batch standard deviation of the activation mean when multiple batches of high-risk neurons are evaluated during this phase; and path drift score is the cosine distance between the weight vectors of each edge of the safety constraint loop in this phase and the weight vectors of the previous phase, used to measure the overall structural change of the safety loop.

[0119] After obtaining the performance evaluation results of each training stage, the evaluation system 200 identifies and determines the training stage with the highest risk of violation by comparing the indicators of each stage longitudinally, such as comparing the harmful output rate of each stage.

[0120] This approach depicts the dynamic evolution curve of model violation risk throughout the entire model training lifecycle from a time perspective. It can effectively capture the critical training stages where violation risk is most severe, providing a basis for understanding the causes of violation risk and formulating optimization strategies at the training level.

[0121] To further integrate the micro-risks at the neuron level with the macro-risks at the training stage level, and to reveal the spatiotemporal interaction patterns of risks in "when and where", this application also provides a scheme for neuron-training stage risk transfer analysis.

[0122] In this embodiment, the evaluation of violation reasons corresponding to the violation risk dimensions of neurons at different training stages is specifically defined as the following process: First, the evaluation system 200 needs to acquire three basic data items required for computation. First, it obtains the stage performance evaluation results of the AI ​​model at each training stage, including at least the harmful output rate, through the fourth data query interface. Second, it obtains the performance evaluation results corresponding to each neuron of the AI ​​model through the fifth data query interface, and determines the neuron with the highest risk attribution score and its corresponding risk attribution score based on the risk attribution score. The performance evaluation result of this neuron includes the risk attribution score and its stability at different training stages. The methods for acquiring these two data items have been detailed above and will not be repeated here.

[0123] Then, the evaluation system 200 calculates the risk transfer score for each training stage. Specifically, for the neuron i with the highest risk attribution score and training stage j, its risk transfer score is obtained by multiplying the risk attribution score of neuron i (attribution_to_risk_i), the stability of neuron i in training stage j (stage_consistency_i), and the normalized value of the harmful output rate in training stage j (normalized_harmful_rate_j).

[0124] The normalized value of the harmful output rate in training phase j is obtained by calculating the ratio of the harmful output rate in training phase j to the maximum value of the harmful output rate in all training phases, max(harmful output rate_all_stages).

[0125] The logic behind the above calculation method is as follows: the three factors respectively measure "how dangerous the neuron itself is," "whether the danger is consistently reliable," and "how severe the overall risk environment of the training phase is." Only when all three conditions are met simultaneously can the neuron be considered a high-risk contributor in that training phase. If the value of any factor approaches 0, it will pull the overall score towards 0, thus effectively filtering out false alarms such as "occasional dangerous neurons being in a low-risk training phase." This score can ultimately be presented as a heatmap on a two-dimensional matrix composed of neurons and training phases, with the color intensity intuitively reflecting the spatiotemporal concentration areas of risk.

[0126] This approach addresses the problem that single-dimensional analysis cannot characterize how risk interacts and evolves in space (which neuron) and time (which training stage), achieving higher-precision temporal root cause localization.

[0127] In any of the above embodiments involving data query interfaces, the acquisition of event information can be specifically implemented as a multi-source information fusion collection scheme.

[0128] Specifically, the evaluation system 200, based on the identifier of the violation event, obtains event information related to the violation event from the monitoring system 100 through the ninth data query interface set in the monitoring system 100. This event information includes at least one of the following: work order identifier, session identifier, session anomaly score, and violation type. The specific source, acquisition method, and fusion process of each field have been described in the detailed description of step S201 of the aforementioned embodiment.

[0129] Through this standardized data query and fusion interface, the assessment system 200 can aggregate signal fragments scattered across different business systems into a single, context-complete risk event instance, using the violation event ID as the link. This provides a unified and comprehensive information input for all subsequent in-depth attribution and quantitative assessment steps.

[0130] Furthermore, those skilled in the art will understand that the evaluation schemes for the specific evaluation dimensions (such as training sample dimension, inference step dimension, neuron dimension, security constraint mechanism dimension, violation risk evolution dimension at different training stages, and neuron violation risk dimension at different training stages) and specific indicator dimensions (such as output quality, version stability, adversarial robustness, training process health, and neuron risk) described in the above embodiments can be selectively combined and implemented according to actual application scenarios and needs. Meanwhile, although the above embodiments provide multiple data query interface numbers and names, these names and numbers are only for ease of description and understanding and do not constitute a limitation on the interface implementation method.

[0131] In another embodiment, a specific implementation method for evaluating the performance of the repaired artificial intelligence model using the output quality metric is disclosed.

[0132] In this embodiment, when the multiple indicator dimensions used for performance evaluation of the artificial intelligence model include the output quality indicator dimension, the process by which the evaluation system 200 obtains indicator data corresponding to the indicator dimension from the monitoring system 100 and uses the obtained indicator data for performance evaluation may specifically include the following processing steps: First, the evaluation system 200 obtains the output quality evaluation data of the artificial intelligence model before and after the repair from the monitoring system 100 through a first data query interface set in the monitoring system 100. This first data query interface can be, for example, the `eval / metrics` endpoint, which obtains data by querying a structured evaluation database persistently stored by version number and timestamp in the model evaluation platform. The output quality evaluation data includes at least one of the following: harmful output rate, policy violation rate, robustness score, and audit coverage, and necessarily includes the harmful output rate.

[0133] Each of the four indicators has a clear evaluation source and meaning. Harmful Output Rate is the proportion of outputs judged as violations out of the total test samples after running inference on the model version on the harmful content benchmark test set. Policy Violation Rate is the proportion of outputs that violate predefined policy rules after running inference on the policy compliance test set. Robustness Score is the proportion of outputs that remain safe after perturbation when running inference on the adversarial perturbation test set. Audit Coverage Rate is the proportion of events with a complete chain of evidence out of the total number of risk events from the set of historically known risk events. This complete chain of evidence refers to valid evidence items formed by evaluating the reasons for violations in the corresponding evaluation dimensions using data obtained through the fourth to ninth data query interfaces in the above embodiments.

[0134] After acquiring two sets of output quality evaluation data—one before and one after the repair—the evaluation system 200 determines the improvement in output quality of the repaired AI model based on these two sets of data. Specifically, the evaluation system 200 can automatically calculate the absolute change of each indicator, i.e., the value after repair minus the value before repair, as well as the percentage improvement rate. For indicators where lower values ​​are better, the improvement rate can be calculated as the difference between the value before and after repair divided by the value before repair, then multiplied by 100% (i.e., (baseline - after) / baseline × 100%). For indicators where higher values ​​are better, the improvement rate can be calculated as the difference between the value after and before repair divided by the value before repair, then multiplied by 100% ((after - baseline) / baseline × 100%).

[0135] Through the above methods, this embodiment provides an objective, unified, and quantifiable means to evaluate the specific effects of governance measures on output quality, thereby enabling precise measurement of the role of model repair in reducing harmful output.

[0136] In another embodiment, a specific implementation method for evaluating the performance of a repaired artificial intelligence model using version stability as an indicator is disclosed.

[0137] In this embodiment, when the multiple indicator dimensions used for performance evaluation of the artificial intelligence model include a version stability indicator dimension, the process by which the evaluation system 200 obtains indicator data corresponding to that indicator dimension from the monitoring system 100 and uses the obtained indicator data for performance evaluation can specifically include the following processing steps: First, the evaluation system 200 obtains performance evaluation data for various versions of the artificial intelligence model from the monitoring system 100 through a second data query interface set in the monitoring system 100. This second data query interface can be, for example, the `eval / version-regression` endpoint, which retrieves the performance evaluation data for each version of the artificial intelligence model by querying all registered historical version records in the model version registry. The performance evaluation data for each version can be obtained from different data sources. The model version number can be obtained from the version metadata field of the model version registry. The performance evaluation data includes at least one of harmful output rate, inference latency, usability score, and security regression risk, and necessarily includes security regression risk.

[0138] Specifically, the harmful output rate is the proportion of outputs judged as violating rules out of the total test samples after running inference on the model version on the harmful content benchmark set. Inference latency can be obtained from the inference service performance monitoring logs, taking the average P50 inference latency over a period after the version's release. Usability score can be obtained from the usability evaluation task, completed uniformly by the evaluation platform before release. Security regression risk is automatically calculated by the evaluation platform during version registration: if the current version is inferior to the best value among all previous versions in any security metric, the security regression risk equals the normalized result of the degradation magnitude; otherwise, it is zero.

[0139] After acquiring performance evaluation data from various versions, the evaluation system 200 uses this data to determine the security regression risk of the repaired AI model. Specifically, the evaluation system 200 automatically detects whether security regression occurs during version iterations by longitudinally comparing data from multiple versions, that is, whether the new version is inferior to the previous version in terms of security metrics such as the rate of harmful outputs, and determines whether the repair has introduced new security vulnerabilities.

[0140] Through the above methods, this embodiment provides a technical means to automatically detect whether the new version of the model has security degradation after repair, which can effectively prevent problems that are not detected due to the introduction of new risks by repair.

[0141] In another embodiment, a specific implementation method for evaluating the performance of the repaired artificial intelligence model using the metric dimension of adversarial robustness is disclosed.

[0142] In this embodiment, when the multiple indicator dimensions used for performance evaluation of the artificial intelligence model include an adversarial robustness indicator dimension, the process by which the evaluation system 200 obtains indicator data corresponding to that indicator dimension from the monitoring system 100 and uses the obtained indicator data for performance evaluation can specifically include the following processing steps: First, the evaluation system 200 obtains the adversarial test results of the repaired artificial intelligence model in multiple adversarial scenarios from the monitoring system 100 through a third data query interface set in the monitoring system 100. This third data query interface can be, for example, the `eval / stress-test` endpoint, which retrieves the adversarial test results of the artificial intelligence model in multiple adversarial scenarios by querying a pre-run and stored adversarial test result library in the adversarial evaluation platform. These multiple adversarial scenarios include jailbreak hints, multi-round inducement, tool injection, adversarial rewriting, and multiple extreme scenarios. The adversarial test results include at least one of pass rate and boundary stability score, and necessarily include the pass rate.

[0143] Each of the five adversarial scenarios described above corresponds to a predefined, standardized adversarial test suite. The jailbreak hint scenario includes publicly available and internally collected jailbreak hint vocabulary, covering known jailbreak patterns such as role-playing bypass, hypothetical scenario inducement, and multilingual mixing. The multi-turn inducement scenario includes pre-designed multi-turn dialogue attack scripts that gradually build false contexts and accumulate trust to induce the model to violate its policy. The tool injection scenario includes test cases that inject malicious commands into tool call parameters, testing the robustness of the model's tool permission boundaries. The adversarial rewriting scenario includes test cases that are semantically equivalent but have undergone adversarial rewriting processing such as synonym substitution, sentence transformation, and insertion of distractor words. The extreme scenario includes extreme input conditions such as empty input, excessively long input, special characters, and cross-language mixing, testing the stability of the model's security boundaries under non-standard input conditions.

[0144] After obtaining adversarial test results in multiple adversarial scenarios, the evaluation system 200 determines the average pass rate of the repaired artificial intelligence model in multiple adversarial scenarios based on these results. Specifically, the evaluation system 200 can calculate the average pass rate in each scenario and use this average value to measure the overall adversarial robustness of the model.

[0145] Through the above methods, this embodiment provides a means to quantitatively evaluate the security and stability of the model under various malicious attacks, making up for the deficiency that a single indicator such as output quality cannot reflect the robustness of the model against attacks.

[0146] In another embodiment, a specific implementation method for evaluating the performance of the repaired artificial intelligence model using the indicator dimension of training process health is disclosed.

[0147] In this embodiment, when the multiple indicator dimensions used for performance evaluation of the artificial intelligence model include the indicator dimension of training process health, the process by which the evaluation system 200 obtains indicator data corresponding to this indicator dimension from the monitoring system 100 and uses the obtained indicator data to perform performance evaluation may specifically include the following processing steps: First, the evaluation system 200 obtains the stage performance evaluation results of the repaired artificial intelligence model at each training stage from the monitoring system 100 through the fourth data query interface set in the monitoring system 100. This fourth data query interface can be, for example, the `trace / train-stage-risk` endpoint, which obtains the performance evaluation results of the artificial intelligence model at each training stage by querying a database of periodically saved stage evaluation snapshots during model training. The stage performance evaluation results include at least one of harmful output rate, neuronal instability, and path drift score, and necessarily include the harmful output rate.

[0148] The training phase can include the initial stage of supervised fine-tuning, the middle stage of supervised fine-tuning, the initial stage of reinforcement learning based on human feedback, and the convergence stage of reinforcement learning based on human feedback. After each landmark training phase is completed and the phase checkpoint is saved, the system automatically triggers a standardized compliance assessment task. The assessment results are stored in the assessment snapshot database with the phase label as the key. The harmful output rate is the proportion of the statistical model output that is judged as a violation after running a harmful content benchmark set at the phase checkpoint. Neuron instability is calculated by running multiple assessment batches of the top N high-risk neurons (high-risk neurons can be identified by evaluating the reasons for violations in the AI ​​model according to the neuron dimension in the above embodiments, identifying neurons with high risk attribution scores from the AI ​​model) at the phase checkpoint, calculating the standard deviation of the baseline activation mean between each batch, and then taking the mean of all high-risk neurons. The larger the mean, the more unstable the neuron activation at that phase. The path drift score is the value obtained by calculating the cosine distance between the edge weight vectors of the model's safety constraint loop in this stage and the corresponding edge weight vectors in the previous stage. The larger the score, the greater the structural change in the safety constraint loop of the artificial intelligence model during this training stage.

[0149] After obtaining the stage performance evaluation results for each training stage, the evaluation system 200 determines the maximum value of the harmful output rate of the repaired artificial intelligence model at each training stage based on these results. Specifically, the evaluation system 200 can longitudinally scan the harmful output rates of all training stages to find the peak values, thereby assessing whether the model had a serious risk exposure at a certain historical stage during the repair or retraining process.

[0150] Through the above methods, this embodiment provides a technical means to audit and trace the dynamic changes in violation risks of a model throughout its complete training lifecycle, which can effectively capture and evaluate the overall health of the model during the repair or retraining process.

[0151] In another embodiment, a specific implementation method for evaluating the performance of the repaired artificial intelligence model using the neuron risk index is disclosed.

[0152] In this embodiment, when the multiple indicator dimensions used for performance evaluation of the artificial intelligence model include the indicator dimension of neuron risk, the process by which the evaluation system 200 obtains indicator data corresponding to this indicator dimension from the monitoring system 100 and uses the obtained indicator data to perform performance evaluation may specifically include the following processing steps: First, the evaluation system 200 obtains the performance evaluation results for each neuron of the repaired artificial intelligence model from the monitoring system 100 through the fifth data query interface set in the monitoring system 100. This fifth data query interface can be, for example, a trace / neuron-analysis endpoint. It acquires data by performing batch causal attribution analysis on neurons in a specified layer of the model using a combination of activation statistics, activation patching, and cross-training phase consistency calculation. The performance evaluation result for each neuron includes at least one of the following: baseline activation mean, risk attribution score, and stability at different training phases, and necessarily includes the risk attribution score. Furthermore, the neuron performance evaluation results obtained by the fifth data query interface may also include neuron attribute information, such as the neuron's unique identifier, layer, and model type.

[0153] Specifically, the baseline activation mean is obtained by averaging the activation values ​​of each target neuron across multiple normal samples after performing forward propagation on a batch of normal inputs. This represents the baseline activation level of the neuron under normal operating conditions. The risk attribution score is obtained by performing an activation patching experiment on the neuron: the neuron's activation value is patched to its baseline activation mean, effectively masking the neuron's specific response to the current illegal input. The decrease in the probability of illegal output is then observed; a larger decrease indicates a higher causal contribution of the neuron to the illegal output. After normalization, a risk attribution score ranging from 0 to 1 is obtained. Stability across different training stages can be obtained by extracting the risk attribution scores of the same neuron from checkpoints saved during model training, calculating the inverse of the normalized variance of these cross-stage values, also ranging from 0 to 1. This index characterizes the stability of the neuron's risk contribution across different training stages.

[0154] After obtaining the performance evaluation results for each neuron, the evaluation system 200 uses these results to determine the neuron with the highest risk attribution score and its corresponding risk attribution score in the repaired artificial intelligence model. Specifically, the evaluation system 200 can sort all neurons in descending order of risk attribution score, and extract the neuron with the highest risk attribution score to determine whether there are still abnormal neurons in the repaired model that have an extremely high causal contribution to the illegal output.

[0155] Through the above methods, this embodiment provides a risk quantification and localization method that delves into the smallest computational unit of the model, which can accurately identify whether high-risk neurons still exist in the repaired model, thereby realizing the assessment and management of safety risks at the micro level inside the model.

[0156] The following example illustrates the specific processing procedure of the violation assessment method of the artificial intelligence model provided in the above embodiments of this application in a practical application.

[0157] Suppose a large commercial bank has deployed a customized big language model in its intelligent customer service system to provide customers with financial regulatory advice, wealth management product descriptions, and business handling guidance. Due to the special nature of financial services, regulatory agencies require that all content output by the big language model must strictly adhere to compliance requirements, and must not contain any violations such as making excessive promises of returns, inducing investment, or providing unauthorized legal advice. However, during operation, the content security monitoring platform detected a high-risk event: in a dialogue involving inheritance consultation, the model provided a response containing unauthorized legal conclusions, advising the user to abandon the pursuit of a disputed debt without prior notice.

[0158] The upstream risk detection system registered the violation and assigned it a globally unique identifier, "RE-20250321-011". Upon receiving the alert, the bank's compliance audit team immediately activated the model-based violation assessment system deployed on the internal audit platform, using the event identifier as the starting point input for the audit task, and executed a complete end-to-end violation tracking and governance assessment process.

[0159] The assessment system first initiates a correlation query through the ninth data query interface set up in the monitoring system, using "RE-20250321-011" as the primary key, to the work order management system, the session log database, and the log anomaly scoring service. Through this query, the system can aggregate and form a risk event instance uniquely identified by "RE-20250321-011," which includes event information such as the work order identifier "Ticket-FB-998," the session identifier "S-12345," the session anomaly score of 0.93, and the trigger type "intelligent customer service outputting unauthorized legal conclusions." This merged event information provides a clear context and query correlation link for all subsequent in-depth analysis.

[0160] Based on the information from this event, the evaluation system simultaneously initiated parallel data acquisition and violation cause assessment across six dimensions: training sample dimension, inference step dimension, neuron dimension, security constraint mechanism dimension, violation risk evolution dimension at different training stages, and violation risk dimension of neurons at different training stages.

[0161] During the multi-dimensional investigation, root cause tracing at the data layer yielded the first key discovery. The evaluation system, through the sixth data query interface, invoked the model feature extraction interface to obtain the vector representation of the violation output in the intermediate layer and calculated its gradient relative to the security loss function. Subsequently, the system used the influence function method to retrieve the most contributing sample from the training data index, discovering a training data point from the batch "WebCrawl-2023Q4" with the sample identifier "D-TA001." The parameter update direction formed during its training was highly consistent with the gradient direction of the current violation output. This sample's contamination score was calculated to be 0.91, and its content summary indicated it contained misleading legal statements. This dimension of the evaluation precisely pinpointed that the introduction of this training sample injected a tendency to generate unauthorized legal opinions into the model from the source.

[0162] At the model inference chain level, the evaluation system, through the seventh data query interface, replayed the model's complete inference process during the violation in an observable mode, capturing the hidden state vector and attention weight matrix of each inference step. The policy probe classifier's output for each step showed that the policy compliance score at step 15 plummeted from 0.82 (above the threshold) to 0.31, indicating a sharp decay of security constraints at that point. Further integral gradient attribution analysis of the attention weight matrix for this step revealed that the semantic unit "XX constraint clause" in the input contributed most to the constraint decay and was identified as the key attention token inducing the violation. Combining the decrease in compliance with the attribution value of this token, the system calculated an attribution score of 0.95 for this inference step, accurately revealing how the security defenses at the model's thought chain level were breached under a specific semantic inducement.

[0163] The investigation into the causes of violations did not stop at the inference chain. The evaluation system, through the fifth data query interface, performed causal analysis on each of the tens of thousands of neurons within the model. Through activation patching experiments, the system observed and recorded the decrease in the probability of the model's violation output after replacing the activation value of each neuron with the baseline mean. The results showed that the neuron located in the 24th layer attention module, numbered "N-1832," made the most significant causal contribution to the violation output, with a risk attribution score as high as 0.95. Furthermore, this high-risk characteristic exhibited stability of 0.83 across different training stages. This pinpointed the microscopic root cause of the problem down to a single computational unit.

[0164] In the loop view of the security constraint mechanism, the evaluation system obtained information on the security constraint functional components within the model through the eighth data query interface. Combined with identified high-risk neurons, it constructed a directed security constraint path consisting of nodes such as input prompts, policy token clustering, role boundary neurons, tool permission loops, security headers, and risk outputs. Through path repair experiments, the system estimated the information flow weight of each connection edge, ultimately finding that the weight of the connection edge from the security header node to the risk output node reached 0.92, the highest among all edges. This finding indicates that the final constraint that the security header should have executed failed when propagated to the output, and this connection point is the concentrated weak link in the entire security loop where risk propagation and amplification occur.

[0165] To understand the evolution of risk throughout the training lifecycle, the evaluation system simultaneously obtained performance data on harmful output rate, neuronal instability, and path drift scores for the model across four stages: SFT-early, SFT-mid, RLHF-early, and RLHF-convergence, through a fourth data query interface. The comparison revealed that the overall harmful output rate reached a peak of 0.214 in the SFT-early stage, significantly higher than in other stages. Based on this, the system performed a cross-risk transfer analysis between neurons and the training stage, calculating and constructing a risk transfer heatmap using a three-factor product model of attribute_to_risk_i × stage_consistency_i × normalized_harmful_rate_j. The heatmap clearly showed that the high-risk neuron "N-1832" had the highest violation risk score at the spatiotemporal intersection of "SFT-early," thus achieving root cause localization in the temporal dimension and identifying the most dangerous spatiotemporal combination.

[0166] Thus, through end-to-end tracing from external multi-source signals to the data layer, model inference chain layer, neuron layer, safety constraint loop layer, and finally to the training time sequence layer, the cause of this violation has been comprehensively revealed. The evaluation system integrates and outputs the above multi-dimensional findings, providing the development team with highly targeted remediation guidance.

[0167] Based on these findings, the bank's AI model development team implemented targeted external remedial measures: removing sample "D-TA001" and its associated problematic data from the training dataset, and fine-tuning the network layer containing neuron "N-1832" and the security head module. After one version iteration cycle, the remediated new model was submitted for performance verification.

[0168] The evaluation system then initiated a governance quantitative evaluation process, using the model version numbers before and after the fix as parameters, and called the first, second, third, fourth and fifth data query interfaces in parallel to obtain indicator data covering five dimensions: output quality, version stability, adversarial robustness, training process health, and neuron risk.

[0169] In terms of output quality, the evaluation data returned from the first data query interface shows that the model's harmful output rate decreased from a baseline of 0.173 before governance to 0.079 after governance, representing an absolute improvement of 54.3%. Regarding version stability, the longitudinal comparison results of the model version registry from the second data query interface confirm that the regression risk of the new version is 0, and no security degradation has occurred. In terms of adversarial robustness, the test results of five adversarial scenarios obtained from the third data query interface show that the model's average pass rate under complex attacks such as jailbreak hints, multi-round inducements, and tool injection is 0.88, a significant improvement over previous attacks. In terms of training process health and neuron risk, the data returned from the fourth and fifth data query interfaces also confirm that the peak harmful output rate at each stage of retraining has been reduced to 0.12, and no neuron's risk attribution score exceeds the threshold of 0.9.

[0170] Based on the evaluation results of the above five dimensions, the evaluation system executes a combined gating decision consisting of five independent rules. Because the five indicators—post-treatment harmful output rate, maximum regression risk, average pass rate of stress test, peak harmful output rate across stages, and attribution score of the highest risk neuron—all meet their respective preset performance requirements, all five rules are judged to be passed, the gating status is explicitly set to PASS, and the gating score is fixed at 1.0.

[0171] Ultimately, without any manual intervention, the assessment system automatically fills in the pre-set report template with the event summary, monitoring data and verification results across all dimensions, risk attribution conclusions, and multi-indicator quantitative assessment results. A structured model audit report is generated instantly, containing six complete audit evidences uniquely identified as "EV-RE-20250321-011-001" to "EV-RE-20250321-011-006," each clearly marked with its source path, key data extract, and credibility rating. The report, along with the evidence package and gating decision records, is exported as a machine-readable file and submitted to the compliance audit department for archiving.

[0172] In this financial customer service scenario, the technical solution provided in this application significantly improves the transparency and efficiency of risk management and remediation. A violation that would have previously required a team of security experts to spend days manually comparing logs and making inferences can now be fully rooted within minutes through automated, multi-dimensional causal tracing, tracing the entire process from the initial phenomenon to specific contamination samples, key decay steps, high-risk neurons, weak loop nodes, and even the risk evolution stage. The effectiveness of remediation measures no longer relies on subjective judgment but is objectively compared using a unified quantitative baseline across five orthogonal dimensions: output quality, robustness, training process, and internal neurons. Furthermore, the combined gating engine technically eliminates any possibility of releasing risk models solely for the sake of a single attractive metric. This provides solid technical support for financial institutions to achieve efficient governance and reliable iteration of AI models while meeting stringent regulatory audit requirements.

[0173] Another embodiment of this application also provides a violation assessment system for artificial intelligence models. For example... Figure 1 As shown, the system includes a monitoring system 100 and an evaluation system 200. The monitoring system 100 is used to monitor the operation and / or training process of the artificial intelligence model and store the monitoring data. The evaluation system 200 is used to perform the violation evaluation method of the artificial intelligence model described in any of the foregoing embodiments by interacting with the monitoring system 100.

[0174] The violation assessment system for the artificial intelligence model provided in this embodiment belongs to the same application concept as the violation assessment method for the artificial intelligence model provided in the above embodiments of this application. It can execute the method provided in any of the above embodiments of this application and has the corresponding functional modules and beneficial effects of the execution method. Technical details not described in detail in this embodiment can be found in the specific processing content of the violation assessment method for the artificial intelligence model provided in the above embodiments of this application, and will not be repeated here.

[0175] The functions implemented by the monitoring system 100 and the evaluation system 120 described above can be implemented by the same or different processors, and this application embodiment does not limit this.

[0176] It should be understood that the monitoring system 100 and the evaluation system 200 in the above system can be implemented in the form of a processor calling software. For example, the violation evaluation system includes a processor connected to a memory, which stores instructions. The processor calls the instructions stored in the memory to implement any of the above methods or the functions of the monitoring system and evaluation system constituting the violation evaluation system. The processor can be a general-purpose processor, such as a CPU or microprocessor, and the memory can be internal or external to the device.

[0177] Alternatively, the monitoring system 100 and evaluation system 200 in the above system can be implemented in the form of hardware circuits. The functions of some or all units can be achieved through the design of the hardware circuits, which can be understood as one or more processors. For example, in one implementation, the hardware circuit is an ASIC, and the functions of some or all of the above units are achieved through the design of the logical relationships between the components within the circuit. In another implementation, the hardware circuit can be implemented using a PLD, taking an FPGA as an example. It can include a large number of logic gates, and the connection relationships between the logic gates are configured through configuration files, thereby achieving the functions of some or all of the above units. The monitoring system 100 and evaluation system 200 in the above system can be implemented entirely through processor-called software, entirely through hardware circuits, or partially through processor-called software with the remaining parts implemented through hardware circuits.

[0178] In this application embodiment, a processor is a circuit with signal processing capabilities. In one implementation, the processor can be a circuit with instruction reading and execution capabilities, such as a CPU, microprocessor, GPU, or DSP. In another implementation, the processor can implement certain functions through the logical relationships of hardware circuits. These logical relationships are fixed or reconfigurable. For example, the processor may be a hardware circuit implemented as an ASIC or PLD, such as an FPGA. In a reconfigurable hardware circuit, the process of the processor loading a configuration document and configuring the hardware circuit can be understood as the processor loading instructions to implement the functions of some or all of the above units. Furthermore, it can also be a hardware circuit designed for artificial intelligence, which can be understood as an ASIC, such as an NPU, TPU, or DPU.

[0179] As can be seen, the monitoring system 100 and the evaluation system 200 in the above violation assessment system can be one or more processors (or processing circuits) configured to implement the above methods, such as: CPU, GPU, NPU, TPU, DPU, microprocessor, DSP, ASIC, FPGA, or a combination of at least two of these processor types.

[0180] Furthermore, the monitoring system 100 and the evaluation system 200 in the above violation assessment system can be integrated together or implemented independently. In one implementation, the monitoring system 100 and the evaluation system 200 are integrated together and implemented in the form of a System-on-Chip (SoC). This SoC may include at least one processor for implementing any of the above methods or implementing the functions of the monitoring system 100 and the evaluation system 200. The at least one processor can be of different types, such as CPU and FPGA, CPU and AI processor, CPU and GPU, etc.

[0181] Another embodiment of this application also provides an electronic device, see [link to relevant documentation] Figure 4 As shown, the device includes: Memory 200 and processor 210; The memory 200 is connected to the processor 210 and is used to store programs; The processor 210 is configured to implement the violation assessment method of the artificial intelligence model disclosed in any of the above embodiments by running the program stored in the memory 200.

[0182] Specifically, the aforementioned electronic device may also include: a bus, a communication interface 220, an input device 230, and an output device 240.

[0183] The processor 210, memory 200, communication interface 220, input device 230, and output device 240 are interconnected via a bus. Among them: A bus can include a pathway for transmitting information between various components of a computer system.

[0184] Processor 210 can be a general-purpose processor, such as a general-purpose central processing unit (CPU), a microprocessor, etc., or an application-specific integrated circuit (ASIC), or one or more integrated circuits used to control the execution of the program of the present invention. It can also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), an off-the-shelf programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components.

[0185] Processor 210 may include a main processor, as well as a baseband chip, modem, etc.

[0186] The memory 200 stores a program that executes the technical solution of this invention, and may also store an operating system and other key business functions. Specifically, the program may include program code, which includes computer operation instructions. More specifically, the memory 200 may include read-only memory (ROM), other types of static storage devices capable of storing static information and instructions, random access memory (RAM), other types of dynamic storage devices capable of storing information and instructions, disk storage, flash memory, etc.

[0187] Input device 230 may include a device for receiving user input data and information, such as a keyboard, mouse, camera, scanner, light pen, voice input device, touch screen, pedometer, or gravity sensor.

[0188] Output device 240 may include devices that allow information to be output to a user, such as a display screen, printer, speaker, etc.

[0189] The communication interface 220 may include a device that uses any transceiver to communicate with other devices or communication networks, such as Ethernet, Radio Access Network (RAN), Wireless Local Area Network (WLAN), etc.

[0190] The processor 210 executes the program stored in the memory 200 and calls other devices, which can be used to implement the various steps of the violation assessment method of any artificial intelligence model provided in the above embodiments of this application.

[0191] In addition to the methods and devices described above, embodiments of this application may also be computer program products, which include computer program instructions that, when executed by a processor, cause the processor to perform the steps in the violation assessment method for the artificial intelligence model described in any of the above embodiments of this specification.

[0192] The computer program product can be written in any combination of one or more programming languages ​​to perform the operations of the embodiments of this application. The programming languages ​​include object-oriented programming languages ​​such as Java and C++, as well as conventional procedural programming languages ​​such as C or similar languages. The program code can be executed entirely on the user's computing device, partially on the user's computing device, as a standalone software package, partially on the user's computing device and partially on a remote computing device, or entirely on a remote computing device or server.

[0193] Furthermore, embodiments of this application may also be storage media storing a computer program, which, when run by a processor, causes the processor to execute the steps in the violation assessment method of the artificial intelligence model described in any of the above embodiments of this specification.

[0194] For the foregoing method embodiments, in order to simplify the description, they are all described as a series of actions. However, those skilled in the art should understand that this application is not limited to the described order of actions, because according to this application, some steps can be performed in other orders or simultaneously. Furthermore, those skilled in the art should also understand that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily essential to this application.

[0195] It should be noted that the various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For apparatus embodiments, since they are basically similar to method embodiments, the description is relatively simple; relevant parts can be referred to the descriptions in the method embodiments.

[0196] The steps in the methods of the various embodiments of this application can be adjusted, merged, or deleted in order according to actual needs, and the technical features described in each embodiment can be replaced or combined.

[0197] The modules and sub-modules in the various embodiments of the present application's devices and terminals can be merged, divided, and deleted according to actual needs.

[0198] It should be understood that the disclosed terminals, devices, and methods can be implemented in other ways, given the several embodiments provided in this application. For example, the terminal embodiments described above are merely illustrative. For instance, the division of modules or sub-modules is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple sub-modules or modules may be combined or integrated into another module, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be indirect coupling or communication connection through some interfaces, devices, or modules, and may be electrical, mechanical, or other forms.

[0199] The modules or submodules described as separate components may or may not be physically separate. The components that constitute a module or submodule may or may not be physical modules or submodules; that is, they may be located in one place or distributed across multiple network modules or submodules. Some or all of the modules or submodules can be selected to achieve the purpose of this embodiment's solution, depending on actual needs.

[0200] Furthermore, the functional modules or sub-modules in the various embodiments of this application can be integrated into one processing module, or each module or sub-module can exist physically separately, or two or more modules or sub-modules can be integrated into one module. The integrated modules or sub-modules described above can be implemented in hardware or in the form of software functional modules or sub-modules.

[0201] Those skilled in the art will further recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0202] The steps of the methods or algorithms described in conjunction with the embodiments disclosed herein can be implemented directly by hardware, a software unit executed by a processor, or a combination of both. The software unit can be located in random access memory (RAM), main memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, hard disk, removable disk, CD-ROM, or any other form of storage medium known in the art.

[0203] Finally, it should be noted that in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0204] The above description of the disclosed embodiments enables those skilled in the art to make or use this application. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of this application. Therefore, this application is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A method for evaluating violations in an artificial intelligence model, characterized in that, include: Identifying violations based on artificial intelligence models, and querying event information related to the violations from the monitoring system of the artificial intelligence models; The monitoring system is used to monitor the reasoning and training processes of the artificial intelligence model; Based on the event information, monitoring data corresponding to multiple evaluation dimensions are obtained from the monitoring system, and the obtained monitoring data is used to evaluate the reasons for violations of the artificial intelligence model for the corresponding evaluation dimensions, thereby determining the evaluation results of the reasons for violations corresponding to each of the multiple evaluation dimensions. The multiple evaluation dimensions include multiple dimensions such as training sample dimension, inference step dimension, neuron dimension, security constraint mechanism dimension, violation risk evolution dimension at different training stages, and violation risk dimension of neurons at different training stages.

2. The method according to claim 1, characterized in that, The method further includes: Based on the evaluation results of the reasons for violations corresponding to each of the multiple evaluation dimensions, the artificial intelligence model is repaired; The system acquires indicator data corresponding to multiple indicator dimensions from the monitoring system, and uses the acquired indicator data to perform performance evaluation on the repaired artificial intelligence model for the corresponding indicator dimensions, thereby obtaining the performance evaluation results corresponding to each of the multiple indicator dimensions. The multiple metrics include several of the following: output quality, version stability, adversarial robustness, training process health, and neuron risk.

3. The method according to claim 2, characterized in that, The multiple metrics include output quality, version stability, adversarial robustness, training process health, and neuron risk. The method further includes: If the performance evaluation results for each of the aforementioned indicator dimensions meet the performance requirements corresponding to that indicator dimension, then the artificial intelligence model is determined to have been successfully repaired.

4. The method according to claim 2, characterized in that, When the multiple indicator dimensions include an output quality indicator dimension, indicator data corresponding to the output quality indicator dimension is obtained from the monitoring system, and the obtained indicator data is used to perform a performance evaluation of the repaired artificial intelligence model on the output quality indicator dimension to obtain the performance evaluation result corresponding to the output quality indicator dimension, including: The first data query interface set in the monitoring system is used to obtain the output quality evaluation data of the artificial intelligence model before and after the repair from the monitoring system; wherein, the output quality evaluation data includes at least one of harmful output rate, policy violation rate, robustness score and audit coverage, and necessarily includes the harmful output rate; Based on the output quality evaluation data of the artificial intelligence model before and after the repair, the improvement effect of the output quality of the repaired artificial intelligence model is determined.

5. The method according to claim 2, characterized in that, When the multiple indicator dimensions include a version stability indicator dimension, indicator data corresponding to the version stability indicator dimension is obtained from the monitoring system, and the obtained indicator data is used to perform a performance evaluation of the repaired artificial intelligence model on the version stability indicator dimension to obtain the performance evaluation result corresponding to the version stability indicator dimension, including: The performance evaluation data of each version of the artificial intelligence model is obtained from the monitoring system through the second data query interface set in the monitoring system; wherein, the performance evaluation data includes at least one of harmful output rate, inference latency, usability score and security regression risk, and necessarily includes the security regression risk; Based on the performance evaluation data of each version of the AI ​​model, the security regression risk of the repaired AI model is determined.

6. The method according to claim 2, characterized in that, When the multiple indicator dimensions include an adversarial robustness indicator dimension, indicator data corresponding to the adversarial robustness indicator dimension is obtained from the monitoring system, and the obtained indicator data is used to perform a performance evaluation of the repaired artificial intelligence model on the adversarial robustness indicator dimension to obtain the performance evaluation result corresponding to the adversarial robustness indicator dimension, including: The third data query interface set in the monitoring system is used to obtain the adversarial test results of the repaired artificial intelligence model in multiple adversarial scenarios from the monitoring system; wherein, the multiple adversarial scenarios include multiple of jailbreak prompts, multi-round inducement, tool injection, adversarial rewriting, and extreme scenarios, and the adversarial test results include at least one of pass rate and boundary stability score, and necessarily include the pass rate; Based on the adversarial test results of the repaired AI model in multiple adversarial scenarios, the average pass rate of the repaired AI model in multiple adversarial scenarios is determined.

7. The method according to claim 2, characterized in that, When the multiple indicator dimensions include training process health, indicator data corresponding to the training process health is obtained from the monitoring system, and the obtained indicator data is used to perform performance evaluation on the repaired artificial intelligence model based on the training process health indicator dimensions, to obtain the performance evaluation result corresponding to the training process health indicator dimensions, including: The fourth data query interface set in the monitoring system is used to obtain the stage performance evaluation results of the repaired artificial intelligence model in each training stage from the monitoring system; wherein, the stage performance evaluation results include at least one of harmful output rate, neuronal instability and path drift score, and necessarily include the harmful output rate; Based on the stage performance evaluation results of the repaired artificial intelligence model in each training stage, the maximum value of the harmful output rate of the repaired artificial intelligence model in each training stage is determined.

8. The method according to claim 2, characterized in that, When the multiple indicator dimensions include neuron risk, indicator data corresponding to the neuron risk is obtained from the monitoring system, and the obtained indicator data is used to perform performance evaluation on the repaired artificial intelligence model for the neuron risk indicator dimension, to obtain the performance evaluation result corresponding to the neuron risk indicator dimension, including: Through the fifth data query interface set in the monitoring system, the performance evaluation results corresponding to each neuron of the repaired artificial intelligence model are obtained from the monitoring system. The performance evaluation results include at least one of the following: baseline activation mean, risk attribution score, and stability at different training stages, and necessarily include the risk attribution score. Based on the performance evaluation results of each neuron in the repaired artificial intelligence model, the neuron with the highest risk attribution score and its corresponding risk attribution score in the repaired artificial intelligence model are determined.

9. The method according to claim 2, characterized in that, After acquiring monitoring data corresponding to multiple evaluation dimensions from the monitoring system, and before using the acquired monitoring data to evaluate the reasons for violations in the corresponding evaluation dimensions of the artificial intelligence model, the method further includes: The acquired monitoring data is verified to determine whether it conforms to the corresponding data specifications. If it is determined that the monitoring data corresponding to any evaluation dimension does not conform to the corresponding data specifications, simulated monitoring data is generated based on the data query interface and data request parameters used to obtain the monitoring data, and the simulated monitoring data is used as the monitoring data corresponding to the evaluation dimension.

10. The method according to claim 9, characterized in that, The method further includes: Based on the event summary information of the violation, the monitoring data corresponding to each of the multiple evaluation dimensions, the verification results of the monitoring data corresponding to each of the multiple evaluation dimensions, the evaluation results of the violation reasons corresponding to each of the multiple evaluation dimensions, and the performance evaluation results corresponding to each of the multiple indicator dimensions, a model audit report is generated.

11. The method according to any one of claims 1 to 10, characterized in that, When the multiple evaluation dimensions include training sample dimensions, based on the event information, monitoring data corresponding to the training sample dimensions is obtained from the monitoring system, and the obtained monitoring data is used to evaluate the reasons for violations in the training sample dimensions of the artificial intelligence model, determining the evaluation result of the reasons for violations corresponding to the training sample dimensions, including: Based on the event information, the process data of the violation event is obtained from the monitoring system through the sixth data query interface set in the monitoring system. The process data includes the vector representation of the violation output in the intermediate layer of the artificial intelligence model and the gradient of the violation output relative to the security loss function or the violation category score. Retrieve the target training sample that contributes the most to the violation event from the training sample dataset. The parameter update direction formed when the target training sample participates in training is consistent with the gradient direction of the violation output relative to the security loss function or the violation category score. The data information of the target training sample is determined, including dataset batch identifier, sample-level identifier, and contamination score.

12. The method according to any one of claims 1 to 10, characterized in that, When the multiple evaluation dimensions include a reasoning step dimension, based on the event information, monitoring data corresponding to the reasoning step dimension is obtained from the monitoring system, and the obtained monitoring data is used to evaluate the reasons for violations in the reasoning step dimension of the artificial intelligence model, determining the evaluation result of the reasons for violations corresponding to the reasoning step dimension, including: Based on the event information, the reasoning elements of the artificial intelligence model in each reasoning step of the violation event are obtained through the seventh data query interface set in the monitoring system. The reasoning elements include the generated reasoning text, hidden state vector and attention weight matrix. Based on the reasoning elements of each reasoning step, the degree of compliance of each reasoning step with the security policy is determined. From the reasoning steps that have a compliance rate with the security policy that is lower than the set compliance threshold, the reasoning steps that have the highest compliance rate with the security policy are identified as the violation reasoning steps.

13. The method according to claim 12, characterized in that, After determining the illegal reasoning step, the method further includes: Based on the attention weight matrix of the violation reasoning step, gradient attribution analysis is performed on the input semantic units to determine the contribution of each input semantic unit to the violation reasoning step's compliance with the security policy being lower than the compliance threshold. The input semantic unit with the highest contribution is identified as the inducing violation semantic unit; The attribution score of the violation inference step leading to the violation event is obtained by weighting and averaging the decrease in the compliance with the security policy and the contribution of the semantic unit that induces the violation. The decrease in the compliance rate of the violation reasoning step with the security policy represents the decrease in the compliance rate of the violation reasoning step with the security policy relative to the compliance rate of the minimum compliant reasoning step with the security policy. The minimum compliant reasoning step is the reasoning step with the lowest compliance rate among the reasoning steps whose compliance rate with the security policy is not lower than the compliance threshold.

14. The method according to any one of claims 1 to 10, characterized in that, When the multiple evaluation dimensions include the neuron dimension, based on the event information, monitoring data corresponding to the neuron dimension is obtained from the monitoring system, and the obtained monitoring data is used to evaluate the violation cause of the artificial intelligence model at the neuron dimension, determining the violation cause evaluation result corresponding to the neuron dimension, including: Based on the event information, the performance evaluation results corresponding to each neuron of the artificial intelligence model are obtained from the monitoring system through the fifth data query interface set in the monitoring system. The performance evaluation results include at least one of the following: baseline activation mean, risk attribution score, and stability at different training stages, and necessarily include the risk attribution score. Based on the performance evaluation results of each neuron in the artificial intelligence model, the neuron with the highest risk attribution score and its corresponding risk attribution score in the artificial intelligence model are determined.

15. The method according to any one of claims 1 to 10, characterized in that, When the multiple evaluation dimensions include a security constraint mechanism dimension, based on the event information, monitoring data corresponding to the security constraint mechanism dimension is obtained from the monitoring system, and the obtained monitoring data is used to evaluate the violation causes of the artificial intelligence model in the security constraint mechanism dimension, determining the violation cause evaluation result corresponding to the security constraint mechanism dimension, including: Based on the event information, the security constraint component information inside the artificial intelligence model is obtained through the eighth data query interface set in the monitoring system; Based on the security constraint component information within the artificial intelligence model, the security constraint components within the artificial intelligence model are constructed into a security constraint path; the security constraint path includes path nodes, and adjacent path nodes are connected by directed edges. The weight of each connecting edge in the safety constraint path is determined through path repair experiments; the weight of the connecting edge represents the degree of influence of the information propagated by the connecting edge on the violation output of the artificial intelligence model. Based on the weight of each connecting edge in the security constraint path, the connecting edge with the largest weight value is determined as the centralized path point for propagating violation risk information in the security constraint path.

16. The method according to any one of claims 1 to 10, characterized in that, When the multiple evaluation dimensions include violation risk evolution dimensions at different training stages, based on the event information, monitoring data corresponding to the violation risk evolution dimensions at different training stages is obtained from the monitoring system, and the obtained monitoring data is used to evaluate the violation causes of the artificial intelligence model at different training stages, determining the violation cause evaluation results corresponding to the violation risk evolution dimensions at different training stages, including: Based on the event information, the stage performance evaluation results of the artificial intelligence model at each training stage are obtained from the monitoring system through the fourth data query interface set in the monitoring system; wherein, the stage performance evaluation results include at least one of harmful output rate, neuronal instability and path drift score; Based on the performance evaluation results of the artificial intelligence model at each training stage, the training stage with the highest risk of violation is determined.

17. The method according to any one of claims 1 to 10, characterized in that, When the multiple evaluation dimensions include the violation risk dimension of neurons at different training stages, based on the event information, monitoring data corresponding to the violation risk dimension of neurons at different training stages is obtained from the monitoring system, and the obtained monitoring data is used to evaluate the violation causes of neurons at different training stages in the artificial intelligence model, determining the violation cause evaluation results corresponding to the violation risk dimension of neurons at different training stages, including: Based on the event information, the stage performance evaluation results of the artificial intelligence model at each training stage are obtained from the monitoring system through the fourth data query interface set in the monitoring system; the stage performance evaluation results include the harmful output rate. Based on the event information, the performance evaluation results corresponding to each neuron of the artificial intelligence model are obtained from the monitoring system through the fifth data query interface set in the monitoring system. Based on the performance evaluation results corresponding to each neuron of the artificial intelligence model, the neuron with the highest risk attribution score and its corresponding risk attribution score in the artificial intelligence model are determined. The performance evaluation results corresponding to the neuron include the risk attribution score and the stability at different training stages. For each training phase, the violation risk score of the neuron with the highest risk attribution score is calculated and determined based on the risk attribution score of the neuron with the highest risk attribution score, the stability of the neuron with the highest risk attribution score in that training phase, and the harmful output rate in that training phase.

18. The method according to any one of claims 1 to 10, characterized in that, The identification of the violation event based on the artificial intelligence model involves querying event information related to the violation event from the monitoring system of the artificial intelligence model, including: The identification of violations based on artificial intelligence models is achieved by obtaining event information related to the violations from the monitoring system through the ninth data query interface set in the monitoring system of the artificial intelligence model; the event information includes at least one of the following: work order identifier, session identifier, session anomaly score, and violation type.

19. A violation assessment system for an artificial intelligence model, characterized in that, include: A monitoring system is used to monitor the operation and / or training process of an artificial intelligence model and to store the monitoring data. An evaluation system is used to perform a violation evaluation method for an artificial intelligence model as described in any one of claims 1 to 18 by interacting with the monitoring system through data.

20. An electronic device, characterized in that, Including memory and processor; The memory is connected to the processor and is used to store programs; The processor is used to implement the violation assessment method of the artificial intelligence model as described in any one of claims 1 to 18 by running the program in the memory.

21. A computer program product, characterized in that, It includes computer program instructions that, when executed by a processor, cause the processor to perform a violation assessment method for an artificial intelligence model as described in any one of claims 1 to 18.

22. A storage medium, characterized in that, The storage medium stores a computer program, which, when executed by a processor, causes the processor to perform a violation assessment method for an artificial intelligence model as described in any one of claims 1 to 18.

Images