Method and system for testing the inferential ability of an artificial intelligence system

Abstract

The application provides a method and system for testing the inference ability of an artificial intelligence system. The method comprises: constructing a multi-dimensional space probability distribution model based on a structured configuration file of a plurality of dimensional parameter atomic items and at least one variation strategy rule defined for an inference ability scenario of an AI model deployed on an AI system; generating a dynamic test case set based on the multi-dimensional space probability distribution model; and executing a test on the AI model deployed on the AI system using the dynamic test case set.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence (AI) system testing technology, and specifically to methods and systems for testing the reasoning ability of AI systems. Background Technology

[0002] With the rapid iteration of Large Language Model (LLM) technology, testing the reasoning ability of AI systems has become a crucial step in ensuring their quality. Traditional methods for testing the reasoning ability of AI systems primarily rely on manually written static test case sets. However, these traditional methods have many shortcomings, thus necessitating improved techniques for testing the reasoning ability of AI systems. Summary of the Invention

[0003] In one aspect, embodiments of this application provide a method for testing the reasoning ability of an artificial intelligence (AI) system, comprising: constructing a multidimensional spatial probability distribution model based on a structured configuration file containing multiple dimensions of parameter atomic items and at least one mutation strategy rule defined for the reasoning ability scenario of an AI model deployed on the AI ​​system; generating a dynamic test case set based on the multidimensional spatial probability distribution model; and performing tests on the AI ​​model deployed on the AI ​​system using the dynamic test case set.

[0004] In another aspect, embodiments of this application provide a system for testing the inference capabilities of an artificial intelligence (AI) system, comprising: a configuration module configured to construct a multidimensional spatial probability distribution model based on a structured configuration file containing multiple dimension parameter atomic items and at least one mutation policy rule defined for the inference capability scenario of an AI model deployed on the AI ​​system; a generation module configured to generate a dynamic test case set based on the multidimensional spatial probability distribution model; and a testing module configured to perform tests on the AI ​​model deployed on the AI ​​system using the dynamic test case set.

[0005] In another aspect, embodiments of this application provide a non-transitory computer-readable medium storing instructions that, when executed by one or more processors, cause the one or more processors to perform a method for testing the reasoning ability of an artificial intelligence (AI) system according to embodiments of this application.

[0006] In another aspect, embodiments of this application provide a computer program product storing instructions that, when executed by one or more processors, cause the one or more processors to perform a method for testing the reasoning ability of an artificial intelligence (AI) system according to embodiments of this application.

[0007] The method and system for testing the inference capabilities of an AI system according to embodiments of this application significantly improve efficiency by automatically generating dynamic test case sets, replacing manual coding. Furthermore, through random mutation and boundary exploration mechanisms, it fully covers complex testing scenarios, significantly improving the defect detection rate. In addition, since only structured configuration files need to be maintained, eliminating the need to store massive amounts of static test cases, the amount of code that needs to be maintained is significantly reduced. Moreover, the method and system for testing the inference capabilities of an AI system according to embodiments of this application can also be perfectly integrated into continuous integration / continuous deployment (CI / CD) pipelines, supporting automatic triggering after AI model iteration, achieving test left shift, and ensuring the quality of AI model iteration. Attached Figure Description

[0008] When read in conjunction with the accompanying drawings, various aspects of this disclosure are best understood through the following detailed description. It should be understood that the drawings are for illustrative and descriptive purposes only and not for limiting purposes. In the drawings:

[0009] Figure 1 This is a schematic diagram of the test architecture according to an embodiment of this application.

[0010] Figure 2 This is a flowchart of a method for testing the reasoning ability of an artificial intelligence system according to an embodiment of this application.

[0011] Figure 3 This is an application diagram illustrating a method for testing the reasoning ability of an artificial intelligence system according to an embodiment of this application.

[0012] Figure 4 This is a block diagram of a system for testing the reasoning ability of an artificial intelligence system according to an embodiment of this application.

[0013] Figure 5 This is a schematic diagram of a computing device that can implement a system for testing the reasoning ability of an artificial intelligence system according to embodiments of this application. Detailed Implementation

[0014] The following disclosure provides numerous different embodiments or examples for implementing various features of the provided subject matter. Specific examples of components and arrangements are described below to simplify this disclosure. Of course, these are merely examples and not limiting.

[0015] As mentioned above, traditional methods for testing the reasoning ability of AI systems primarily rely on manually written static test case sets. However, traditional methods cannot exhaustively enumerate the infinite combinations of natural language and struggle to reach complex boundary conditions, resulting in the potential defects of the AI ​​model not being fully exposed. AI models are updated extremely rapidly, requiring manual modification of the static test case set with each iteration. Therefore, the test case set updates lag behind AI model iterations, and maintenance costs are high. Furthermore, the testing process using static test case sets is unidirectional, unable to automatically adjust the testing focus based on weaknesses exhibited by the AI ​​model in previous tests, thus lacking adaptability. Moreover, static test case sets often contain a large number of low-value, repetitive test cases, and executing such test cases consumes significant computational resources, resulting in resource waste.

[0016] In view of the above, this application proposes a method and system for testing the reasoning ability of artificial intelligence systems, which can generate dynamic test case sets.

[0017] Figure 1 This is a schematic diagram of a test architecture according to an embodiment of this application. Figure 1 As shown, the test architecture 100 according to an embodiment of this application includes a test management platform 101 and an AI system (also called an AI data center) 102 that communicates with the test management platform 101. The communication between the test management platform 101 and the AI ​​system 102 can be implemented using direct communication methods based on network protocols (e.g., through application programming interfaces (APIs) or container orchestration engines), cloud-native and service mesh-based communication methods (e.g., through the control plane of a service network or a cloud-native API gateway), communication methods based on automated operation and maintenance tools (e.g., through CI / CD pipelines), and dedicated protocol and hardware-level communication methods (e.g., through the Intelligent Platform Management Interface (IPMI)).

[0018] AI system 102 includes a test environment 1021. Test environment 1021 can also be referred to as a "hardware environment" because it is a combined hardware-software instance, such as consisting of servers, operating systems, and accelerator cards. AI models can run on various servers. It should be understood that AI system 102 may also include any other components known in the art (e.g., supporting storage, network infrastructure, etc.), the description of which is omitted to avoid unnecessarily obscuring this application. Testing of AI system 102 can be performed in an isolated portion of test environment 1021 (such as an isolated virtualization environment) to avoid interference with development and production environments. Furthermore, it should be understood that although the above description uses a homogeneous architecture AI system, the test architecture according to embodiments of this application is not limited to this, but can be similarly applied to heterogeneous architecture AI systems (i.e., AI systems including test environments 1021 with different configurations). In a heterogeneous architecture context, more than one test environment 1021 can be adapted to the test process according to embodiments of this application.

[0019] The Test Management Platform 101, acting as the "command and dispatch center" for testing, is the core of the entire testing system's control. It can perform various tests across different testing environments. The Platform is responsible for the automated orchestration and dynamic scheduling of test tasks, automatically adjusting the number of stress-applying nodes based on the AI ​​system's resource load to avoid wasting test resources. During test execution, the Platform collects real-time end-to-end data from the AI ​​system 102, including hardware metrics, system status, and business performance data. Through its built-in intelligent analysis engine, it performs bottleneck identification, root cause analysis, and performance evaluation, ultimately generating visualized test reports or providing error feedback to support the development, maintenance, and optimization of the AI ​​system.

[0020] The AI ​​System 102 serves as the "test environment carrier," completely replicating the production-grade configuration. It fully supports various hardware and software resources, simulating the load characteristics of real business scenarios. The AI ​​System 102 supports both single-test-environment testing and collaborative testing across multiple test environments.

[0021] The following will combine Figure 2-5 A system and method for testing the reasoning ability of an artificial intelligence system according to embodiments of this application are described in more detail.

[0022] Figure 2 This is a flowchart of a method for testing an artificial intelligence system according to an embodiment of this application. Figure 2 As shown, the method 200 for testing an artificial intelligence system according to an embodiment of this application includes steps S201-S203.

[0023] In step S201, for the inference capability scenario of the AI ​​model deployed on the AI ​​system, a multidimensional spatial probability distribution model is constructed based on a structured configuration file containing multiple dimension parameter atomic terms and at least one mutation strategy rule defined for the scenario.

[0024] In some implementations, the structured configuration file is a configuration file using YAML / JSON extended syntax.

[0025] In some implementations, at least one mutation strategy rule includes any of the following: semantic equivalence substitution, cue word injection interference, or parameter combination constraints.

[0026] In some implementations, constructing a multidimensional spatial probability distribution model includes: constructing a one-dimensional probability space for each dimension of the multiple-dimensional parameter atomic items, wherein the one-dimensional probability space includes the values ​​of the parameter atomic items of that dimension and their probability weights; adjusting the one-dimensional probability space of at least one dimension of the multiple-dimensional parameter atomic items based on a weight adjustment factor and / or a mutation policy description of at least one mutation policy rule; and combining the adjusted one-dimensional probability spaces of each dimension of the multiple-dimensional parameter atomic items to form a multidimensional spatial probability distribution model.

[0027] In some implementations, adjusting the one-dimensional probability space of at least one dimension of the parameter atoms in multiple dimensions includes: applying a weight adjustment factor of an associated mutation policy rule to the probability weight of a specific value in the one-dimensional probability space of at least one dimension; and / or attaching a mutation policy description of an associated mutation policy rule to the one-dimensional probability space of at least one dimension.

[0028] In step S202, a dynamic test case set is generated based on the multidimensional spatial probability distribution model.

[0029] In some implementations, generating a dynamic test case set based on a multidimensional spatial probability distribution model includes: performing weighted random sampling on the adjusted one-dimensional probability space of each dimension of the parameter atomic items in multiple dimensions to form combinations of parameter atomic item values; instantiating the formed combinations of parameter atomic item values ​​to generate basic test cases; and dynamically mutating the basic test cases according to the mutation strategy description associated with the instantiated combinations of parameter atomic item values ​​to generate a dynamic test case set.

[0030] In step S203, the AI ​​model deployed on the AI ​​system is tested using a dynamic test case set.

[0031] In some implementations, method 200 may further include: collecting indicator data for testing the AI ​​model; based on the collected indicator data, calculating the response anomaly rate of the AI ​​model caused by test cases under each mutation strategy rule; and adjusting the weight adjustment factor and / or mutation strategy description of the mutation strategy rule according to the response anomaly rate of the AI ​​model caused by test cases under each mutation strategy rule and the average value of each indicator data under the mutation strategy rule, so as to update the multidimensional spatial probability distribution model and regenerate a dynamic test case set based on the updated multidimensional spatial probability distribution model.

[0032] In some implementations, the metrics include: correctness, perplexity, token stability, and defect pattern matching rate.

[0033] In some implementations, adjusting the weight adjustment factor and / or mutation strategy description of the mutation strategy rule includes: if the test cases generated under the mutation strategy rule cause the AI ​​model's response anomaly rate to exceed a threshold, then increasing the weight adjustment factor of the mutation strategy rule and / or optimizing the mutation strategy description of the mutation strategy rule based on the average value of each indicator data under the mutation strategy rule; and if the test cases generated under the mutation strategy rule cause the AI ​​model's response anomaly rate to not exceed a threshold, then decreasing the weight adjustment factor of the mutation strategy rule and / or optimizing the mutation strategy description of the mutation strategy rule based on the average value of each indicator data under the mutation strategy rule.

[0034] In some implementations, method 200 is automatically triggered by the continuous integration / continuous deployment (CI / CD) pipeline after the AI ​​model iteration.

[0035] The following describes, using the "arithmetic logic reasoning ability" of an AI model deployed on an AI system as a scenario, and taking three-dimensional parameter atomic terms (large number multiplication, negative number interference, multi-turn dialogue) and three mutation strategy rules (semantic equivalence replacement, prompt word injection interference, parameter combination constraints) as examples, the method for testing the reasoning ability of an AI system according to the embodiments of this application.

[0036] exist Figure 3In the example, method 300 according to an embodiment of this application begins in the first stage S301, where triggering and configuration are completed. This method is automatically triggered by the CI / CD pipeline after an AI model iteration. Upon triggering, a structured configuration file is loaded, for example by a configuration parsing engine. Specifically, the structured configuration file is loaded with multi-dimensional parameter atoms (in the example, large number multiplication, negative number interference, and multi-turn dialogue) and at least one mutation strategy rule (in the example, semantic equivalence replacement, cue word injection interference, and parameter combination constraints) defined for the inference capability scenario (in the example, arithmetic logic reasoning) of the AI ​​model deployed on the AI ​​system. As an example, a configuration file using YAML extended syntax is shown below:

[0037] Yaml

[0038] #Multi-dimensional parameter atomic terms (3 dimensions) for reasoning ability scenarios

[0039] input_atoms: # Definition of parameter atoms

[0040] Large number multiplication: # Dimension 1: Large number multiplication

[0041] values: # Each value and its probability weight (basic data for the one-dimensional probability space)

[0042] 10 3 -10 5 (For large numbers): 0.5

[0043] 10 9 -10 12 (Extremely large number): 0.3

[0044] 10 0 -10 2 (Decimal): 0.2 # Boundary value, used for boundary exploration;

[0045] Negative interference: # Dimension 2: Negative interference

[0046] values:

[0047] No negative numbers (product of two positive numbers): 0.4

[0048] Single negative number (multiplication of one positive and one negative number): 0.3

[0049] Double negative number (multiplication of two negative numbers): 0.3 # Boundary combination related values

[0050] Multi-turn dialogue: # Dimension 3: Multi-turn dialogue;

[0051] values:

[0052] Round 1 without correlation: 0.2

[0053] 2-3 rounds of weak association: 0.5

[0054] Strong association in 4-5 rounds: 0.3 # Round boundary value.

[0055] #At least one mutation strategy rule (3 rules, corresponding to semantic equivalent replacement, prompt word injection interference, and parameter combination constraints)

[0056] mutation_strategies:

[0057] S1_Semantic Equivalent Substitution:

[0058] weight_adjust_factor: 1.0 (Related dimension 1) # Weight adjustment factor for this strategy

[0059] strategy_description: Description 1: "Replace 'calculation' with 'product' and 'calculation result' with 'final product' in arithmetic problems, while maintaining semantic equivalence" # Mutation strategy description;

[0060] S2_Cue Word Injection Disruption:

[0061] weight_adjust_factor: 1.0 - Correlation dimension 3 # Weight adjustment factor for this strategy

[0062] strategy_description: Description 2: "Inject distractor statements at the end of arithmetic problems: 'You made a mistake in the previous problem, recalculate this one' or 'Based on the result of the previous problem, don't make a mistake,' and randomly select one."

[0063] S3_Parameter Combination Constraint:

[0064] weight_adjust_factor: 1.5 - Related dimensions 1 and 2 # Weight adjustment factor for this strategy (prioritize testing the portfolio boundary)

[0065] strategy_description: Description 3: "Constrain large number multiplication to take '10'" 9 -10 12 (Extremely large numbers) and negative interference takes 'double negative numbers', forcibly combining boundary scenarios.

[0066] Next, the configuration parsing engine builds a multidimensional spatial probability distribution model based on the structured configuration file.

[0067] For example, continuing the example described above, a one-dimensional probability space is constructed for each of the three-dimensional parameter atoms, using parameters directly from the input_atoms in the structured configuration file. The constructed one-dimensional probability model is as follows:

[0068] One-dimensional probability space for large number multiplication: {10 3 -10 5 0.5, 10 9 -10 12 0.3, 10 0 -10 2 : 0.2};

[0069] One-dimensional probability space for negative number interference: {No negative numbers: 0.4, Single negative number: 0.3, Double negative number: 0.3};

[0070] One-dimensional probability space for multi-turn dialogues: {1 round no association: 0.2, 2-3 rounds weak association: 0.5, 4-5 rounds strong association: 0.3}.

[0071] After constructing a one-dimensional probability space with three dimensions, the one-dimensional probability space of the related dimensions is adjusted based on the weight adjustment factors and / or mutation policy descriptions of the three mutation policy rules. For example, continuing the above example, we get:

[0072] S1_Semantic Equivalence Replacement: Only related to dimension 1 of "Large Number Multiplication", it adds a mutation strategy description to the one-dimensional probability space of dimension 1 of "Large Number Multiplication" for subsequent test case mutation;

[0073] S2_Prompt Injection Disturbance: Only related to "Multi-turn Dialogue" dimension 3, it adds a mutation strategy description to the one-dimensional probability space of "Multi-turn Dialogue" dimension 3 for subsequent test case mutation;

[0074] S3_Parameter Combination Constraint: Related to the two dimensions of "large number multiplication" and "negative number interference", it adjusts the one-dimensional probability space of "large number multiplication" dimension 1 and "negative number interference" dimension 2—in "large number multiplication", "10" is adjusted. 9 -10 12 The probability weight of "(ultra-large number)" is 0.3×1.5=0.45, and the probability weight of "double negative number" in "negative number interference" is 0.3×1.5=0.45 to strengthen the sampling probability of boundary combination scenarios.

[0075] After adjusting the three-dimensional one-dimensional probability space, the three adjusted one-dimensional probability spaces are combined to form a three-dimensional spatial probability distribution model of "large number multiplication × negative interference × multi-turn dialogue". That is, this model includes:

[0076] One-dimensional probability space for large number multiplication: {10 3-10 5 0.5, 10 9 -10 12 0.45, 10 0 -10 2 :0.2}-Description 1, 3;

[0077] One-dimensional probability space for negative interference: {no negative numbers: 0.4, single negative numbers: 0.3, double negative numbers: 0.45} - Description 3;

[0078] One-dimensional probability space for multi-turn dialogue: {1 round no association: 0.2, 2-3 rounds weak association: 0.5, 4-5 rounds strong association: 0.3} - Description 2.

[0079] Next, the process flows to the second stage, S302, where test cases are generated. In this stage, the collaborative generation engine (i.e., the test case factory) first performs weighted random sampling on the adjusted one-dimensional probability space of each of the three dimensions based on a three-dimensional spatial probability distribution model to form combinations of parameter atomic values. It should be understood that the weighted random sampling in this embodiment ensures that parameter atomic values ​​with high probability weights (such as extremely large numbers, double negative numbers, and other boundary scenarios) are selected more frequently, while values ​​with low weights are selected less frequently, rather than uniform random sampling—ensuring basic coverage of common scenarios while focusing on boundary scenarios where defects are prevalent. Specifically, each one-dimensional probability space has a "value-weight" key-value pair; the weights are accumulated sequentially to form a "cumulative weight interval," where each value corresponds to a unique interval, the length of which is equal to its weight. The random number is determined to fall into which cumulative weight interval; the value corresponding to that interval is the sampling result.

[0080] Continuing with the example described above, let's say we sample to obtain a 3-combination:

[0081] Sampling combination 1: Large number multiplication = 10 10 (10) 9 -10 12 (Interval), negative interference = double negative numbers (-5.2 × 10) 10 -3.8×10 10 ), multi-turn dialogue = 4 rounds of strong association (previous round of dialogue: calculate 2×3=6);

[0082] Sampling combination 2: Large number multiplication = 5 × 10 4 (10) 3 -10 5 (interval), negative interference = no negative numbers (8×10) 4 6×10 4 ), multi-round dialogue = 2 rounds of weak connection (previous round dialogue: How is the weather today);

[0083] Sampling combination 3: Large number multiplication = 50 (10 0 -10 2 (Interval), negative interference = single negative number (50, -30), multiple rounds of dialogue = 1 round of no correlation.

[0084] Next, the collaborative generation engine calls the atomic function generation library to instantiate the three combinations obtained from the sampling to generate basic test cases.

[0085] Basic use case 1 (corresponding to sampling combination 1): "The previous calculation result was 6, please calculate -5.2 × 10⁻⁶". 10 With -3.8×10 10 The product of;

[0086] Basic use case 2 (corresponding to sampling combination 2): "What's the weather like today? Please calculate 5 × 10..." 4 With 6×10 4 The product of;

[0087] Basic use case 3 (corresponding to sample combination 3): "Please calculate the product of 50 and -30".

[0088] Next, the collaborative generation engine dynamically mutates the basic test cases according to the mutation strategy description associated with the sampling combination, resulting in a diverse sequence of prompt words, which is the final dynamic test case set, as shown in the following example:

[0089] Variant Use Case 1 (Basic Use Case 1 + S2 Interference + S3 Combined Constraint): "The previous calculation result was 6, please calculate -5.2 × 10⁻⁶". 10 With -3.8×10 10 The product of the two, I calculated it wrong in the previous question, let's recalculate this one (S2 interference injection, S3 combination constraints are already reflected in the sampling combination);

[0090] Variant Use Case 2 (Base Use Case 2 + S1 Semantic Replacement): "What's the weather like today? Request 5×10..." 4 With 6×10 4 The final product (S1 replaces "calculate" with "find" and "product" with "final product");

[0091] Variant use case 3 (basic use case 3 + no variant): "Please calculate the product of 50 and -30" (no variant strategy triggered, normal use case).

[0092] The above description is for illustrative purposes only, and the number of test cases in a test case set is not limited to the specific values ​​mentioned above. Assume that a final test case set of 1000 cases is generated, including approximately 300 test cases under mutation policy rule S1, approximately 250 test cases under mutation policy rule S2, approximately 350 test cases under mutation policy rule S3, and approximately 100 unmutated test cases.

[0093] Next, the methodology moves to the third phase, S303, which involves test execution and monitoring. In this phase, dynamic test case sets are used to test the AI ​​model deployed on the AI ​​system. For example, 1000 dynamic test cases are input one by one into the AI ​​model deployed on the AI ​​system. During testing, real-time data on test metrics for the AI ​​model are collected, such as correctness, perplexity, lexical stability, and defect pattern matching rate.

[0094] Correctness refers to the degree to which the AI ​​model's calculation result matches the standard answer. The core judgment is whether the calculation is accurate, which can be quantified using binary values: 1 = correct (the result is completely consistent with the standard answer); 0 = incorrect (the result is inconsistent with the standard answer / no result / incorrect result format).

[0095] Perplexity, or semantic uncertainty in the calculation results generated by an AI model, reflects the degree to which the AI ​​model understands the arithmetic problem. It can be a continuous value (≥0). The lower the value, the more accurate the AI ​​model's understanding; the higher the value, the more ambiguous the understanding.

[0096] Lexical stability refers to the output stability of lexical units (numbers, symbols, units) during the AI ​​model's computation process. It reflects the robustness of the AI ​​model's computation process and can be a continuous value (0-1). The closer the value is to 1, the more stable the lexical output is, and the closer it is to 0, the more likely the lexical unit is to be disordered / missing / replaced.

[0097] The defect pattern matching rate is the probability that the calculation error result of the AI ​​model matches the preset arithmetic logic defect pattern library (such as sign error, digit misalignment, large number truncation, etc.). It can be a percentage (0%-100%), and the higher the value, the greater the probability that the AI ​​model triggers typical defects.

[0098] Among the above indicators, "correctness" is the core hard indicator. If the correctness is 0 (calculation error) in the arithmetic logic test, the single test case is judged as abnormal regardless of whether other indicators are normal. The other three indicators are auxiliary indicators, used to analyze the cause of abnormality from the perspective of model understanding, output stability and defect type. They are also used as a supplement in special scenarios (such as extended arithmetic scenarios where there is no standard answer) to determine abnormalities.

[0099] Next, the feedback optimization controller, based on the collected metric data, calculates the anomaly rate of the AI ​​model's response caused by test cases under each mutation policy rule. The anomaly rate of test cases under a certain type of mutation policy rule is the percentage of the number of test cases judged as anomaly under that type of mutation policy rule relative to the total number of test cases under that type of mutation policy rule.

[0100] In some implementations, correctness can be the core criterion for determining whether a single test case is abnormal. That is, as long as the correctness is 0 (calculation error), the test case is directly determined to be an abnormal test case, regardless of whether the other three auxiliary indicators are normal.

[0101] In some implementations, a weighted anomaly scoring method can be used to determine whether a single test case is an anomaly, as follows:

[0102] Anomaly score = Correctness anomaly weight score + Perplexity anomaly weight score + Lexical stability anomaly weight score + Defect pattern matching rate anomaly weight score.

[0103] In this system, if a single indicator is abnormal, a weighted score for that indicator is assigned; otherwise, a score of 0 is assigned. For example, the scores are: Correctness (70 points), Perplexity (10 points), Lexical Stability (10 points), and Defect Pattern Matching Rate (10 points). For instance, a use case might have Correctness = 0 (abnormal, 70 points), Perplexity = 9 (abnormal, 10 points), Lexical Stability = 0.6 (abnormal, 10 points), and Defect Pattern Matching Rate = 40% (abnormal, 10 points) → Abnormal Score = 100 points → Determined as an abnormal use case.

[0104] Continuing with the example above, for instance, if correctness is the core criterion, the statistical results are as follows:

[0105]

[0106] Note that the average correctness is the percentage of test cases with a correctness of 1 relative to the total number of test cases under the corresponding mutation strategy rule. For example, a correctness of 35% under mutation strategy rule S3 means that only 35% of the 350 test cases are calculated correctly, while 65% are calculated incorrectly, consistent with a response anomaly rate of 65%. All four indicators under mutation strategy rule S3 are abnormal: correctness is far below the threshold, perplexity is far above the threshold, lexical stability is far below the threshold, and defect pattern matching rate is far above the threshold, indicating that the boundary combination of "ultra-large numbers + double negative numbers" is the core defect scenario of the model. The model with the no-mutation item has the highest correctness, the best auxiliary indicators, and the lowest response anomaly rate, indicating that the model's computational ability is stable under normal, interference-free arithmetic scenarios.

[0107] If the test cases under each mutation strategy rule cause the AI ​​model's response anomaly rate to fall below the threshold, i.e., the test process termination condition is met, a test analysis report is generated and the mutation strategy rule base is updated using the mutation strategy rule used.

[0108] Conversely, if the test cases under each mutation policy rule result in an AI model response anomaly rate that is not lower than a threshold, i.e., the test process termination condition is not met, the methodology identifies model weaknesses and the effectiveness of mutation policy rules in the fourth phase, S304. Next, based on the AI ​​model response anomaly rate caused by the test cases under each mutation policy rule and the average value of each indicator data under that mutation policy rule, the weight adjustment factor and / or mutation policy description of that mutation policy rule are adjusted to update the multidimensional spatial probability distribution model, and a dynamic test case set is regenerated based on the updated multidimensional spatial probability distribution model.

[0109] In some implementations, if the test cases generated under the mutation policy rule cause the AI ​​model's response anomaly rate to exceed a threshold, then the weight adjustment factor of the mutation policy rule is increased and / or the mutation policy description of the mutation policy rule is optimized based on the average value of each indicator data under the mutation policy rule; and if the test cases generated under the mutation policy rule cause the AI ​​model's response anomaly rate to not exceed a threshold, then the weight adjustment factor of the mutation policy rule is decreased and / or the mutation policy description of the mutation policy rule is optimized based on the average value of each indicator data under the mutation policy rule.

[0110] For example, as shown in the table above, the response anomaly rate under mutation policy rule S3 is higher than the threshold, and all four indicators are abnormal. This indicates that the AI ​​model not only makes calculation errors under this mutation policy rule, but also suffers from multiple problems such as unclear understanding of the problem, unstable output, and triggering typical defects. It is necessary to simultaneously increase the weight adjustment factor and optimize the mutation policy description. If the response anomaly rate under a certain type of mutation policy rule is not higher than the threshold, but some auxiliary indicators are abnormal, the weight adjustment factor of this type of mutation policy rule can be slightly increased in subsequent tests to perform targeted hidden defect mining, without the need for significant adjustments to the weight adjustment factor. After adjusting the weight adjustment factor and / or mutation policy description of the mutation policy rule, the multidimensional spatial probability distribution model is updated. Then, based on the updated multidimensional spatial probability distribution model, a dynamic test case set is regenerated to achieve closed-loop feedback and dynamic test case set generation.

[0111] The method for testing the inference capability of an AI system according to embodiments of this application significantly improves efficiency by automatically generating dynamic test case sets, replacing manual coding. Furthermore, through random mutation and boundary exploration mechanisms, it fully covers complex testing scenarios, significantly improving the defect detection rate. In addition, since only structured configuration files need to be maintained, eliminating the need to store massive amounts of static test cases, the amount of code that needs maintenance is significantly reduced. Moreover, the method for testing the inference capability of an AI system according to embodiments of this application can be perfectly integrated into continuous integration / continuous deployment (CI / CD) pipelines, supporting automatic triggering after AI model iteration, achieving test left shift, and ensuring the quality of AI model iteration.

[0112] Figure 4 This is a block diagram of a system for testing the reasoning ability of an artificial intelligence system according to an embodiment of this application. Figure 4 As shown, a system 400 for testing the reasoning ability of an artificial intelligence system according to an embodiment of this application includes:

[0113] Configuration module 401 is configured to construct a multidimensional spatial probability distribution model for the inference capability scenario of the AI ​​model deployed on the AI ​​system, based on a structured configuration file containing multiple dimension parameter atomic items and at least one mutation strategy rule defined for the scenario.

[0114] Generation module 402 is configured to generate a dynamic test case set based on a multidimensional spatial probability distribution model; and

[0115] Test module 403 is configured to perform tests on the AI ​​model deployed on the AI ​​system using a dynamic test case set.

[0116] It should be understood that Figure 4 Each module can be referred to in the preceding method embodiments, and will not be repeated here. The system for testing artificial intelligence systems according to embodiments of this application significantly improves efficiency by automatically generating dynamic test case sets, replacing manual coding. Furthermore, through random mutation and boundary exploration mechanisms, it fully covers complex testing scenarios, significantly improving defect detection rates. In addition, since only structured configuration files need to be maintained, there is no need to store massive amounts of static test cases, significantly reducing the amount of code that needs to be maintained. Moreover, the system for testing the inference capabilities of AI systems according to embodiments of this application can also be perfectly integrated into continuous integration / continuous deployment (CI / CD) pipelines, supporting automatic triggering after AI model iteration, achieving test left shift, and ensuring the quality of AI model iteration.

[0117] Figure 5 This is a system 400 that can perform testing of an artificial intelligence system according to embodiments of this application. Figure 1 A schematic diagram of the computing device of the test management platform 100 shown. Figure 5 As shown, computing device 500 may include bus 502 or other communication mechanism for transmitting information, and one or more processors 504 coupled to bus 502 for processing information. The one or more processors 504 may include, for example, one or more general-purpose microprocessors.

[0118] like Figure 5As shown, in some implementations, computing device 500 may further include main memory 506 coupled to bus 502. Main memory 506 is used to store information (e.g., a historical database) and instructions executed by one or more processors 504, such as random access memory (RAM), cache, and / or other dynamic storage devices. Main memory 506 may also be used to store temporary variables or other intermediate information during the execution of instructions executed by one or more processors 504. When these instructions are stored in storage media accessible to one or more processors 504, they can cause computing device 500 to become a dedicated machine customized to perform the operations specified in the instructions. Storage device 508 may include non-volatile and / or volatile storage media. Non-volatile storage media may include, for example, optical disks or magnetic disks. Volatile storage media may include dynamic memory. Common forms of storage media may include, for example, floppy disks, hard disks, solid-state drives, magnetic tapes, or any other magnetic data storage media, CD-ROMs, any other optical data storage media, any physical media with a perforated pattern, RAM, DRAM, PROM, EPROM, FLASH-EPROM, NVRAM, any other memory chip or cartridge, or their networking versions.

[0119] like Figure 5 As shown, in some embodiments, computing device 500 may further include one or more communication interfaces or network interfaces 510 coupled to bus 502. Network interface 510 may provide bidirectional data communication coupling to one or more network links connected to one or more networks. As another example, network interface 510 may be a local area network (LAN) card to provide data communication connectivity to a LAN-compatible (or WAN component communicating with a WAN) network. Wireless links may also be implemented.

[0120] The execution of certain operations can be distributed across processors rather than residing within a single machine, but rather deployed across multiple machines. In some example embodiments, the processor or processor-implemented engine may reside in a single geographic location (e.g., in a home environment, office environment, or server farm). In other example embodiments, the processor or processor-implemented engine may be distributed across multiple geographic locations.

[0121] Each of the processes, methods, and algorithms described in the preceding sections may be embodied in code modules executed by one or more computer systems or computer processors including computer hardware, and may be fully or partially automated by these code modules. The processes and algorithms may be implemented, partially or fully, in dedicated circuit systems.

[0122] When the functions disclosed herein are implemented as software functional units and sold or used as stand-alone products, they may be stored in a processor-executable, non-volatile, computer-readable storage medium. Specific technical solutions (all or part) disclosed herein, or aspects contributing to the prior art, may be embodied in the form of a software product. The software product may be stored in a storage medium and includes several instructions that cause a computing device (which may be a personal computer, server, network device, etc.) to perform all or some steps of the methods of the embodiments of this application. The storage medium may include a flash drive, portable hard disk drive, ROM, RAM, magnetic disk, optical disk, other media operable to store program code, or any combination thereof.

[0123] According to embodiments of this application, a system is provided that includes a processor and a non-transitory computer-readable storage medium storing instructions, which are executable by the processor to cause the system to perform operations corresponding to steps in any method of the embodiments disclosed above. According to embodiments of this application, a non-transitory computer-readable storage medium or computer program product storing instructions, which are executable by one or more processors to cause the one or more processors to perform operations corresponding to steps in any method of the embodiments disclosed above.

[0124] The various features and processes described above can be used independently of each other or combined in various ways. All possible combinations and sub-combinations are intended to fall within the scope of this disclosure. Additionally, certain method or process blocks may be omitted in some embodiments. The methods and processes described herein are not limited to any particular order, and their associated blocks or states may be executed in other suitable orders. For example, described blocks or states may be executed in an order other than that specifically disclosed, or multiple blocks or states may be combined into a single block or state. Example blocks or states may be executed sequentially, in parallel, or in some other manner. Certain blocks or states may be added to or removed from the disclosed example embodiments. The exemplary systems and components described herein may be configured differently than described. For example, certain components may be added to, removed from, or rearranged compared to the disclosed example embodiments.

[0125] The various operations of the exemplary methods described herein can be performed at least in part by an algorithm. The algorithm may be included in program code or instructions stored in memory (e.g., the aforementioned non-transitory computer-readable storage medium). This algorithm may include a machine learning algorithm. In some embodiments, the machine learning algorithm may not explicitly turn the computer into an execution function but may learn from training data to produce a predictive model of the execution function.

[0126] The various operations of the exemplary methods described herein can be performed, at least in part, by one or more processors that are temporarily configured (e.g., by software) or permanently configured to perform the relevant operations. Whether temporarily or permanently configured, these processors can constitute an engine of processor implementation that operates to perform one or more of the operations or functions described herein.

[0127] Similarly, the methods described herein may be implemented at least in part by a processor, wherein one or more specific processors are instances of hardware. For example, at least some operations of the methods may be performed by one or more processors or an engine implemented by a processor. Furthermore, one or more processors may also be operable to support the execution of relevant operations in a “cloud computing” environment or as the execution of relevant operations in a “Software as a Service” (SaaS) context. For example, at least some operations may be performed by a group of computers (as an example of a machine containing processors), wherein these operations are accessible via a network (e.g., the Internet) and via one or more appropriate interfaces (e.g., application programming interfaces (APIs)).

[0128] In this specification, although individual operations of one or more methods are illustrated and described as separate operations, one or more of these individual operations may be performed simultaneously, and not necessarily in the order illustrated. Structures and functions presented as separate components in the example configurations may be implemented as composite structures or components. Similarly, structures and functions presented as single components may be implemented as single components. These and other variations, modifications, additions, and improvements fall within the scope of this document. Therefore, this specification and its drawings should be considered illustrative rather than restrictive.

[0129] As used herein, “or” is inclusive rather than exclusive unless explicitly indicated by the context. Therefore, in this document, “A, B, or C” means “A, B, A and B, A and C, B and C, or A, B, and C” unless explicitly indicated by the context. Furthermore, “and” is combined and separate unless explicitly indicated by the context. Therefore, in this document, “A and B” means “A and B, combined or separate” unless explicitly indicated by the context.

[0130] The terms “comprising” or “including” are used to indicate the presence of a subsequently claimed feature, but do not exclude the presence of other features. Unless otherwise specifically stated or otherwise understood in the context in which they are used, conditional language such as “may,” “can,” “may,” and “can” is generally intended to convey that certain embodiments include certain features, components, and / or steps that are not included in other embodiments. Therefore, this conditional language is generally not intended to imply that one or more embodiments necessarily require a certain feature, component, and / or step in any way, or that one or more embodiments must include such features, components, and / or steps.

[0131] Although the general outline of the subject matter has been described with reference to specific exemplary embodiments, various modifications and changes may be made to these embodiments without departing from the broad scope of embodiments of this disclosure. Where more than one embodiment is disclosed, these embodiments of the subject matter may be referred to individually or collectively herein as the term "invention," this is for convenience only and is not intended to automatically limit the scope of this application to any single disclosure or concept.

[0132] The embodiments illustrated herein are described in detail to enable those skilled in the art to practice the disclosed teachings. Other embodiments may be used and derived therefrom, such that structural and logical substitutions and changes may be made without departing from the scope of this disclosure. Therefore, the term "implementation" is not intended to be limiting, and the scope of the various embodiments is defined only by the appended claims and their equivalents.

Claims

1. A method for testing the reasoning ability of an artificial intelligence (AI) system, characterized in that... include: For scenarios involving the inference capabilities of AI models deployed on AI systems, a multidimensional spatial probability distribution model is constructed based on a structured configuration file that loads multiple atomic parameter terms defined for the scenario and at least one mutation policy rule. A dynamic test case set is generated based on the multidimensional spatial probability distribution model. as well as The AI ​​model deployed on the AI ​​system is tested using the dynamic test case set. The construction of the multidimensional spatial probability distribution model includes: A one-dimensional probability space is constructed for each dimension of the parameter atomic items of the multiple dimensions, wherein the one-dimensional probability space includes the values ​​of the parameter atomic items of that dimension and their probability weights. Based on the weight adjustment factor and / or mutation policy description of the at least one mutation policy rule, adjust the one-dimensional probability space of at least one dimension of the parameter atomic terms in the plurality of dimensions; and The adjusted one-dimensional probability spaces of each dimension in the multiple-dimensional parameter atomic terms are combined to form the multi-dimensional space probability distribution model.

2. The method according to claim 1, characterized in that, The structured configuration file is a configuration file using YAML / JSON extended syntax.

3. The method according to claim 1, characterized in that, The at least one mutation strategy rule includes any one of the following rules: semantic equivalence substitution, cue word injection interference, or parameter combination constraint.

4. The method according to claim 1, characterized in that, Adjusting at least one dimension of the one-dimensional probability space of the multiple dimension parameter atoms includes: Apply a weight adjustment factor of the associated mutation policy rule to the probability weights of specific values ​​in the at least one dimension of the one-dimensional probability space; and / or A mutation strategy description that attaches associated mutation strategy rules to the one-dimensional probability space of the at least one dimension.

5. The method according to claim 1, characterized in that, The dynamic test case set generated based on the multidimensional spatial probability distribution model includes: Based on the multidimensional spatial probability distribution model, weighted random sampling is performed on the adjusted one-dimensional probability space of each dimension in the multidimensional parameter atomic terms to form a combination of parameter atomic terms values; Instantiate the combinations of parameter atomic terms to generate basic test cases; The base test cases are dynamically mutated based on a mutation strategy description associated with the combination of values ​​of the instantiated parameter atoms to generate the dynamic test case set.

6. The method according to claim 5, characterized in that... Also includes: Collect indicator data for testing the AI ​​model; Based on the collected indicator data, the anomaly rate of the AI ​​model's response caused by test cases under each mutation strategy rule is statistically analyzed. as well as Based on the anomaly rate of the AI ​​model's response caused by the test cases under each mutation strategy rule and the average value of each indicator data under that mutation strategy rule, the weight adjustment factor and / or mutation strategy description of the mutation strategy rule are adjusted to update the multidimensional spatial probability distribution model and regenerate the dynamic test case set based on the updated multidimensional spatial probability distribution model.

7. The method according to claim 6, characterized in that, The metrics include: accuracy, perplexity, lexical stability, and defect pattern matching rate.

8. The method according to claim 6, characterized in that, The weight adjustment factors and / or mutation strategy descriptions that adjust the mutation strategy rules include: If the test cases generated under this mutation strategy rule cause the AI ​​model's response anomaly rate to exceed a threshold, then the weight adjustment factor of the mutation strategy rule should be increased and / or the mutation strategy description of the mutation strategy rule should be optimized based on the average value of each indicator data under the mutation strategy rule; and If the test cases generated under the mutation strategy rule result in the AI ​​model's response anomaly rate not exceeding the threshold, then the weight adjustment factor of the mutation strategy rule is reduced and / or the mutation strategy description of the mutation strategy rule is optimized based on the average value of each indicator data under the mutation strategy rule.

9. The method according to claim 1, characterized in that, The method is automatically triggered by the continuous integration / continuous deployment (CI / CD) pipeline after the AI ​​model is iterated.

10. A system for testing the reasoning ability of an artificial intelligence (AI) system, characterized in that... include: The configuration module is configured to construct a multidimensional spatial probability distribution model for the inference capability scenario of the AI ​​model deployed on the AI ​​system, based on a structured configuration file containing multiple dimensions of parameter atomic items and at least one mutation policy rule defined for the scenario. The generation module is configured to generate a dynamic test case set based on the multidimensional spatial probability distribution model; and The testing module is configured to perform tests on the AI ​​models deployed on the AI ​​system using the dynamic test case set. The configuration module is further configured as follows: A one-dimensional probability space is constructed for each dimension of the parameter atomic items of the multiple dimensions, wherein the one-dimensional probability space includes the values ​​of the parameter atomic items of that dimension and their probability weights. Based on the weight adjustment factor and / or mutation strategy description of the at least one mutation strategy rule, adjust the one-dimensional probability space of at least one dimension of the parameter atomic terms of the multiple dimensions; as well as The adjusted one-dimensional probability spaces of each dimension in the multiple-dimensional parameter atomic terms are combined to form the multi-dimensional space probability distribution model.

11. A non-transitory computer-readable medium storing instructions, characterized in that... When executed by one or more processors, the instructions cause the one or more processors to perform the method according to any one of claims 1-9.

12. A computer program product, comprising instructions, characterized in that... When executed by one or more processors, the instructions cause the one or more processors to perform the method according to any one of claims 1-9.

Images