Test method, device, electronic device, and storage medium

Abstract

The present disclosure relates to a test method, device, electronic device and storage medium. The method comprises: obtaining a test case; the test case comprising a test constraint condition, a candidate operation set and a reference operation link; the test constraint condition comprising initial state information and target state information; the candidate operation set comprising a plurality of candidate operations; the reference operation link being an operation sequence obtained by arranging the candidate operations in the candidate operation set in order; the initial state information being transformed into the target state information through the reference operation link; inputting the test constraint condition and the candidate operation set into a reasoning model to be tested to obtain a target operation link; the target operation link being an operation sequence obtained by sorting the candidate operations in the candidate operation set based on the test constraint condition by the reasoning model; and obtaining a test result for the reasoning model based on the reference operation link and the target operation link, which can automatically test the reasoning model and reduce the labor cost in the test process.

Description

Technical Field

[0001] This disclosure relates to the field of computer technology, and more particularly to a testing method, apparatus, electronic device, and storage medium. Background Technology

[0002] With the groundbreaking advancements in pre-trained Large Language Models (LLMs) in natural language processing, they have demonstrated powerful performance in complex tasks such as knowledge reasoning, multi-turn dialogue, and cross-modal understanding. However, existing evaluation systems struggle to automate the assessment of the true capabilities of large language models. Therefore, how to automate the testing of large language models has become a pressing issue. Summary of the Invention

[0003] To solve the above-mentioned technical problems, or at least partially solve them, this disclosure provides a testing method, apparatus, electronic device, and storage medium.

[0004] Firstly, this disclosure provides a testing method, including:

[0005] Obtain test cases; the test cases include test constraints, a set of candidate operations, and a reference operation chain; the test constraints include initial state information and target state information; the set of candidate operations includes multiple candidate operations; the reference operation chain is an operation sequence obtained by arranging the candidate operations in the set of candidate operations in order; the initial state information can be transformed into target state information through the reference operation chain;

[0006] The test constraints and the candidate operation set are input into the inference model to be tested to obtain the target operation chain; the target operation chain is the operation sequence obtained by the inference model sorting the candidate operations in the candidate operation set based on the test constraints.

[0007] Based on the reference operation chain and the target operation chain, test results are obtained for the inference model.

[0008] Secondly, this disclosure also provides a testing apparatus, comprising:

[0009] The acquisition module is used to acquire test cases; the test cases include test constraints, a set of candidate operations, and a reference operation chain; the test constraints include initial state information and target state information; the set of candidate operations includes multiple candidate operations; the reference operation chain is an operation sequence obtained by arranging the candidate operations in the set of candidate operations in order; the initial state information can be transformed into target state information through the reference operation chain.

[0010] The input module is used to input the test constraints and the candidate operation set into the inference model to be tested to obtain the target operation link; the target operation link is the operation sequence obtained by the inference model sorting the candidate operations in the candidate operation set based on the test constraints;

[0011] The evaluation module is used to obtain test results for the inference model based on the reference operation link and the target operation link.

[0012] Thirdly, this disclosure also provides an electronic device, the electronic device comprising:

[0013] One or more processors;

[0014] Storage device for storing one or more programs;

[0015] When the one or more programs are executed by the one or more processors, the one or more processors implement the testing method as described above.

[0016] Fourthly, this disclosure also provides a computer-readable storage medium having a computer program stored thereon that, when executed by a processor, implements the test method described above.

[0017] The technical solution provided in this disclosure has the following advantages compared with the prior art:

[0018] The technical solution provided in this disclosure involves setting up test cases. Each test case includes test constraints, a set of candidate operations, and a reference operation chain. Test constraints include initial state information and target state information. The set of candidate operations includes multiple candidate operations. The reference operation chain is an operation sequence obtained by arranging the candidate operations in the set in order. The reference operation chain transforms the initial state information into target state information. The test constraints and the set of candidate operations are input into the inference model to be tested to obtain the target operation chain. The target operation chain is an operation sequence obtained by the inference model sorting the candidate operations in the set based on the test constraints. Based on the reference operation chain and the target operation chain, test results for the inference model are obtained. Essentially, this provides a method for automating the testing of inference models, reducing the human cost of testing inference models. Attached Figure Description

[0019] The accompanying drawings, which are incorporated in and form a part of this specification, illustrate embodiments consistent with this disclosure and, together with the description, serve to explain the principles of this disclosure.

[0020] To more clearly illustrate the technical solutions in the embodiments of this disclosure or the prior art, the accompanying drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0021] Figure 1 A flowchart of a testing method provided in an embodiment of this disclosure;

[0022] Figure 2 A schematic diagram of a testing method provided in an embodiment of this disclosure;

[0023] Figure 3 This is a schematic diagram of the structure of a testing device according to an embodiment of the present disclosure;

[0024] Figure 4 This is a schematic diagram of the structure of an electronic device according to an embodiment of the present disclosure. Detailed Implementation

[0025] To better understand the above-mentioned objectives, features, and advantages of this disclosure, the solutions disclosed herein will be further described below. It should be noted that, unless otherwise specified, the embodiments and features described herein can be combined with each other.

[0026] Numerous specific details are set forth in the following description in order to provide a full understanding of this disclosure, but this disclosure may also be implemented in other ways different from those described herein; obviously, the embodiments in the specification are only some, and not all, of the embodiments of this disclosure.

[0027] Figure 1 This flowchart illustrates a testing method provided in an embodiment of the present disclosure. This embodiment is applicable to testing on a client-side basis. The method can be executed by a testing device, which can be implemented in software and / or hardware. This device can be configured in an electronic device, such as a terminal, including but not limited to smartphones, PDAs, tablets, wearable devices with displays, desktop computers, laptops, all-in-one computers, and smart home devices. Alternatively, this embodiment can be applied to testing on a server-side basis. The method can also be executed by a testing device, which can be implemented in software and / or hardware and can be configured in an electronic device, such as a server.

[0028] like Figure 1 As shown, the method may specifically include:

[0029] S110. Obtain test cases; test cases include test constraints, candidate operation sets, and reference operation chains; test constraints include initial state information and target state information; candidate operation sets include multiple candidate operations; reference operation chains are operation sequences obtained by arranging the candidate operations in the candidate operation set in order; the initial state information can be transformed into target state information through the reference operation chains.

[0030] Test cases can be, for example, datasets used to test the inference model under test. An inference model can be, for example, a trained machine learning model capable of making predictions or decisions on new data; exemplarily, it can be a neural network model. In some scenarios, the inference model can be a large language model.

[0031] In a test, the reasoning process of the inference model is analogized to a problem-solving process. The test constraints can be seen as the problem the inference model needs to solve; the candidate operations in the candidate operation set can be seen as knowledge points related to the problem given by the test constraints; and the reference operation chain can be seen as a problem-solving method that can be used to solve the problem given by the test constraints. This problem-solving method includes multiple solution steps, and the reference operation chain includes multiple candidate operations, each of which can be considered a solution step. This problem-solving method (i.e., the reference operation chain) is composed of multiple solution steps (i.e., candidate operations) arranged in a specific order.

[0032] Initial state information can be, for example, information describing the starting state of the inference. Target state information can be, for example, information describing the ending state of the inference. In the reference operation chain, each candidate operation is considered a step. The total number of steps in the reference operation chain is the sum of the number of times each candidate operation is used in the reference operation chain. If the initial state information is incorporated into the reference operation chain, and inference is performed step-by-step according to the steps included in the reference operation chain, the final inference result obtained is the target state information.

[0033] It should be noted that the total number of steps in the reference operation chain is not necessarily equal to the total number of candidate operations in the candidate operation set. In some scenarios, the total number of steps in the reference operation chain is greater than or equal to the total number of candidate operations in the candidate operation set. For example, any two candidate operations in the candidate operation set are different, and the reference operation chain includes all candidate operations in the candidate operation set. In this case, if the total number of steps in the reference operation chain is equal to the total number of candidate operations in the candidate operation set, it means that every candidate operation in the candidate operation set is used in the reference operation chain, and each is used once. If the total number of steps in the reference operation chain is greater than the total number of candidate operations in the candidate operation set, it means that every candidate operation in the candidate operation set is used in the reference operation chain, and at least some candidate operations are used more than once.

[0034] S120. Input the test constraints and candidate operation set into the inference model to be tested to obtain the target operation chain; the target operation chain is the operation sequence obtained by the inference model sorting the candidate operations in the candidate operation set based on the test constraints.

[0035] The essence of this step is that the reasoning model under test provides a solution method by testing the given problem under the constraints. The target operation chain is the solution method provided by the reasoning model under test.

[0036] S130. Based on the reference operation link and the target operation link, the test results for the inference model are obtained.

[0037] There are various ways to implement this step, and this application does not limit it. For example, the implementation method of this step includes: obtaining test results for the inference model based on at least one of the following: the difference between the order of candidate operations in the reference operation chain and the order of candidate operations in the target operation chain; the difference between the total number of steps in the reference operation chain and the total number of steps in the target operation chain; and the difference between the inference results corresponding to each candidate operation step when inferring according to the reference operation chain and the target operation chain with initial state information as input.

[0038] For example, consider testing the mathematical computational capability of a certain inference model. The constraints in the test cases include "2" and "-4". Here, "2" represents the initial state information, and "-4" represents the target state information. The candidate operation set includes three candidate operations: Operation 1, Operation 2, and Operation 3. Operation 1 is the "-2" operation, described as "the value obtained by subtracting 2 from the input number". Operation 2 is the "+1" operation, described as "the value obtained by adding 1 to the input number". Operation 3 is the "×4" operation, described as "the value obtained by multiplying the input number by 4". The reference operation chain is "Operation 1 → Operation 1 → Operation 2 → Operation 3". The total number of steps in this reference operation chain is 4. Using the initial state information "2" as input, after step 1 (i.e., the first operation 1), the reasoning result is 0, that is, the reasoning result corresponding to step 1 is 0; after step 2 (i.e., the second operation 1), the reasoning result is -2, that is, the reasoning result corresponding to step 2 is -2; after step 3 (i.e., operation 2), the reasoning result is -1, that is, the reasoning result corresponding to step 3 is -1; after step 4 (i.e., operation 3), the reasoning result is -4, that is, the reasoning result corresponding to step 4 is -4.

[0039] The test constraints and candidate operation set from the test cases are input into the first inference model to be tested. Assume the target operation chain is "Operation 1 → Operation 2 → Operation 2 → Operation 3". The total number of steps in this target operation chain is 4. Using the initial state information "2" as input, after step 1 (operation 1), the inference result is 0; after step 2 (the first operation 2), the inference result is 1; after step 3 (the second operation 2), the inference result is 2; and after step 4 (operation 3), the inference result is 8.

[0040] The test constraints and candidate operation set from the test cases are input into the second inference model to be tested. Assume the target operation chain is "Operation 2 → Operation 1 → Operation 1 → Operation 3". The total number of steps in this target operation chain is 4. Using the initial state information "2" as input, after step 1 (i.e., operation 2), the inference result is 3, meaning the inference result corresponding to step 1 is 3; after step 2 (i.e., the first operation 1), the inference result is 1, meaning the inference result corresponding to step 2 is 1; after step 3 (i.e., the second operation 1), the inference result is -1, meaning the inference result corresponding to step 3 is -1; after step 4 (i.e., operation 3), the inference result is -4, meaning the inference result corresponding to step 4 is -4.

[0041] The test constraints and candidate operation set from the test cases are input into the third inference model to be tested. Assume the target operation chain is "Operation 1 → Operation 1 → Operation 2 → Operation 3". The total number of steps in this target operation chain is 4. Using the initial state information "2" as input, after step 1 (i.e., the first operation 1), the inference result is 0; after step 2 (i.e., the second operation 1), the inference result is -2; after step 3 (i.e., operation 2), the inference result is -1; and after step 4 (i.e., operation 3), the inference result is -4.

[0042] When determining the test results for the reasoning model, evaluation is conducted from three aspects:

[0043] Firstly, is the order of candidate operations in the reference operation chain the same as the order of candidate operations in the target operation chain? In the example above, the order of candidate operations in the reference operation chain is different from the order of candidate operations in the target operation chains given by the first two inference models to be tested. Specifically, comparing the reference operation chain with the target operation chain given by the first inference model to be tested reveals that step 2 is operation 1 in the reference operation chain, but step 2 is operation 2 in the target operation chain given by the first inference model to be tested. Comparing the reference operation chain with the target operation chain given by the second inference model to be tested reveals that steps 1 and 2 are both operation 1 in the reference operation chain, and step 3 is operation 2. However, in the target operation chain given by the second inference model to be tested, step 1 is operation 2, and steps 2 and 3 are both operation 1. The reference operation chain and the target operation chain given by the third inference model to be tested are completely identical.

[0044] Secondly, it's important to determine whether the total number of steps in the reference operation chain is the same as the total number of steps in the target operation chain. In the example above, the total number of steps in the reference operation chain is the same as the total number of steps in the target operation chain given by the three inference models under test, both being 4.

[0045] Thirdly, using initial state information as input, inference is performed according to the reference operation link and the target operation link, and the differences in the inference results corresponding to each candidate operation step are analyzed. In the example above, using initial state information as input, inference is performed according to the reference operation link and the target operation link given by the first inference model to be tested. From step 2 onwards, the inference results corresponding to each step are different, and the inference result (8) given by the last step of the target operation link given by the first inference model to be tested is not equal to the target state information (-4). Using initial state information as input, inference is performed according to the reference operation link and the target operation link given by the second inference model to be tested. The inference results corresponding to the candidate operation steps in steps 1 and 2 are different, but the inference result (-4) given by the last step of the target operation link given by the second inference model to be tested is equal to the target state information (-4). Using initial state information as input, inference is performed according to the reference operation link and the target operation link given by the third inference model to be tested. The inference results corresponding to each step are the same.

[0046] Based on the evaluation results from these three aspects, test results for the inference model can be obtained. It should also be noted that in practice, specific scoring rules and weight values ​​can be set for each of the three aspects, allowing the evaluation results to be quantified.

[0047] It should be noted that in the above example, the technical solution provided by this application is applied to test the mathematical computation ability of the inference model. This is merely a specific example of this application and not a limitation thereof. In practice, other capabilities of the inference model (such as programming ability, text writing ability, multi-turn dialogue ability, and cross-modal understanding ability) can also be tested. For example, if the method provided by the technical solution of this application is used to test the programming ability of the inference model, the candidate operations can be functions that need to be called during the programming process. For example, the initial state information can be, for example, a sales record table including multiple personnel, which includes the salesperson's name, sales date, and sales amount for each sales date. The target state information can be the total sales amount for each salesperson. In the candidate operation set, each candidate operation is a function that needs to be called during the programming process to obtain the total sales amount for each person. These functions can be functions for calling the sales record table, functions for calculating the total sales amount for each salesperson, and functions for exporting the total sales amount for each salesperson in a specific format.

[0048] Furthermore, this step may also include: determining whether the target operation link meets the test constraints to obtain a first judgment result; determining whether the reasoning results corresponding to each candidate operation step obtained by the inference model according to the target operation link with the initial state information as input are correct to obtain a second judgment result; determining the time taken by the inference model to obtain the target operation link; and obtaining a test result for the inference model based on the first judgment result, the second judgment result, the time taken by the inference model to obtain the target operation link, the reference operation link, and the target operation link.

[0049] In practice, situations may arise where, using initial state information as input and reasoning along the target operation chain, the target state information cannot be obtained. For example, in the test constraints, the target state information might be "-4," but using the initial state information as input and reasoning along the target operation chain yields a result of "4" corresponding to the last operation step, which differs from the target state information (i.e., "-4"). Furthermore, in practice, test constraints may include restrictions on the total number of target steps, but the target operation chain may not satisfy these restrictions. The total number of target steps is the total number of steps included in the target operation chain. The restrictions on the total number of target steps are conditions that are constructed during the test case development process to achieve the desired outcome. For example, test constraints might specify that the total number of target steps is greater than or equal to 4 and less than or equal to 6. However, if the target operation chain has a target total of 7 steps, it does not satisfy the restriction that the total number of target steps is greater than or equal to 4 and less than or equal to 6. These situations all fall under the category of the target operation chain not meeting the test constraints.

[0050] In other scenarios, given the initial state information, the inference model infers the correct results for each candidate operation step by following the target operation chain. For example, suppose the initial state information is "2" and the target operation chain is "Operation 1 (i.e., '+4' operation) → Operation 1 (i.e., '-1' operation)". Using the initial state information "2" as input, after step 1 (i.e., operation 1), the inference model obtains a result of "7" corresponding to step 1. However, if the calculation is correct, the result corresponding to step 1 should be "6". This situation illustrates how using the initial state information as input and inferring the correct results for each candidate operation step by following the target operation chain can lead to incorrect inference results.

[0051] The time taken for the inference model to obtain the target operation chain can be, for example, the time taken for the inference model to obtain the target operation chain and the inference results corresponding to each candidate operation step.

[0052] By setting the time consumption of the target operation link, the reference operation link, and the target operation link based on the first judgment result, the second judgment result, and the reasoning model, test results for the reasoning model are obtained. The purpose is to comprehensively evaluate the reasoning model from the perspectives of logical reasoning, instruction compliance, and thinking efficiency, thereby improving the accuracy of the test results for the reasoning model.

[0053] The above technical solution obtains test cases by setting up test constraints, candidate operation sets, and reference operation chains. Test constraints include initial state information and target state information. The candidate operation set includes multiple candidate operations. The reference operation chain is an operation sequence obtained by arranging the candidate operations in the candidate operation set in order. The reference operation chain transforms the initial state information into target state information. The test constraints and candidate operation set are input into the inference model to be tested to obtain the target operation chain. The target operation chain is an operation sequence obtained by the inference model sorting the candidate operations in the candidate operation set based on the test constraints. Based on the reference operation chain and the target operation chain, the test results for the inference model are obtained. Essentially, it provides a method for automating the testing of inference models, reducing the human cost of testing inference models.

[0054] Based on the above technical solution, the method may optionally include obtaining test cases. There are various specific implementation methods for "obtaining test cases," and this application does not limit this. For example, obtaining test cases includes: obtaining initial state information and a set of candidate operations; arranging the candidate operations in the set of candidate operations to obtain a reference operation chain; obtaining target state information based on the initial state information and the reference operation chain; using the initial state information and the target state information as test constraints, and associating the test constraints, the set of candidate operations, and the reference operation chain to obtain test cases.

[0055] In some scenarios, during the process of "arranging candidate operations in the candidate operation set to obtain a reference operation chain", the arrangement of candidate operations in the candidate operation set can be random or according to a certain sorting rule. This application does not impose any restrictions on this.

[0056] In some scenarios, the initial state information can be randomly generated. "Obtaining target state information based on initial state information and reference operation links" essentially means using the initial state information as input and reasoning according to the reference operation links to obtain the target state information.

[0057] By setting up initial state information and a set of candidate operations, arranging the candidate operations in the set to obtain a reference operation chain, obtaining target state information based on the initial state information and the reference operation chain, and using the initial state information and target state information as test constraints, and associating the test constraints, the set of candidate operations, and the reference operation chain to obtain test cases, it can be ensured that, given the initial state information as input, the reference operation chain can transform the initial state information into target state information, thereby achieving the goal of automatically generating high-quality test cases.

[0058] In practical applications, different inference models exhibit significant performance differences across various domains. For example, some inference models excel in code generation, while others stand out in copywriting, and still others have advantages in knowledge-based question answering and language translation. Furthermore, the performance of the same inference model varies across different domains; some models perform exceptionally well in coding tasks but poorly in copywriting. Moreover, with technological advancements, the capabilities of inference models rapidly improve during iterations. Therefore, when testing inference models, it is necessary to design differentiated test cases that accurately reflect the differences in capabilities of each model while also covering a wide range of difficulty levels to comprehensively evaluate the performance of the inference models. Optionally, in some embodiments, obtaining a candidate operation set may include: obtaining test difficulty levels, test target function information, and a set of optional operations; the set of optional operations includes multiple first operations; each first operation has a corresponding difficulty level and test function information; based on the test difficulty level and test target function information, the first operations in the set of optional operations are filtered to obtain candidate operations; and the candidate operations are added to the candidate operation set.

[0059] Test target functionality information could be, for example, information reflecting which aspect of the inference model's functionality we want to test. Test difficulty level could be, for example, information reflecting which level of test cases we want to use to test the inference model in order to determine whether the inference model meets the basic requirements of that level.

[0060] The first operation has a corresponding difficulty level and test function description information, meaning that the first operation includes a difficulty level attribute and a test function attribute. The test function attribute of the first operation reflects what functions it can be used to test. The difficulty level attribute of the first operation reflects that if the inference model can correctly apply the first operation, it means that the inference model meets the basic requirements of the corresponding level.

[0061] By setting parameters based on the test difficulty level and the target function information, the first operation in the set of optional operations is filtered to obtain candidate operations; these candidate operations are then added to the candidate operation set. The purpose is to enable the system to automatically select candidate operations from the set of optional operations when the test difficulty level and target function information are known, thereby constructing test cases suitable for the inference model under test and completing the automated testing of the inference model.

[0062] Furthermore, obtaining the test difficulty level and test target function information may include: determining the model characteristics of the inference model to be tested; the model characteristics of the inference model include the number of parameters and / or the internal structure of the inference model; and determining the test difficulty level and test target function information based on the model characteristics of the inference model to be tested.

[0063] The number of parameters in an inference model can be, for example, the number of learnable variables in the inference model. For instance, the inference model is a neural network, and the learnable variables in the inference model can include, for example, weights and biases in the neural network.

[0064] The internal structure of an inference model can be, for example, the architecture of the inference model, which can specifically include the selection and organization of its components (such as layers, activation functions, and connection methods).

[0065] By setting and determining the model characteristics of the inference model to be tested, including the number of parameters and / or the internal structure of the inference model, the test difficulty level and test target function information are determined based on the model characteristics of the inference model to be tested. Essentially, this involves using the model characteristics of the inference model to predict the functions that the inference model has and its performance in those functions, thereby determining the more suitable test difficulty level and test target function information for the inference model, so as to further achieve the purpose of automated testing of the inference model.

[0066] Based on the above technical solution, optionally, the candidate operations in the candidate operation set are arranged to obtain a reference operation link, which may include: determining the total number of reference steps; arranging the candidate operations in the candidate operation set based on the total number of reference steps to obtain a reference operation link; the total number of steps in the reference operation link is equal to the total number of reference steps; the total number of steps in the reference operation link is the sum of the number of times each candidate operation is used in the reference operation link; determining the range of values ​​for the total number of target steps based on the total number of reference steps; and using the initial state information and the target state information as test constraints, including: using the range of values ​​for the total number of target steps, the initial state information, and the target state information as test constraints.

[0067] The total number of reference steps can be a pre-specified value that limits the total number of steps included in the reference step count. In some scenarios, the total number of reference steps can be generated using a random number generator and is therefore a random number.

[0068] There are various specific methods for "determining the range of values ​​for the target total number of steps based on the total number of reference steps," and this application does not limit this. For example, the range of values ​​for the target total number of steps can be set to a numerical range with the total number of reference steps as the median. For instance, if the total number of reference steps is 4, the range of values ​​for the target total number of steps is greater than or equal to 3 and less than or equal to 5.

[0069] By setting the range of the total number of target steps, initial state information, and target state information as test constraints, the purpose is to make the testing process more controllable, thereby improving the accuracy of the test results.

[0070] It should be noted that, for the sake of simplicity, the foregoing method embodiments are all described as a series of actions. However, those skilled in the art should understand that the present invention is not limited to the described order of actions, because according to the present invention, some steps can be performed in other orders or simultaneously. Furthermore, those skilled in the art should also understand that the embodiments described in the specification are preferred embodiments, and the actions and modules involved are not necessarily essential to the present invention.

[0071] Figure 3 This is a schematic diagram of a testing device according to an embodiment of this disclosure. The testing device provided in this embodiment can be configured in a client or a server. See also Figure 3 The testing device specifically includes:

[0072] The acquisition module 310 is used to acquire test cases; the test cases include test constraints, a set of candidate operations, and a reference operation chain; the test constraints include initial state information and target state information; the set of candidate operations includes multiple candidate operations; the reference operation chain is an operation sequence obtained by arranging the candidate operations in the set of candidate operations in order; the initial state information can be transformed into target state information through the reference operation chain;

[0073] The input module 320 is used to input the test constraints and the candidate operation set into the inference model to be tested to obtain the target operation link; the target operation link is the operation sequence obtained by the inference model sorting the candidate operations in the candidate operation set based on the test constraints;

[0074] Evaluation module 330 is used to obtain test results for the inference model based on the reference operation link and the target operation link.

[0075] Furthermore, module 310 is used for:

[0076] Obtain initial state information and candidate operation set;

[0077] Arrange the candidate operations in the candidate operation set to obtain a reference operation chain;

[0078] Based on the initial state information and the reference operation link, the target state information is obtained;

[0079] The initial state information and the target state information are used as the test constraints, and the test constraints, the candidate operation set, and the reference operation chain are associated to obtain test cases.

[0080] Furthermore, module 310 is used for:

[0081] The test difficulty level, test target function information, and optional operation set are obtained; the optional operation set includes multiple first operations; each first operation has a corresponding difficulty level and test function description information.

[0082] Based on the test difficulty level and the test target function information, the first operation in the set of optional operations is filtered to obtain candidate operations;

[0083] Add the candidate operation to the candidate operation set.

[0084] Furthermore, module 310 is used for:

[0085] Determine the model features of the inference model to be tested; the model features of the inference model include the number of parameters of the inference model and / or the internal structure of the inference model;

[0086] Based on the model characteristics of the inference model to be tested, the test difficulty level and the test target function information are determined.

[0087] Furthermore, module 310 is used for:

[0088] Determine the total number of reference steps;

[0089] Based on the total number of reference steps, the candidate operations in the candidate operation set are arranged to obtain a reference operation chain; the total number of steps in the reference operation chain is equal to the total number of reference steps; the total number of steps in the reference operation chain is the sum of the number of times each candidate operation in the reference operation chain is used.

[0090] Based on the total number of reference steps, determine the range of values ​​for the total number of target steps;

[0091] The range of the total number of target steps, the initial state information, and the target state information are used as the test constraints.

[0092] Furthermore, the evaluation module 330 is used for:

[0093] The test results for the inference model are obtained based on at least one of the following: the difference between the order of the candidate operations in the reference operation chain and the order of the candidate operations in the target operation chain; the difference between the total number of steps in the reference operation chain and the total number of steps in the target operation chain; and the difference between the inference results corresponding to each candidate operation step when inferring according to the reference operation chain and the target operation chain with the initial state information as input.

[0094] Furthermore, the evaluation module 330 is used for:

[0095] Determine whether the target operation link meets the test constraints to obtain a first determination result;

[0096] A second judgment result is obtained by determining whether the reasoning results corresponding to each candidate operation step are correct when the reasoning model performs reasoning according to the target operation chain with the initial state information as input.

[0097] Determine the time taken by the inference model to obtain the target operation link;

[0098] Based on the first judgment result, the second judgment result, the time consumption of the target operation link obtained by the inference model, the reference operation link, and the target operation link, test results are obtained for the inference model.

[0099] The testing apparatus provided in this disclosure embodiment can execute the steps performed by the client or server in the testing method provided in this disclosure method embodiment, and has the execution steps and beneficial effects, which will not be described in detail here.

[0100] The following is a detailed reference. Figure 4 The diagram illustrates a structural schematic suitable for implementing the electronic device 1000 in the embodiments of this disclosure. The electronic device 1000 in the embodiments of this disclosure may include, but is not limited to, mobile terminals such as mobile phones, laptops, digital broadcast receivers, PDAs (personal digital assistants), PADs (tablet computers), PMPs (portable multimedia players), in-vehicle terminals (e.g., in-vehicle navigation terminals), wearable electronic devices, etc., as well as fixed terminals such as digital TVs, desktop computers, smart home devices, etc. Figure 4 The electronic device shown is merely an example and should not be construed as limiting the functionality and scope of the embodiments disclosed herein.

[0101] like Figure 4 As shown, the electronic device 1000 may include a processing device (e.g., a central processing unit, a graphics processing unit, etc.) 1001, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 1002 or a program loaded from a storage device 1008 into a random access memory (RAM) 1003 to implement the test methods as described in the embodiments of this disclosure. The RAM 1003 also stores various programs and information required for the operation of the electronic device 1000. The processing device 1001, ROM 1002, and RAM 1003 are interconnected via a bus 1004. An input / output (I / O) interface 1005 is also connected to the bus 1004.

[0102] Typically, the following devices can be connected to the I / O interface 1005: input devices 1006 including, for example, a touchscreen, touchpad, keyboard, mouse, camera, microphone, accelerometer, gyroscope, etc.; output devices 1007 including, for example, a liquid crystal display (LCD), speaker, vibrator, etc.; storage devices 1008 including, for example, magnetic tape, hard disk, etc.; and communication devices 1009. Communication device 1009 allows electronic device 1000 to exchange information with other devices wirelessly or via wired communication. Although Figure 4 An electronic device 1000 with various devices is shown; however, it should be understood that it is not required to implement or possess all of the devices shown. More or fewer devices may be implemented or possessed alternatively.

[0103] In particular, according to embodiments of this disclosure, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments of this disclosure include a computer program product comprising a computer program carried on a non-transitory computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts, thereby implementing the testing methods as described above. In such embodiments, the computer program can be downloaded and installed from a network via communication device 1009, or installed from storage device 1008, or installed from ROM 1002. When the computer program is executed by processing device 1001, it performs the functions defined in the methods of embodiments of this disclosure.

[0104] It should be noted that the computer-readable medium described in this disclosure can be a computer-readable signal medium or a computer-readable storage medium, or any combination thereof. A computer-readable storage medium can be, for example,—but not limited to—an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this disclosure, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. In this disclosure, a computer-readable signal medium can include information signals propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated information signals can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A computer-readable signal medium can be any computer-readable medium other than a computer-readable storage medium, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to: wires, optical fibers, RF (radio frequency), etc., or any suitable combination thereof.

[0105] In some implementations, clients and servers may communicate using any known or future network protocol, such as HTTP (Hypertext Transfer Protocol), and may interconnect with any form or medium of digital information communication (e.g., a communication network). Examples of communication networks include local area networks (“LANs”), wide area networks (“WANs”), the Internet (e.g., the Internet of Things), and peer-to-peer networks (e.g., ad hoc peer-to-peer networks), as well as any known or future network.

[0106] The aforementioned computer-readable medium may be included in the aforementioned electronic device; or it may exist independently and not assembled into the electronic device.

[0107] The aforementioned computer-readable medium carries one or more programs that, when executed by the electronic device, cause the electronic device to:

[0108] Obtain test cases; the test cases include test constraints, a set of candidate operations, and a reference operation chain; the test constraints include initial state information and target state information; the set of candidate operations includes multiple candidate operations; the reference operation chain is an operation sequence obtained by arranging the candidate operations in the set of candidate operations in order; the initial state information can be transformed into target state information through the reference operation chain;

[0109] The test constraints and the candidate operation set are input into the inference model to be tested to obtain the target operation chain; the target operation chain is the operation sequence obtained by the inference model sorting the candidate operations in the candidate operation set based on the test constraints.

[0110] Based on the reference operation chain and the target operation chain, test results are obtained for the inference model.

[0111] Optionally, when one or more of the above-described procedures are executed by the electronic device, the electronic device may also execute other steps described in the above embodiments.

[0112] Computer program code for performing the operations of this disclosure can be written in one or more programming languages ​​or a combination thereof, including but not limited to object-oriented programming languages ​​such as Java, Smalltalk, and C++, as well as conventional procedural programming languages ​​such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0113] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0114] The units described in the embodiments of this disclosure can be implemented in software or hardware. The names of the units are not, in some cases, intended to limit the specific unit.

[0115] The functions described above in this document can be performed, at least in part, by one or more hardware logic components. For example, exemplary types of hardware logic components that can be used, without limitation, include: Field Programmable Gate Arrays (FPGAs), Application-Specific Integrated Circuits (ASICs), Application Standard Products (ASSPs), System-on-Chip (SoCs), Complex Programmable Logic Devices (CPLDs), and so on.

[0116] In the context of this disclosure, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0117] According to one or more embodiments of this disclosure, this disclosure provides an electronic device, including:

[0118] One or more processors;

[0119] Memory, used to store one or more programs;

[0120] When the one or more programs are executed by the one or more processors, the one or more processors implement any of the test methods provided in this disclosure.

[0121] According to one or more embodiments of the present disclosure, the present disclosure provides a computer-readable storage medium having a computer program stored thereon that, when executed by a processor, implements any of the test methods provided in the present disclosure.

[0122] This disclosure also provides a computer program product, which includes a computer program or instructions that, when executed by a processor, implement the test method described above.

[0123] It should be noted that, in this document, relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0124] The above description is merely a specific embodiment of this disclosure, enabling those skilled in the art to understand or implement it. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of this disclosure. Therefore, this disclosure is not to be limited to the embodiments described herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A testing method, characterized in that, include: Obtain test cases; the test cases include test constraints, candidate operation sets, and reference operation chains; The test constraints include initial state information and target state information; The candidate operation set includes multiple candidate operations; the reference operation chain is an operation sequence obtained by arranging the candidate operations in the candidate operation set in order; the initial state information can be transformed into target state information through the reference operation chain; The test constraints and the candidate operation set are input into the inference model to be tested to obtain the target operation chain; The target operation chain is an operation sequence obtained by the inference model sorting the candidate operations in the candidate operation set based on the test constraints; Based on the reference operation chain and the target operation chain, test results are obtained for the inference model.

2. The method according to claim 1, characterized in that, Obtain test cases, including: Obtain initial state information and candidate operation set; Arrange the candidate operations in the candidate operation set to obtain a reference operation chain; Based on the initial state information and the reference operation link, the target state information is obtained; The initial state information and the target state information are used as the test constraints, and the test constraints, the candidate operation set, and the reference operation chain are associated to obtain test cases.

3. The method according to claim 2, characterized in that, Obtaining the candidate operation set includes: The test difficulty level, test target function information, and optional operation set are obtained; the optional operation set includes multiple first operations; each first operation has a corresponding difficulty level and test function description information. Based on the test difficulty level and the test target function information, the first operation in the set of optional operations is filtered to obtain candidate operations; Add the candidate operation to the candidate operation set.

4. The method according to claim 3, characterized in that, The acquisition of the test difficulty level and the test target function information includes: Determine the model features of the inference model to be tested; the model features of the inference model include the number of parameters of the inference model and / or the internal structure of the inference model; Based on the model characteristics of the inference model to be tested, the test difficulty level and the test target function information are determined.

5. The method according to claim 2, characterized in that, The step of arranging the candidate operations in the candidate operation set to obtain a reference operation chain includes: Determine the total number of reference steps; Based on the total number of reference steps, the candidate operations in the candidate operation set are arranged to obtain a reference operation chain; the total number of steps in the reference operation chain is equal to the total number of reference steps; the total number of steps in the reference operation chain is the sum of the number of times each candidate operation in the reference operation chain is used. Based on the total number of reference steps, determine the range of values ​​for the total number of target steps; The step of using the initial state information and the target state information as the test constraints includes: The range of the total number of target steps, the initial state information, and the target state information are used as the test constraints.

6. The method according to claim 1, characterized in that, The step of obtaining test results for the inference model based on the reference operation link and the target operation link includes: The test results for the inference model are obtained based on at least one of the following: the difference between the order of the candidate operations in the reference operation chain and the order of the candidate operations in the target operation chain; the difference between the total number of steps in the reference operation chain and the total number of steps in the target operation chain; and the difference between the inference results corresponding to each candidate operation step when inferring according to the reference operation chain and the target operation chain with the initial state information as input.

7. The method according to claim 1, characterized in that, The step of obtaining test results for the inference model based on the reference operation link and the target operation link includes: Determine whether the target operation link meets the test constraints to obtain a first determination result; A second judgment result is obtained by determining whether the reasoning results corresponding to each candidate operation step are correct when the reasoning model performs reasoning according to the target operation chain with the initial state information as input. Determine the time taken by the inference model to obtain the target operation link; Based on the first judgment result, the second judgment result, the time consumption of the target operation link obtained by the inference model, the reference operation link, and the target operation link, test results are obtained for the inference model.

8. A testing device, characterized in that, include: The acquisition module is used to acquire test cases; The test cases include test constraints, candidate operation sets, and reference operation chains. The test constraints include initial state information and target state information; The candidate operation set includes multiple candidate operations; the reference operation chain is an operation sequence obtained by arranging the candidate operations in the candidate operation set in order; the initial state information can be transformed into target state information through the reference operation chain; The input module is used to input the test constraints and the candidate operation set into the inference model to be tested to obtain the target operation link; The target operation chain is an operation sequence obtained by the inference model sorting the candidate operations in the candidate operation set based on the test constraints; The evaluation module is used to obtain test results for the inference model based on the reference operation link and the target operation link.

9. An electronic device, characterized in that, The electronic device includes: One or more processors; Storage device for storing one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors implement the method as described in any one of claims 1-7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the method as described in any one of claims 1-7.

Images