Model accuracy test method and device, electronic equipment and storage medium
By optimizing test cases based on test requirements and use case types and building an adaptive test case set, the problem of large errors in model accuracy measurement results in the prior art is solved, and more accurate and reliable model accuracy measurement is achieved.
Patent Information
- Application Number
- CN202411643274.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-11-18
- Publication Date
- 2025-03-07
AI Technical Summary
Due to the fixed test cases and the diversity of test requirements, existing model testing methods cannot guarantee adaptability, resulting in large errors in model accuracy measurement results and the accuracy of the model cannot be accurately measured.
By determining the set of cases to be tested based on test needs and optimizing the test cases to be tested based on the use case type and optimization information, a more adaptable set of test cases is built to improve the accuracy of test results.
By optimizing the test case set, the accuracy of the model can be measured more accurately, the error can be reduced, and the reliability of the test results can be improved.
Smart Images
Figure CN119576775A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of testing technology, and in particular to a method, device, electronic device and storage medium for testing model accuracy. Background Art
[0002] In the current model testing process, test cases of fixed case types are generally directly input into a specified model to determine the accuracy of the specified model based on the test results output by the specified model.
[0003] Since the test cases are relatively fixed and the test requirements are diverse, there is no guarantee that they are suitable for the specified model, resulting in the test results output by the specified model not answering the question, causing large errors in the accuracy of the measured specified model, and thus the accuracy of the specified model cannot be accurately measured. Summary of the invention
[0004] In view of the above problems, the present application provides a model accuracy testing method, device, electronic device and storage medium, which are used to improve the accuracy of the test results.
[0005] According to one aspect of the present application, a method for testing model accuracy is provided, the testing method comprising: determining a set of test cases corresponding to each use case list according to the number of use cases required by the test requirements and the number of use cases in each use case list; wherein the use case types of the test cases in the same set of test cases are the same; traversing the test cases in each set of test cases, and taking the traversed test cases as target test cases; optimizing the target test cases according to the use case types of the target test cases and the optimization information corresponding to the set of test cases to which the target test cases belong, to obtain target test cases, so as to obtain test case sets corresponding to each set of test cases; testing the specified model according to the test requirements and each set of test cases to obtain a test result.
[0006] In an optional manner, the method of determining the set of cases to be tested corresponding to each use case list according to the number of use cases required by the test requirements and the number of use cases in each use case list further includes: determining the set of cases to be tested corresponding to the target use case list according to the size relationship between the number of use cases required by the test requirements and the number of use cases in the target use case list, so as to determine the set of cases to be tested corresponding to each use case list; wherein the target use case list is any use case list in any use case file under any use case directory; if the number of use cases in the target use case list is less than or equal to the number of use cases required by the test requirements, then based on all use cases and repeated use cases in the target use case list, a set of cases to be tested corresponding to the target use case list is constructed; if the number of use cases in the target use case list is greater than the number of use cases required by the test requirements, then the number of use cases required by the test requirements is selected from the target use case list to construct a set of cases to be tested corresponding to the target use case list.
[0007] In an optional manner, the construction of a set of cases to be tested corresponding to the target use case list based on all use cases and repeated use cases in the target use case list further includes: subtracting the number of use cases required by the test requirement from the number of use cases in the target use case list to obtain a target number of repeated use cases; if the current number of reuses of the target use case is less than a preset number of reuses, using the target use case as the repeated use case to obtain the target number of repeated use cases; wherein the target use case is any use case in the target use case list; and the construction of a set of cases to be tested corresponding to the target use case list based on all use cases in the target use case list and the target number of repeated use cases.
[0008] In an optional manner, the target test case includes a target test question and a target test answer; optimizing the target test case according to the use case type of the target test case and the optimization information corresponding to the set of test cases to which the target test case belongs to obtain the target test case, further includes: determining a target optimization method according to the use case type of the target test case; determining the target optimization information corresponding to the use case type of the target test case from the optimization information corresponding to the set of test cases to which the target test case belongs; optimizing the target answer based on the target optimization method, and optimizing the target test question based on the target optimization information to obtain the target test case.
[0009] In an optional manner, the test requirement is a requirement for performing accuracy testing on a specified model for different test fields; the testing of the specified model according to the test requirement and each test case set to obtain a test result further includes: mixing a test case set belonging to the same test field with a test case set in a general field in a preset ratio to obtain a plurality of mixed test case sets; and testing the specified model based on the plurality of mixed test case sets to obtain a test result characterizing the accuracy of the specified model in different test fields.
[0010] In an optional manner, the test requirement is a requirement for performing accuracy testing on specified models for a target test domain, and the number of the specified models is multiple; the specified models are tested according to the test requirement and each test case set to obtain test results, further comprising: taking a test case set that matches the test domain with the target test domain as a target test case set; and testing each specified model based on the target test case set to obtain test results that characterize the accuracy of each specified model in the target test domain.
[0011] In an optional manner, the testing of each designated model based on the target test case set to obtain a test result characterizing the accuracy of each designated model in the target test domain further includes: inputting preset test questions in the target test case set into each designated model to obtain a test answer output by each designated model; wherein each preset test question in the target test case set corresponds to a preset test answer; and performing similarity matching between the test answer output by each designated model and the corresponding preset test answer to obtain the accuracy of each designated model in the target test domain.
[0012] According to another aspect of the present application, a testing device for model accuracy is provided, the testing device comprising: a determination module, for determining a set of test cases corresponding to each use case list according to the number of use cases required by the test requirements and the number of use cases in each use case list; wherein the use case types of the test cases in the same set of test cases are the same; a test case traversal module, for traversing the test cases in each set of test cases, and taking the traversed test cases as target test cases; an optimization module, for optimizing the target test cases according to the use case types of the target test cases and the optimization information corresponding to the set of test cases to which the target test cases belong, to obtain target test cases, so as to obtain test case sets corresponding to each set of test cases; and a testing module, for testing a specified model according to the test requirements and each set of test cases, to obtain a test result.
[0013] According to one aspect of the present application, an electronic device is provided, including: a controller; and a memory for storing one or more programs, which, when the one or more programs are executed by the controller, perform the above-mentioned test method.
[0014] According to one aspect of the present application, a computer-readable storage medium is also provided, on which computer-readable instructions are stored. When the computer-readable instructions are executed by a processor of a computer, the computer executes the above-mentioned test method.
[0015] According to one aspect of the present application, a computer program product or a computer program is also provided, the computer program product or the computer program comprising computer instructions, the computer instructions being stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device performs the above-mentioned test method.
[0016] The present application selects the number of test cases to be tested that meets the test requirements from the use case list, and optimizes them according to the use case type of each test case and the optimization information corresponding to the test case set to which each test case belongs, so that the constructed test case set is more suitable for the test of the specified model, so that the test results can accurately represent the accuracy of the specified model.
[0017] The above description is only an overview of the technical solution of the embodiment of the present application. In order to more clearly understand the technical means of the present application, it can be implemented in accordance with the contents of the specification. In order to make the above and other purposes, features and advantages of the present application more obvious and easy to understand, the specific implementation methods of the present application are listed below. BRIEF DESCRIPTION OF THE DRAWINGS
[0018] The drawings herein are incorporated into the specification and constitute a part of the specification, illustrate embodiments consistent with the present application, and are used together with the specification to explain the principles of the present application. Obviously, the drawings described below are only some embodiments of the present application, and for ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.
[0019] Figure 1 It is a flowchart of a method for testing model accuracy shown in an exemplary embodiment of the present application.
[0020] Figure 2 is based on Figure 1 The exemplary embodiment shown is a flow chart of another method for testing model accuracy.
[0021] Figure 3 is based on Figure 1The exemplary embodiment shown is a flow chart of another method for testing model accuracy.
[0022] Figure 4 is based on Figure 1 The exemplary embodiment shown is a flow chart of another method for testing model accuracy.
[0023] Figure 5 is based on Figure 1 The exemplary embodiment shown is a flow chart of another method for testing model accuracy.
[0024] Figure 6 It is a schematic diagram of the structure of a testing device for model accuracy shown in an exemplary embodiment of the present application.
[0025] Figure 7 It is a structural diagram of a computer system of an electronic device shown in an exemplary embodiment of the present application. DETAILED DESCRIPTION
[0026] Here, exemplary embodiments will be described in detail, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The implementations described in the following exemplary embodiments do not represent all implementations consistent with the present application. Instead, they are only examples of devices and methods consistent with some aspects of the present application as detailed in the attached claims.
[0027] The block diagrams shown in the accompanying drawings are merely functional entities and do not necessarily correspond to physically independent entities. That is, these functional entities may be implemented in software form, or in one or more hardware modules or integrated circuits, or in different networks and / or processor devices and / or microcontroller devices.
[0028] The flowcharts shown in the accompanying drawings are only exemplary and do not necessarily include all the contents and operations / steps, nor must they be executed in the order described. For example, some operations / steps can be decomposed, and some operations / steps can be combined or partially combined, so the actual execution order may change according to actual conditions.
[0029] The term "multiple" as used in this application refers to two or more than two. "And / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone. The character " / " generally indicates that the related objects are in an "or" relationship.
[0030] Currently, test cases of fixed case types are generally directly input into a specified model to determine the accuracy of the specified model based on the test results output by the specified model.
[0031] Since the test cases are relatively fixed and the test requirements are diverse, there is no guarantee that they are suitable for the specified model, resulting in the test results output by the specified model not answering the question, causing large errors in the accuracy of the measured specified model, and thus the accuracy of the specified model cannot be accurately measured.
[0032] To this end, one aspect of the present application provides a method for testing model accuracy. Figure 1 , Figure 1 1 is a flow chart of a method for testing model accuracy shown in an exemplary embodiment of the present application. The testing method at least includes S110 to S140, which are described in detail as follows:
[0033] S110: Determine a set of to-be-tested cases corresponding to each use case list according to the number of use cases required by the test requirements and the number of use cases in each use case list; wherein the to-be-tested cases in the same set of to-be-tested cases have the same use case type.
[0034] Test requirements include not only the number of test cases required for testing, but also the test objects (specified models), test types, etc.
[0035] The use case types include but are not limited to multiple choice question types, fill-in-the-blank question types, true-or-false question types, etc. The use cases to be tested include preset test questions and preset test answers.
[0036] Each use case directory stores a corresponding use case file, and each use case file stores a corresponding use case list. The use case list records the basic use cases to be tested, and the use case types of the use cases to be tested in each use case list are the same. The number of use cases in each use case list may be different or the same, and this embodiment does not limit it.
[0037] This step combines the number of use cases required by the test requirements with the number of use cases recorded in each use case list, and selects the use cases to be tested from each use case list to obtain a set of use cases to be tested corresponding to each use case list. The use case type of the set of use cases to be tested is the same as the use case type of the corresponding use case list.
[0038] S120: traverse the test cases in each test case set, and use the traversed test cases as target test cases.
[0039] By traversing each set of test cases and each test case, it is ensured that all test cases are optimized without omission, so as to accurately determine the test case set corresponding to each set of test cases.
[0040] S130: Optimize the target case to be tested according to the case type of the target case to be tested and the optimization information corresponding to the case set to be tested to which the target case to be tested belongs, to obtain the target test case, so as to obtain the test case set corresponding to each case set to be tested.
[0041] Exemplarily, the use case type of the traversed second test case is a fill-in-the-blank question type, and the optimization information of the corresponding fill-in-the-blank question type is selected from the optimization information corresponding to the set of test cases to which the second test case belongs to optimize the second test case, thereby obtaining the optimized second test case, i.e., the target test case, and so on, to complete the optimization of each set of test cases, and use the optimized set of test cases as the corresponding test case set.
[0042] S140: Testing the specified model according to the test requirements and each test case set to obtain the test results.
[0043] The test requirements include expected test results, that is, the test results expected to characterize the accuracy of the specified model in the relevant test process.
[0044] This embodiment selects test cases of the number required to meet the test requirements from the use case list, and optimizes them according to the use case type of each test case and the optimization information corresponding to the test case set to which each test case belongs, so that the constructed test case set is more suitable for the test of the specified model, so that the test results can accurately represent the accuracy of the specified model.
[0045] In another exemplary embodiment of the present application, it is described in detail how to determine the set of test cases corresponding to each use case list according to the number of use cases required by the test requirements and the number of use cases in each use case list. For details, please refer to Figure 2 , Figure 2 is based on Figure 1 The exemplary embodiment shown is a flow chart of another method for testing the accuracy of a model. Figure 1 The S110 shown further includes S210 to S230, which are described in detail as follows:
[0046] S210: Determine the set of cases to be tested corresponding to the target use case list according to the relationship between the number of use cases required by the test requirements and the number of use cases in the target use case list, so as to determine the set of cases to be tested corresponding to each use case list; wherein the target use case list is any use case list in any use case file under any use case directory.
[0047] The test cases are selected for each use case list respectively, and the selection method of the test cases is determined mainly based on the relationship between the number of use cases required by the test requirements and the number of use cases in each use case list.
[0048] S220: If the number of use cases in the target use case list is less than or equal to the number of use cases required by the test requirements, a set of to-be-tested use cases corresponding to the target use case list is constructed based on all use cases and repeated use cases in the target use case list.
[0049] For example, the number of use cases in the target use case list is 10, and the number of use cases required for the test requirements is 20. Obviously, there are not enough use cases in the target use case list, so it is necessary to reuse the relevant use cases in the target use case list. The 10 use cases in the target use case list and the reused duplicate use cases are used to construct the set of use cases to be tested corresponding to the target use case list.
[0050] Specifically, the number of use cases required by the test requirements is subtracted from the number of use cases in the target use case list to obtain the target number of repeated use cases; if the current number of reuses of the target use case is less than the preset number of reuses, the target use case is used as a repeated use case to obtain the target number of repeated use cases; wherein the target use case is any use case in the target use case list; based on all the use cases in the target use case list and the target number of repeated use cases, a set of use cases to be tested corresponding to the target use case list is constructed.
[0051] Among them, the preset reuse times corresponding to each use case may be different. The preset reuse times represents the number of times the same use case can be repeated as a repeated use case. Each time a use case is reused, its reuse times decreases by one until its reuse times are less than the preset reuse times, which indicates that it can no longer be selected as a repeated use case into the set of use cases to be tested.
[0052] S230: If the number of use cases in the target use case list is greater than the number of use cases required by the test requirement, select the number of use cases required by the test requirement from the target use case list to construct a set of to-be-tested use cases corresponding to the target use case list.
[0053] For example, the number of use cases in the target use case list is 10, and the number of use cases required by the test requirements is 8. Then, 8 use cases can be directly selected from the target use case list (selected randomly or in sequence according to the order of the list) to construct a set of use cases to be tested corresponding to the target use case list.
[0054] This embodiment determines the selection method of the corresponding test cases through simple comparison of quantity, so as to accurately construct the test case sets corresponding to each test case list under the premise of meeting the test requirements.
[0055] In the preferred embodiment, the process of determining the set of test cases to be tested is further described as follows:
[0056] According to the "original prompt input file" information configured in the configuration file (file that represents the test requirements), obtain the data (use cases) under all excel files / sheets (use case lists) under the use case directory / use case files; wherein, each use case directory stores at least one use case file, and each use case file stores at least one use case list.
[0057] Get the number of prompts executed in the configuration file (N) (that is, the number of test cases required for the test requirements)
[0058] Traverse the directory / file / sheet, and when reading data from each sheet (the maximum number of valid rows of a sheet is M), randomly generate a non-repeating random number random_list. (In order to have a different prompt for each evaluation)
[0059] a) If M<=N, random_list is a natural number sequence from 1 to M (i.e., all use cases in the target use case list are selected, and use cases are repeatedly selected from 1 to M).
[0060] b) If M>N, random_list is N non-repeating positive integers in the range of 1-M (i.e., the number of use cases required for the test requirements is selected from the target use case list).
[0061] c) Random_list is arranged in order from small to large (sorting the test cases in the test case set)
[0062] In another exemplary embodiment of the present application, it is described in detail how to optimize the target test case according to the test case type of the target test case and the optimization information corresponding to the test case set to which the target test case belongs to obtain the target test case. For details, please refer to Figure 3 , Figure 3 is based on Figure 1 The exemplary embodiment shown is a flow chart of another method for testing the accuracy of a model. Figure 1 The S130 shown further includes S310 to S330; wherein the target test case includes a target test question and a target test answer, which are described in detail as follows:
[0063] S310: Determine a target optimization method according to the use case type of the target use case to be tested.
[0064] Different optimization methods correspond to different use case types, and each optimization method includes an optimization sub-method for treating test questions and / or an optimization sub-method for treating test answers.
[0065] S320: Determine target optimization information corresponding to the use case type of the target use case to be tested from the optimization information corresponding to the use case set to be tested to which the target use case to be tested belongs.
[0066] The set of test cases corresponds to the optimization information of all test case types. In order to optimize the test cases of different use case types in a targeted manner, it is necessary to consider their use case types, that is, to match the corresponding target optimization information through the use case types.
[0067] S330: Optimizing the target answer to be tested based on the target optimization method, and optimizing the target question to be tested based on the target optimization information to obtain a target test case.
[0068] For example, true or false question type: supplement the information of the test question to optimize the test question, and optimize the answer to the test through regular expressions, alias optimization, etc. Fill-in-the-blank question type: supplement the information of the test question to optimize the test question, and optimize the answer to the test through separator optimization, etc. Multiple-choice question type: supplement the information of the test question to optimize the test question, and optimize the answer to the test through regular expressions, increase or decrease the text before and after, etc.
[0069] This embodiment optimizes the test questions and the test answers respectively, and takes into account the use case types of the test cases to adaptively select optimization methods to complete the optimization of each test case, thereby completing the optimization of each set of test cases to obtain a corresponding test case set.
[0070] In another exemplary embodiment of the present application, it is described in detail how to test a specified model according to the test requirements and each test case set to obtain the test results. Figure 4 , Figure 4 is based on Figure 1 The exemplary embodiment shown is a flow chart of another method for testing the accuracy of a model. Figure 1 The S140 shown further includes S410 to S420; wherein the test requirement is a requirement for accuracy testing of a specified model for a target test field, and the number of specified models is multiple, as described in detail as follows:
[0071] S410: Taking a test case set whose test domain matches the target test domain as a target test case set.
[0072] S420: Testing each designated model based on the target test case set to obtain a test result representing the accuracy of each designated model in the target test domain.
[0073] Each specified model corresponds to a test result, which includes but is not limited to the sub-test results of each test case, the accuracy scores corresponding to each sub-test result, the average score, median, variance, mode, etc. of the sub-test results, so as to intuitively and meticulously display the accuracy of each specified model in the target test field.
[0074] Exemplarily, all threads in thread_list are started (in some embodiments, multiple threads may be used to test each specified model at the same time):
[0075] a) Traverse the prompt data in each sheet.
[0076] b) When the current prompt is in the random_list, use the prompt as the basic prompt for the interface call.
[0077] c) According to the prompt type (single-choice question type, true-or-false question type, fill-in-the-blank question type) in the "original prompt input file" and the prompt optimization information in the configuration file, optimize the basic prompt in step b to obtain the final requested prompt (i.e., the target test case).
[0078] d) Make interface calls according to the number of prompt calls configured in the configuration file:
[0079] i. Replace the current call model, and the prompt of the corresponding parameter in the configuration file is the prompt of the final request
[0080] ii. Initiate an interface call
[0081] iii. After getting the interface return, make model judgment based on the answers in the test set
[0082] 1. Verify the results according to the single-choice question type, true-or-false question type, and fill-in-the-blank question type
[0083] 2. Record the scores of the current model, domain, and question type. The following shows the demo_gpt model. Two test sets were executed on the science and technology multiple-choice questions. [0,1] corresponds to failure and pass:
[0084] {'demo_gpt:{'Technology single choice':[0,1],'Technology judgment':[1,1],'Technology fill-in-the-blank':[0,0]}}
[0085] 3. After recording, output the results according to the configured "test result output file" (it can be output by model or to one file. It is recommended to output by model for manual review)
[0086] In another exemplary embodiment, the preset test questions in the target test case set are input into each specified model to obtain the test answers output by each specified model; wherein each preset test question in the target test case set corresponds to a preset test answer; the test answers output by each specified model are similarly matched with the corresponding preset test answers to obtain the accuracy of each specified model in the target test field.
[0087] The higher the matching degree of the test answer, the higher the accuracy of the specified model. By establishing a connection between the matching degree of the test answer and the accuracy of the model test result, the accuracy of each specified model in the target test field can be quickly determined.
[0088] This embodiment determines a target test case set through a target test domain in the test requirements, and then uses the set to test multiple specified models separately, or uses the same thread to test each specified model step by step to obtain the accuracy of each specified model in the target test domain.
[0089] In the process of domain testing, if only test cases of a single test domain are used for testing, the testing ability of the specified model in the general domain will be degraded and forgotten. To this end, in another exemplary embodiment of the present application, another method of obtaining test results is described in detail. For details, please refer to Figure 5 , Figure 5 is based on Figure 1 The exemplary embodiment shown is a flow chart of another method for testing the accuracy of a model. Figure 1 The S140 shown further includes S510 to S520; wherein the test requirement is a requirement for performing accuracy testing on a specified model in different test fields, which is described in detail as follows:
[0090] S510: Mix the test case sets belonging to the same test field with the test case sets in the general field according to a preset ratio to obtain multiple mixed test case sets.
[0091] Test cases in a general domain are test cases that characterize non-domain characteristics, that is, they can overlap with test cases in various domains.
[0092] This embodiment introduces test cases in general fields and test cases in related test fields in a preset ratio to obtain a set of mixed test cases corresponding to each test field. While ensuring that the general capabilities of the specified model are not degraded, its accuracy in different test fields is more reliable.
[0093] As shown in Table 1, by mixing the test cases in the general field and the professional test field in different proportions, there are differences in the accuracy of the test. Through corresponding test verification, the appropriate mixing ratio can be determined.
[0094] Table 1:
[0095]
[0096] S520: Testing the specified model based on multiple mixed test case sets to obtain test results that characterize the accuracy of the specified model in different test fields.
[0097] Each test field corresponds to a test result, which includes but is not limited to the sub-test results of each test case, the accuracy scores corresponding to each sub-test result, the average score, median, variance, mode, etc. of the sub-test results, so as to intuitively and meticulously display the accuracy performance of the specified model in different test fields.
[0098] Another aspect of the present application also provides a device for testing the accuracy of a model, such as Figure 6 As shown, Figure 6 It is a schematic diagram of the structure of a testing device for model accuracy shown in an exemplary embodiment of the present application.
[0099] The testing device 600 comprises:
[0100] The determination module 610 is used to determine the set of to-be-tested cases corresponding to each use case list according to the number of use cases required by the test requirements and the number of use cases in each use case list; wherein the use case types of the to-be-tested cases in the same set of to-be-tested cases are the same;
[0101] A test case traversal module 630 is used to traverse the test cases in each test case set and use the traversed test cases as target test cases;
[0102] The optimization module 650 is used to optimize the target to-be-tested case according to the case type of the target to-be-tested case and the optimization information corresponding to the to-be-tested case set to which the target to-be-tested case belongs, to obtain the target test case, so as to obtain the test case set corresponding to each to-be-tested case set;
[0103] The testing module 670 is used to test the specified model according to the test requirements and various test case sets to obtain the test results.
[0104] In another exemplary embodiment, the determination module 610 further includes:
[0105] A determination unit is used to determine the set of to-be-tested cases corresponding to the target use case list according to the size relationship between the number of use cases required by the test requirements and the number of use cases in the target use case list, so as to determine the set of to-be-tested cases corresponding to each use case list; wherein the target use case list is any use case list in any use case file under any use case directory;
[0106] The first determination subunit is used to construct a set of to-be-tested cases corresponding to the target use case list based on all the use cases and repeated use cases in the target use case list if the number of use cases in the target use case list is less than or equal to the number of use cases required by the test requirements;
[0107] The second determination subunit is used to select the number of use cases required by the test requirements from the target use case list if the number of use cases in the target use case list is greater than the number of use cases required by the test requirements, and construct a set of use cases to be tested corresponding to the target use case list.
[0108] In another exemplary embodiment, the first determining subunit further includes:
[0109] The calculation section is used to perform a subtraction operation on the number of use cases required by the test requirements and the number of use cases in the target use case list to obtain the target number of repeated use cases;
[0110] The first section is used to treat the target use case as a repeated use case if the current reuse count of the target use case is less than the preset reuse count, so as to obtain a target number of repeated use cases; wherein the target use case is any use case in the target use case list;
[0111] The second section is used to construct a set of test cases corresponding to the target use case list based on all the use cases in the target use case list and the target number of repeated use cases.
[0112] In another exemplary embodiment, the target test case includes a target test question and a target test answer; the optimization module 650 further includes:
[0113] An optimization method unit, used to determine a target optimization method according to the use case type of the target test case;
[0114] An optimization information determination unit, configured to determine target optimization information corresponding to the use case type of the target use case to be tested from the optimization information corresponding to the use case set to be tested to which the target use case to be tested belongs;
[0115] The case optimization unit is used to optimize the target answer to be tested based on the target optimization method, and to optimize the target question to be tested based on the target optimization information to obtain the target test case.
[0116] In another exemplary embodiment, the test requirement is to characterize the requirement of performing accuracy test on a specified model for different test fields; the test module 670 further includes:
[0117] A mixing unit, used to mix the test case sets belonging to the same test field with the test case sets of the general field according to a preset ratio to obtain multiple mixed test case sets;
[0118] The hybrid test unit is used to test the specified model based on multiple hybrid test case sets to obtain test results that characterize the accuracy of the specified model in different test fields.
[0119] In another exemplary embodiment, the test requirement is a requirement for performing accuracy testing on a specified model for a target test domain, and the number of the specified models is multiple; the test module 670 further includes:
[0120] A matching unit, used for taking a test case set that matches a test domain with a target test domain as a target test case set;
[0121] The target test unit is used to test each specified model based on a target test case set to obtain a test result that characterizes the accuracy of each specified model in the target test field.
[0122] In another exemplary embodiment, the target test unit further comprises:
[0123] The test answer section is used to input the preset test questions in the target test case set into each specified model to obtain the test answers output by each specified model; wherein each preset test question in the target test case set corresponds to a preset test answer;
[0124] The target test section is used to perform similarity matching between the test answers output by each specified model and the corresponding preset test answers to obtain the accuracy of each specified model in the target test field.
[0125] The testing device of the present application selects the number of test cases to be tested that meets the test requirements from the use case list, and optimizes the test cases according to the use case type of each test case and the optimization information corresponding to the test case set to which each test case belongs, so that the constructed test case set is more suitable for the test of the specified model, so that the test results can accurately represent the accuracy of the specified model.
[0126] It should be noted that the testing device provided in the above embodiment and the testing method provided in the above embodiment belong to the same concept, wherein the specific manner in which each module and unit performs operations has been described in detail in the method embodiment and will not be repeated here.
[0127] Another aspect of the present application provides an electronic device, including: a controller; a memory for storing one or more programs, and when the one or more programs are executed by the controller, the above-mentioned test method is executed.
[0128] See also Figure 7 , Figure 7 1 is a schematic diagram of the structure of a computer system of an electronic device shown in an exemplary embodiment of the present application, which shows a schematic diagram of the structure of a computer system of an electronic device suitable for implementing an embodiment of the present application.
[0129] It should be noted that Figure 7 The computer system 700 of the electronic device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.
[0130] like Figure 7 As shown, the computer system 700 includes a central processing unit (CPU) 701, which can perform various appropriate actions and processes according to the program stored in the read-only memory (ROM) 702 or the program loaded from the storage part 708 to the random access memory (RAM) 703, such as executing the method in the above embodiment. In the RAM 703, various programs and data required for system operation are also stored. The CPU 701, ROM 702 and RAM 703 are connected to each other through a bus 704. An input / output (I / O) interface 705 is also connected to the bus 704.
[0131] The following components are connected to the I / O interface 705: an input section 706 including a keyboard, a mouse, etc.; an output section 707 including a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 708 including a hard disk, etc.; and a communication section 709 including a network interface card such as a LAN (Local Area Network) card, a modem, etc. The communication section 709 performs communication processing via a network such as the Internet. A drive 710 is also connected to the I / O interface 705 as needed. A removable medium 711, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 710 as needed so that a computer program read therefrom is installed into the storage section 708 as needed.
[0132] In particular, according to an embodiment of the present application, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present application includes a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes a computer program for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network through a communication section 709, and / or installed from a removable medium 711. When the computer program is executed by a central processing unit (CPU) 701, various functions defined in the system of the present application are executed.
[0133] It should be noted that the computer-readable medium shown in the embodiment of the present application can be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium can be, for example, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media can include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM), a flash memory, an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present application, a computer-readable storage medium can be any tangible medium containing or storing a program, which can be used by an instruction execution system, device or device or used in combination with it. In the present application, a computer-readable signal medium can include a data signal propagated in a baseband or as a part of a carrier wave, wherein a computer-readable computer program is carried. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. Computer-readable signal media may also be any computer-readable medium other than computer-readable storage media, which may send, propagate, or transmit programs for use by or in conjunction with an instruction execution system, apparatus, or device. The computer program contained on the computer-readable medium may be transmitted using any appropriate medium, including but not limited to: wireless, wired, etc., or any suitable combination of the above.
[0134] The flowchart and block diagram in the accompanying drawings illustrate the possible architecture, functions and operations of the system, method and computer program product according to various embodiments of the present application. Wherein, each box in the flowchart or block diagram can represent a module, a program segment, or a part of the code, and the above-mentioned module, program segment, or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order than the order marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram or flowchart, and the combination of boxes in the block diagram or flowchart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.
[0135] The units involved in the embodiments described in this application may be implemented by software or hardware, and the units described may also be set in a processor. The names of these units do not, in some cases, constitute limitations on the units themselves.
[0136] Another aspect of the present application further provides a computer-readable storage medium on which a computer program is stored, and when the computer program is executed by a processor, the above-mentioned test method is implemented. The computer-readable storage medium may be included in the electronic device described in the above embodiment, or may exist independently without being assembled into the electronic device.
[0137] Another aspect of the present application also provides a computer program product or a computer program, which includes a computer instruction stored in a computer-readable storage medium. A processor of a computer device reads the computer instruction from the computer-readable storage medium, and the processor executes the computer instruction, so that the computer device executes the test method provided in each of the above embodiments.
[0138] According to one aspect of an embodiment of the present application, a computer system is also provided, including a central processing unit (CPU), which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) or a program loaded from a storage portion into a random access memory (RAM), such as executing the method in the above embodiment. In RAM, various programs and data required for system operation are also stored. CPU, ROM and RAM are connected to each other via a bus. An input / output (I / O) interface is also connected to the bus.
[0139] The following components are connected to the I / O interface: an input part including a keyboard, a mouse, etc.; an output part including a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker; a storage part including a hard disk, etc.; and a communication part including a network interface card such as a LAN (Local Area Network) card, a modem, etc. The communication part performs communication processing via a network such as the Internet. A drive is also connected to the I / O interface as needed. Removable media, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., are installed on the drive as needed so that the computer program read therefrom is installed into the storage part as needed.
[0140] The above content is only a preferred exemplary embodiment of the present application and is not intended to limit the implementation scheme of the present application. A person skilled in the art can easily make corresponding changes or modifications based on the main concept and spirit of the present application. Therefore, the scope of protection of the present application shall be based on the scope of protection required by the claims.
Claims
1. A method for testing model accuracy, characterized in that: The test method includes: According to the number of use cases required by the test requirements and the number of use cases in each use case list, determine the set of use cases to be tested corresponding to each use case list; wherein the use case types of the use cases to be tested in the same set of use cases to be tested are the same; Traverse the test cases in each test case set, and use the traversed test cases as target test cases; According to the use case type of the target use case to be tested and the optimization information corresponding to the set of use cases to be tested to which the target use case to be tested belongs, the target use case to be tested is optimized to obtain a target test case, so as to obtain a test case set corresponding to each set of use cases to be tested; The specified model is tested according to the test requirements and each test case set to obtain the test results.
2. The testing method according to claim 1, characterized in that: The step of determining the set of to-be-tested cases corresponding to each use case list according to the number of use cases required by the test requirements and the number of use cases in each use case list further includes: According to the size relationship between the number of use cases required by the test requirements and the number of use cases in the target use case list, determine the set of use cases to be tested corresponding to the target use case list, so as to determine the set of use cases to be tested corresponding to each use case list; wherein the target use case list is any use case list in any use case file under any use case directory; If the number of use cases in the target use case list is less than or equal to the number of use cases required by the test requirement, then construct a set of use cases to be tested corresponding to the target use case list based on all use cases and repeated use cases in the target use case list; If the number of use cases in the target use case list is greater than the number of use cases required by the test requirement, then the number of use cases required by the test requirement is selected from the target use case list to construct a set of use cases to be tested corresponding to the target use case list.
3. The testing method according to claim 2, characterized in that: The step of constructing a set of test cases corresponding to the target use case list based on all the use cases and repeated use cases in the target use case list further includes: Subtract the number of use cases required by the test requirement from the number of use cases in the target use case list to obtain a target number of repeated use cases; If the current reuse count of the target use case is less than the preset reuse count, the target use case is used as the repeated use case to obtain the target number of repeated use cases; wherein the target use case is any use case in the target use case list; Based on all the use cases in the target use case list and the target number of repeated use cases, a set of use cases to be tested corresponding to the target use case list is constructed.
4. The testing method according to claim 1, characterized in that: The target test case includes a target test question and a target test answer; The step of optimizing the target test case according to the test case type of the target test case and the optimization information corresponding to the test case set to which the target test case belongs, to obtain the target test case, further includes: Determining a target optimization method according to the use case type of the target use case to be tested; Determining target optimization information corresponding to the use case type of the target use case to be tested from the optimization information corresponding to the set of use cases to be tested to which the target use case to be tested belongs; The target answer to be tested is optimized based on the target optimization method, and the target question to be tested is optimized based on the target optimization information to obtain a target test case.
5. The testing method according to any one of claims 1 to 4, characterized in that: The test requirements represent the requirements for accuracy testing of a specified model for different test fields; The testing of the specified model according to the test requirements and each test case set to obtain the test result further includes: The test case sets belonging to the same test field are mixed with the test case sets in the general field according to a preset ratio to obtain multiple mixed test case sets; The specified model is tested based on the multiple mixed test case sets to obtain test results that characterize the accuracy of the specified model in different test fields.
6. The testing method according to any one of claims 1 to 4, characterized in that: The test requirement represents the requirement for accuracy testing of a specified model in a target test field, and the number of the specified models is multiple; The testing of the specified model according to the test requirements and each test case set to obtain the test result further includes: Taking a test case set whose test domain matches the target test domain as a target test case set; Each designated model is tested based on the target test case set to obtain a test result representing the accuracy of each designated model in the target test domain.
7. The testing method according to claim 6, characterized in that: The step of testing each designated model based on the target test case set to obtain a test result characterizing the accuracy of each designated model in the target test domain further includes: Input the preset test questions in the target test case set into each specified model to obtain the test answers output by each specified model; wherein each preset test question in the target test case set corresponds to a preset test answer; The test answers output by each specified model are matched with the corresponding preset test answers for similarity to obtain the accuracy of each specified model in the target test field.
8. A device for testing model accuracy, characterized in that: The testing device comprises: A determination module is used to determine the set of test cases corresponding to each use case list according to the number of use cases required by the test requirements and the number of use cases in each use case list; wherein the use case types of the test cases in the same set of test cases are the same; A test case traversal module is used to traverse the test cases in each test case set and use the traversed test cases as target test cases; An optimization module, configured to optimize the target case to be tested according to the case type of the target case to be tested and the optimization information corresponding to the set of cases to be tested to which the target case to be tested belongs, to obtain a target test case, so as to obtain a test case set corresponding to each set of cases to be tested; The test module is used to test the specified model according to the test requirements and various test case sets to obtain test results.
9. An electronic device, characterized in that: include: Controller; A memory for storing one or more programs, which, when executed by a controller, enables the controller to implement the test method according to any one of claims 1 to 7.
10. A computer-readable storage medium, characterized in that: Computer-readable instructions are stored thereon, and when the computer-readable instructions are executed by a processor of a computer, the computer is caused to execute the testing method according to any one of claims 1 to 7.