A method and device for automatically generating business test cases by fuzzy testing of a large model

Abstract

The application provides a large model automatic generation business test case fuzzy test method and device, the method comprises the following steps: uploading a test target field to a fuzzy test system; uploading the interface document corresponding to the test target field to the fuzzy test system; generating an interface call process and each parameter data in each interface through a fuzzy test engine; calling the test target field service according to the interface call process and the parameter data; collecting the coverage data of the test target field service under the execution condition of the test case; judging whether a new path is generated in the coverage compared with the previous test case according to the coverage data; judging whether the test of the test target field is sufficient; when a new path is generated or a new path is not generated and the test is not completed, feeding back the interface call process coverage to the large model and generating a new interface call process again. The large model automatic generation business test case fuzzy test method and device provided by the application effectively improve the coverage of fuzzy test.

Description

TECHNICAL FIELD

[0001] The embodiment of the application relates to the technical field of network information security, in particular to a fuzzy test method and device for automatically generating business test cases of a large model. BACKGROUND

[0002] In the prior art, when the fuzzy test method is used for system-level testing, the running coverage of most codes is low, and the fuzzy test method cannot improve the coverage in a short time. There are problems of low test coverage of a tested target under the condition of sufficient test time, and the fuzzy test cannot cover the test cases containing test calling sequences.

[0003] Therefore, it is necessary to provide a fuzzy test method and device for automatically generating business test cases of a large model, which can effectively solve the above problems. SUMMARY

[0004] The application provides a fuzzy test method and device for automatically generating business test cases of a large model, which effectively improve the coverage of the fuzzy test.

[0005] The embodiment of the application provides a fuzzy test method for automatically generating business test cases of a large model, which comprises the following steps:

[0006] uploading a test target field to a fuzzy test system;

[0007] uploading an interface document corresponding to the test target field to the fuzzy test system;

[0008] generating an interface calling process and parameter data in each interface by using a fuzzy test engine;

[0009] calling a test target field service according to the interface calling process and the parameter data;

[0010] collecting coverage data of the test target field service under the execution of the test case, wherein the test case comprises the interface calling process and the parameter data;

[0011] determining whether a new path is generated in coverage compared with previous test cases according to the coverage data;

[0012] determining whether the test of the test target field is sufficient;

[0013] when a new path is generated or no new path is generated and the test is not completed, feeding back the interface calling process coverage to a large model and generating a new interface calling process again.

[0014] Preferably, the interface call flow coverage feedback large model is covered and a new interface call flow is generated again by the Targeted-Gen algorithm.

[0015] Preferably, the interface call flow and the parameter data are used as an initial population, and the coverage of each interface arrangement is calculated; a standard differential evolution algorithm is used for iteration to generate a new candidate solution; the coverage of each candidate solution is calculated, and the current optimal solution is updated; the contribution of each dimension to the current optimal solution is calculated, and the dimension with higher contribution is selected as a potential bottleneck dimension; a target vector similar to the current optimal solution is generated, and part of the dimension values in the target vector are inserted into the current optimal solution to replace or modify the values of the bottleneck dimension; the coverage of the modified solution is measured, and if the coverage is improved, the current optimal solution is updated; when the preset number of iterations is reached or the coverage is no longer significantly improved, the current optimal solution is output.

[0016] Preferably, the interface call flow generated by the large model is optimized by increasing the intention question of the interface information, specifically including:

[0017] The understanding of the interface by the large model is improved through the intention question, and the interface call flow is generated in combination with the intention.

[0018] Preferably, the interface call flow generated by the large model is optimized by increasing the standardized prompt word, specifically including:

[0019] By adding the COT commonly used prompt word, the understanding of the interface by the large model is improved, and the generation result of each step of the large model is standardized.

[0020] Preferably, the interface call flow generated by the large model is optimized by increasing the json keyword to require the large model to output a json answer, thereby reducing the number of generated tokens of the large model.

[0021] Preferably, the large model includes Starchat-13b, Codellama-34b, and Mixtral-8x7b.

[0022] Preferably, the large model is evaluated by a standard test set, and the standard test set includes:

[0023] MMLU is used to evaluate the large-scale multi-task language understanding ability;

[0024] HellaS is used to evaluate the ability of the model on complex reasoning tasks;

[0025] WinoG is used to evaluate the ability of the model to solve the pronoun resolution problem;

[0026] PIOA is used to evaluate the ability of the model to optimize the context given the prompt;

[0027] Arc-e and Arc-c are used to evaluate scientific reasoning tasks representing the basic and university levels, respectively;

[0028] NQ is used to measure the accuracy and understanding ability of the model on the natural question answering task;

[0029] TriQA is used to evaluate the ability of the model on the cross-language question answering task;

[0030] HumanE is used to evaluate the ability of the model on the human common sense reasoning task;

[0031] MBPP is used to evaluate the ability of the model on the programming problem solving task;

[0032] Math is used to evaluate the ability of the model on the mathematical problem solving task;

[0033] GSM8K is used to evaluate the ability of the model on the primary school mathematics problem solving task.

[0034] Preferably, the correlation weight of the standard test set measures the algorithm formula as follows:

[0035] The comprehensive score = w1xMMLU+w2xHellaS+w3xWinoG+w4xPIOA+w5xArc-e+w6xArc-c+w7xNQ+w8xTriQA+w9xHumanE+w10xMBPP+w11xMath+w12xGSM8K

[0036] Wherein, the weight values of w7, w8, w9 are greater than w1, w2, w3, w4, w5, w6, w10, w11, w12.

[0037] The embodiment of the application also provides a fuzzy test device for automatically generating business test cases of a large model, and the device comprises:

[0038] A test target field uploading module is configured to upload a test target field to a fuzzy test system;

[0039] An interface document uploading module is configured to upload an interface document corresponding to the test target field to the fuzzy test system;

[0040] An interface calling flow generation module is configured to generate an interface calling flow and parameter data in each interface through a fuzzy test engine;

[0041] A test target field service calling module is configured to call a test target field service according to the interface calling flow and the parameter data;

[0042] a coverage data collection module configured to collect coverage data of the test target service in execution of the test case, the test case including the interface call flow and the parameter data;

[0043] a new path generation judgment module configured to judge whether a new path is generated in coverage compared with a previous test case according to the coverage data;

[0044] a test sufficiency judgment module configured to judge whether the test on the test target is sufficient;

[0045] a new interface call flow generation module configured to feed back the interface call flow coverage situation to a large model and cyclically generate a new interface call flow again when a new path is generated or a new path is not generated and the test is not completed.

[0046] Compared with the prior art, the technical scheme of the embodiment of the present application has the following beneficial effects:

[0047] The fuzzy test method and device for automatically generating a business test case by a large model provided by the embodiment of the present application, the method comprising: uploading a test target into a fuzzy test system; uploading an interface document corresponding to the test target into the fuzzy test system; generating an interface call flow and parameter data in each interface by a fuzzy test engine; calling a test target service according to the interface call flow and the parameter data; collecting coverage data of the test target service in execution of a test case, the test case including the interface call flow and the parameter data; judging whether a new path is generated in coverage compared with a previous test case according to the coverage data; judging whether the test on the test target is sufficient; feeding back the interface call flow coverage situation to a large model and cyclically generating a new interface call flow again when a new path is generated or a new path is not generated and the test is not completed, thereby effectively improving each coverage rate of the fuzzy test. BRIEF DESCRIPTION OF DRAWINGS

[0048] In order to more clearly illustrate the technical scheme in the embodiments of the present application or the prior art, the following will briefly introduce the drawings needed to be used in the embodiments or the prior art description. Obviously, the drawings in the following description are some embodiments of the present application, not all embodiments. For those skilled in the art, other drawings can also be obtained without creative labor.

[0049] Figure 1 The flowchart of the fuzzy test method for automatically generating a business test case by a large model provided by an embodiment of the present application;

[0050] Figure 2A module schematic diagram of the fuzzy test device for automatically generating a business test case by a large model is provided for an embodiment of the present application. DETAILED DESCRIPTION

[0051] To make the objects, technical solutions and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be described below in connection with the drawings of the embodiments of the present application. Obviously, the described embodiments are only some but not all of the embodiments of the present application. Based on the embodiments of the present application, all other embodiments obtained by a person of ordinary skill in the art without creative work fall within the protection scope of the present application.

[0052] The technical solutions of the present application will be described in detail below with specific embodiments. The following specific embodiments can be combined with each other, and some embodiments can not be described again for the same or similar concepts or processes.

[0053] Based on the problems in the prior art, the present application provides a fuzzy test method and device for automatically generating a business test case by a large model, which effectively improves various coverage rates of the fuzzy test.

[0054] Figure 1 A flowchart of the fuzzy test method for automatically generating a business test case by a large model is provided for an embodiment of the present application. Now referring to Figure 1 The present application provides a fuzzy test method for automatically generating a business test case by a large model, which comprises the following steps.

[0055] Step S101: uploading a test target field to a fuzzy test system;

[0056] Step S102: uploading an interface document corresponding to the test target field to the fuzzy test system;

[0057] Step S103: generating an interface calling flow and each parameter data in each interface by a fuzzy test engine;

[0058] Step S104: calling a test target field service according to the interface calling flow and the each parameter data;

[0059] Step S105: collecting coverage rate data of the test target field service in the execution of the test case, wherein the test case comprises the interface calling flow and the each parameter data;

[0060] Step S106: determining whether a new path is generated in the coverage rate compared with a previous test case according to the coverage rate data;

[0061] Step S107: determining whether the test on the test target field is sufficient;

[0062] Step S108: When a new path is generated or no new path is generated and the test is not completed, feed back the interface call flow coverage situation to the large model and generate a new interface call flow again.

[0063] Specifically, all valid interface call flows are recorded and saved for use in fuzzy testing.

[0064] In specific implementation, the interface call flow coverage situation is fed back to the large model and a new interface call flow is generated again through a Targeted-Gen algorithm.

[0065] In specific implementation, the interface call flow and the parameter data are used as an initial population, and the individual coverage of each interface arrangement is calculated; a standard differential evolution algorithm is used for iteration to generate a new candidate solution; the coverage of each candidate solution is calculated, and the current optimal solution is updated; the contribution of each dimension to the current optimal solution is calculated, and the dimension with higher contribution is selected as a potential bottleneck dimension; a target vector similar to the current optimal solution is generated, and part of the dimension values in the target vector are inserted into the current optimal solution to replace or modify the values of the bottleneck dimension; the coverage of the modified solution is measured, and if the coverage is improved, the current optimal solution is updated; when the preset number of iterations is reached or the coverage is no longer significantly improved, the current optimal solution is output.

[0066] In specific implementation, the interface call flow generated by the large model is optimized by increasing the intention question of interface information, specifically including:

[0067] The understanding of the interface by the large model is improved through the intention question, and the interface call flow is generated in combination with the intention.

[0068] Specifically, the interface call flow is generated by directly using the interface information, which is modified to first ask the large model for the interface intention through the interface information, and then generate the interface call flow in combination with the interface intention.

[0069] In specific implementation, the interface call flow generated by the large model is optimized by increasing the standardized prompt word, specifically including:

[0070] By adding the COT commonly used prompt word, the understanding of the interface by the large model is improved, and the generation result of each step of the large model is standardized.

[0071] Specifically, the COT commonly used prompt words can be, for example, Let's think step by step. For example, first ask the intention question before generating, and limit the generation interface to understanding Let's think step by step. If we want to find test cases, we first need to know the system interface, so what's the explaination of this interface {interface_name}? Then connect the subsequent generation call process question through And then.

[0072] In specific implementation, the generation interface call process of the large model is optimized by adding a json keyword to require the large model to output a json answer, thereby reducing the number of tokens generated by the large model.

[0073] Specifically, for example, add the prompt word: Return the intention of interface in JSON format {{\"intention\":\"\"\}}. Do not generate anything outside of JSON.

[0074] In specific implementation, the large model includes Starchat-13b, Codellama-34b, and Mixtral-8x7b.

[0075] In specific implementation, the large model is evaluated by a standard test set, which includes:

[0076] MMLU is used to evaluate large-scale multi-task language understanding capabilities;

[0077] HellaS is used to evaluate the model's ability to perform complex reasoning tasks;

[0078] WinoG is used to evaluate the model's ability to solve pronoun resolution problems;

[0079] PIOA is used to evaluate the model's ability to optimize context given a prompt;

[0080] Arc-e and Arc-c are used to evaluate scientific reasoning tasks representing basic and college levels, respectively;

[0081] NQ is used to measure the accuracy and understanding of the model in natural question answering tasks;

[0082] TriQA is used to evaluate the ability of the model on the cross-language question answering task;

[0083] HumanE is used to evaluate the ability of the model on the human common sense reasoning task;

[0084] MBPP is used to evaluate the ability of the model on the programming problem solving task;

[0085] Math is used to evaluate the ability of the model on the mathematical problem solving task;

[0086] GSM8K is used to evaluate the ability of the model on the primary school mathematics problem solving task.

[0087] Specifically, the evaluation results of the three large models Starchat-13b, Codellama-34b and Mixtral-8x7b through the standard test set are as follows:

[0088] The evaluation results of the large model Mixtral-8x7b are as follows: MMLU: 70.6%; HellaS: 84.4%; WinoG: 77.2%; PIOA: 83.6%; Arc-e: 59.7%; Arc-c: 30.6%; NQ: 71.5%; TriQA: 40.2%; HumanE: 40.1%; MBPP: 60.7%; Math: 74.4%; GSM8K: 58.4%.

[0089] The evaluation results of the large model Starchat-13b are as follows: MMLU: 65.0%; HellaS: 78.0%; WinoG: 75.0%; PIOA: 80.0%; Arc-e: 55.0%; Arc-c: 28.0%; NQ: 68.0%; TriQA: 38.0%; HumanE: 37.4%; MBPP: 58.0%; Math: 70.0%; GSM8K: 55.0%.

[0090] The evaluation results of the large model Codellama-34b are as follows: MMLU: 72.0%; HellaS: 82.0%; WinoG: 78.0%; PIOA: 82.0%; Arc-e: 60.0%; Arc-c: 32.0%; NQ: 72.0%; TriQA: 42.0%; HumanE: 41.5%; MBPP: 57%; Math: 76.0%; GSM8K: 60.0%.

[0091] In specific implementation, the correlation weight of the standard test set is measured by the following formula:

[0092] The comprehensive score = w1 x MMLU + w2 x HellaS + w3 x WinoG + w4 x PIOA + w5 x Arc-e + w6 x Arc-c + w7 x NQ + w8 x TriQA + w9 x HumanE + w10 x MBPP + w11 x Math + w12 x GSM8K

[0093] The weight values of w7, w8 and w9 are greater than those of w1, w2, w3, w4, w5, w6, w10, w11 and w12.

[0094] Specifically, NQ relates to the ability to retrieve information from long context; TriQA requires understanding and extracting accurate information from a large amount of context; HumanE evaluates the performance of the model in human tasks, including context understanding and generation ability; the three test sets are more relevant to the present application, so higher weights are allocated in the relevance weight measurement algorithm. PIOA, Arc-e, Arc-c, MBPP, Math and GSM8K are less relevant to the present application, so lower weights are allocated in the relevance weight measurement algorithm.

[0095] For example, the weights can be allocated as follows:

[0096] NQ: 0.15; TriQA: 0.15; HumanE: 0.15; MMLU: 0.10; HellaS: 0.10; WinoG: 0.10; PIOA: 0.05; Arc-e: 0.05; Arc-c: 0.05; MBPP: 0.05; Math: 0.05; GSM8K: 0.05.

[0097] The application results of the interface calling process automatically generated by the large model are measured as follows.

[0098] Target 1: the number of interfaces is 235, the number of test calling processes after deduplication is 158, the number of effective test calling processes in 1000 test calling processes generated by the large model is 51, so the efficiency of the large model test calling process is 5.1%, and the coverage rate of the test calling process is 32.28%.

[0099] Target 2: the number of interfaces is 169, the number of test calling processes after deduplication is 248, the number of effective test calling processes in 1000 test calling processes generated by the large model is 77, so the efficiency of the large model test calling process is 7.7%, and the coverage rate of the test calling process is 31.05%.

[0100] Figure 2 The module schematic diagram of the fuzzy test device for automatically generating business test cases of the large model provided by an embodiment of the present application is shown in FIG. 1. Now referring to FIG. 1, Figure 2The embodiment of the present application also provides a fuzzy testing device for automatically generating a business test case by a large model, characterized in that the device comprises:

[0101] a test target uploading module 21 configured to upload a test target into a fuzzy testing system;

[0102] an interface document uploading module 22 configured to upload an interface document corresponding to the test target into the fuzzy testing system;

[0103] an interface calling flow generating module 23 configured to generate an interface calling flow and parameter data in each interface by using a fuzzy testing engine;

[0104] a test target service calling module 24 configured to call a test target service according to the interface calling flow and the parameter data;

[0105] a coverage data collecting module 25 configured to collect coverage data of the test target service under execution of a test case, wherein the test case comprises the interface calling flow and the parameter data;

[0106] a new path generation judging module 26 configured to judge whether a new path is generated in coverage compared with a previous test case according to the coverage data;

[0107] a test sufficiency judging module 27 configured to judge whether a test on the test target is sufficient;

[0108] a new interface calling flow generating module 28 configured to feed back the interface calling flow coverage to a large model and cyclically generate a new interface calling flow again when a new path is generated or a new path is not generated and the test is not completed.

[0109] In summary, the fuzzy testing method and device for automatically generating a business test case by a large model, which comprises the following steps: uploading a test target into a fuzzy testing system; uploading an interface document corresponding to the test target into the fuzzy testing system; generating an interface calling flow and parameter data in each interface by using a fuzzy testing engine; calling a test target service according to the interface calling flow and the parameter data; collecting coverage data of the test target service under execution of a test case, wherein the test case comprises the interface calling flow and the parameter data; judging whether a new path is generated in coverage compared with a previous test case according to the coverage data; judging whether a test on the test target is sufficient; and feeding back the interface calling flow coverage to a large model and cyclically generating a new interface calling flow again when a new path is generated or a new path is not generated and the test is not completed, so as to effectively improve each coverage rate of the fuzzy testing.

[0110] It should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, and are not intended to limit the present application; although the present application has been described in detail with reference to the above embodiments, those skilled in the art should understand that the technical solutions recorded in the above embodiments can be modified, or some or all of the technical features can be replaced by equivalents; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the scope of the technical solutions of the embodiments of the present application.

Claims

1. A fuzz testing method for automatically generating business test cases from a large model, characterized in that, The method includes: Upload the test range to the fuzzy testing system; Upload the interface documentation corresponding to the test range to the fuzz testing system; The API call flow and parameter data of each API are generated using a fuzz testing engine. Based on the interface call process and the parameter data, the test range service is invoked. Collect coverage data of the test range service under the execution of the test case, wherein the test case includes the interface call process and the parameter data; Based on the coverage data, determine whether new paths have been created in terms of coverage compared to previous test cases; Determine whether the testing of the test range is sufficient; When a new path is generated or no new path is generated and the test is not completed, the coverage of the interface call process is fed back to the large model and a new interface call process is generated again in a loop. The method of feeding back the coverage of the interface call process to the large model and repeatedly generating a new interface call process is achieved through the Targeted-Gen algorithm. Using the interface call flow and parameter data as the initial population, the individual coverage rate of each interface permutation is calculated. A standard differential evolution algorithm is used for iteration to generate new candidate solutions. The coverage rate of each candidate solution is calculated, and the current optimal solution is updated. The contribution of each dimension to the current optimal solution is calculated, and dimensions with higher contributions are selected as potential bottleneck dimensions. A target vector similar to the current optimal solution is generated, and some dimension values ​​from the target vector are inserted into the current optimal solution, replacing or modifying the values ​​of the bottleneck dimensions. The coverage rate of the modified solution is measured; if the coverage rate increases, the current optimal solution is updated. When a preset number of iterations is reached or the coverage rate no longer significantly increases, the current optimal solution is output.

2. The fuzzing method for automatically generating business test cases from a large model according to claim 1, characterized in that, The API call process for large model generation is optimized by adding attention-based questions to the API information, specifically including: By asking questions using intention, the large model's understanding of the interface is improved, and then the interface call flow is generated by combining intention.

3. The fuzzing method for automatically generating business test cases from a large model according to claim 1, characterized in that, The API call process for large model generation has been optimized by adding standardized prompt words, specifically including: By incorporating commonly used COT prompts, we can improve the understanding of the interface by large models and standardize the generation of results at each step of the large model.

4. The fuzzing method for automatically generating business test cases from a large model according to claim 1, characterized in that, By adding the keyword "json" to require JSON responses when outputting large models, the API call process for generating large models is optimized, thereby reducing the number of tokens generated for large models.

5. The fuzzing method for automatically generating business test cases from a large model according to claim 1, characterized in that, The large models include Starchat-13b, Codellama-34b, and Mixtral-8x7b.

6. The fuzzing method for automatically generating business test cases from a large model according to claim 1, characterized in that, The large model was evaluated using a standard test set, which includes: MMLU is used to evaluate large-scale multi-task language comprehension capabilities; HellaS is used to evaluate a model’s ability to perform complex reasoning tasks; WinoG is used to evaluate the model's ability to solve the pronoun resolution problem; PIOA is used to evaluate a model's ability to perform contextual optimization given cues; Arc-e and Arc-c are used to evaluate scientific reasoning tasks that represent basic and university levels, respectively. NQ is used to measure a model’s accuracy and understanding of natural question answering tasks; TriQA is used to evaluate the model's ability to perform cross-language question answering tasks; HumanE is used to evaluate the model's ability to perform human commonsense reasoning tasks; MBPP is used to evaluate the model’s ability to solve programming problems. Math is used to evaluate a model’s ability to solve mathematical problems. GSM8K was used to evaluate the model's ability to solve elementary school math problems.

7. The fuzzing method for automatically generating business test cases from a large model according to claim 6, characterized in that, The relevance weighting algorithm formula for the standard test set is as follows: Overall score = w1×MMLU+w2×HellaS+w3×WinoG+w4×PIOA+w5×Arc-e+w6×Arc-c+w7×NQ+w8×TriQA+w9×HumanE+w10×MBPP+w11×Math+w12×GSM8K Among them, the weight values ​​of w7, w8, and w9 are greater than those of w1, w2, w3, w4, w5, w6, w10, w11, and w12.

8. A fuzz testing device for automatically generating business test cases from a large model, characterized in that, The device includes: The test range upload module is used to upload the test range to the fuzz testing system; The interface document upload module is used to upload the interface document corresponding to the test range to the fuzz testing system. The interface call flow generation module is used to generate the interface call flow and parameter data of each interface through the fuzz testing engine. The test range service invocation module is used to invoke the test range service according to the interface invocation process and the parameter data. The coverage data collection module is used to collect coverage data of the test range service under the execution of the test case, wherein the test case includes the interface call process and the parameter data; The new path determination module is used to determine, based on coverage data, whether a new path has been generated in terms of coverage compared to previous test cases. The test sufficiency assessment module is used to determine whether the test of the test range is sufficient. A new API call flow generation module is used to feed back the API call flow coverage status to the large model and generate a new API call flow again in a loop when a new path is generated or no new path is generated and the test is not completed. The method of feeding back the coverage of the interface call process to the large model and repeatedly generating a new interface call process is achieved through the Targeted-Gen algorithm. Using the interface call flow and parameter data as the initial population, the individual coverage rate of each interface permutation is calculated. A standard differential evolution algorithm is used for iteration to generate new candidate solutions. The coverage rate of each candidate solution is calculated, and the current optimal solution is updated. The contribution of each dimension to the current optimal solution is calculated, and dimensions with higher contributions are selected as potential bottleneck dimensions. A target vector similar to the current optimal solution is generated, and some dimension values ​​from the target vector are inserted into the current optimal solution, replacing or modifying the values ​​of the bottleneck dimensions. The coverage rate of the modified solution is measured; if the coverage rate increases, the current optimal solution is updated. When a preset number of iterations is reached or the coverage rate no longer significantly increases, the current optimal solution is output.

Images