Hybrid testing method and device for application program, storage medium and program product

Abstract

The embodiment of the invention provides a hybrid test method and device for an application program, a storage medium and a program product, and belongs to the field of software testing, the method comprises the following steps: running a plurality of test cases generated by a fuzzy test method in the application program to determine the access probability of a plurality of non-accessed program paths of the plurality of test cases, and screening out a target non-accessed program path with relatively low access probability for performing symbolic execution from the plurality of non-accessed program paths, and continuously testing the application program based on a test case generated by symbolic execution. Through combination of fuzzy testing and symbolic execution, the coverage rate and efficiency of application program testing are improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of software testing, and particularly relates to a hybrid testing method, device, storage medium and program product of an application program. BACKGROUND

[0002] In order to find bugs and errors in an application program, the application program needs to be tested. A conventional testing scheme is that test cases covering different application scenarios are manually written, and the test cases are run in the application program to be tested to complete the testing. The manual testing scheme costs a large amount of manpower and time.

[0003] Another commonly used testing method is fuzz testing. Fuzz testing is a commonly used software testing technique. The main idea is to automatically generate a large number of random, invalid or abnormal data test cases, input them into the application program to be tested for execution, and monitor the running state of the application program to find test cases that cause the running state of the application program to be abnormal, and then perform mutation to generate more high-quality test cases to find bugs in the application program that have not been exposed.

[0004] Fuzz testing has natural scalability when testing large application programs, but fuzz testing only provides best-effort testing and evaluates the testing effect by coverage. In simple terms, coverage is that the application program to be tested includes N program paths, and fuzz testing can only access M of them, so the coverage is M / N, where M < N. Large application programs often contain many program paths, and due to the high randomness and low testing efficiency of fuzz testing, it is often difficult to achieve good coverage. SUMMARY

[0005] The hybrid testing method, device, storage medium and program product of the application program provided by the embodiments of the present application can improve the coverage of application program testing and ensure the scalability of application program testing.

[0006] In a first aspect, the embodiments of the present application provide a hybrid testing method of an application program, and the method comprises:

[0007] Running a plurality of test cases generated by a fuzz testing method in the application program to be tested to determine the access probability of a plurality of unvisited paths in the application program for the plurality of test cases;

[0008] According to the access probability of the plurality of unvisited program paths, a target unvisited program path is selected from the plurality of unvisited program paths, wherein the access probability of the target unvisited program path is lower than the access probability of the unvisited program paths that are not selected;

[0009] performing symbolic execution on the target unvisited path to obtain test cases generated by the symbolic execution;

[0010] continuing to test the application based on the test cases generated by the symbolic execution.

[0011] In a second aspect, an embodiment of the present application provides a hybrid testing device for an application, the device comprising:

[0012] a first testing module configured to run a plurality of test cases generated by a fuzz testing method in the application under test, and determine access probabilities of a plurality of unvisited program paths in the application under test;

[0013] a path determining module configured to filter a target unvisited program path from the plurality of unvisited program paths according to the access probabilities of the plurality of unvisited program paths, wherein the access probability of the target unvisited program path is lower than that of an unvisited program path not filtered out;

[0014] a second testing module configured to perform symbolic execution on the target unvisited path to obtain test cases generated by the symbolic execution, and continue to test the application based on the test cases generated by the symbolic execution.

[0015] In a third aspect, an embodiment of the present application provides an electronic device, comprising a memory, a processor, and a communication interface, wherein the memory stores executable code, and when the executable code is executed by the processor, the processor can at least implement the hybrid testing method for an application according to the first aspect.

[0016] In a fourth aspect, an embodiment of the present application provides a non-transitory machine readable storage medium, wherein the non-transitory machine readable storage medium stores executable code, and when the executable code is executed by a processor of an electronic device, the processor can at least implement the hybrid testing method for an application according to the first aspect.

[0017] In a fifth aspect, an embodiment of the present application provides a computer program product, wherein the computer program product contains a computer program, and when the computer program is executed by a processor of an electronic device, the processor can at least implement the hybrid testing method for an application according to the first aspect.

[0018] The application program hybrid test method provided by the embodiment of the application comprehensively uses two different methods of fuzzy test and symbolic execution. First, the application program is tested by the fuzzy test method. Specifically, a plurality of test cases generated by the fuzzy test method are run in the application program to be tested to determine a plurality of unvisited program paths in the application program and corresponding access probabilities of the plurality of test cases, a target unvisited program path with a lower access probability is screened from the plurality of unvisited program paths, and symbolic execution is performed on the target unvisited program path to obtain test cases generated by the symbolic execution, so that the test of the application program is continued based on the test cases generated by the symbolic execution.

[0019] In the embodiment of the application, the fuzzy test of the application program is performed by using a plurality of test cases, and the scalability of the test of the application program is ensured. For the program paths that cannot be accessed by the plurality of test cases of the fuzzy test, that is, the unvisited program paths, a target unvisited program path with a lower access probability is screened by calculating the access probabilities, and another technology, that is, symbolic execution, is switched to use. Therefore, the test cases generated based on the symbolic execution result of the target unvisited program path can access the target unvisited program path that cannot be covered by the previous test cases, so that the coverage of the test of the application program is improved. BRIEF DESCRIPTION OF DRAWINGS

[0020] In order to more clearly illustrate the technical solutions in the embodiments of the application, the drawings needed in the embodiment description will be briefly introduced. Obviously, the drawings in the following description are some embodiments of the application, and other drawings can be obtained by those skilled in the art without creative labor.

[0021] Figure 1 A composition schematic diagram of an application program hybrid test system provided by the embodiment of the application;

[0022] Figure 2 A flowchart of an application program hybrid test method provided by the embodiment of the application;

[0023] Figure 3 A schematic diagram of a control flow graph provided by the embodiment of the application;

[0024] Figure 4 A schematic diagram of an unvisited program path determination result provided by the embodiment of the application;

[0025] Figure 5 A flowchart of an access probability determination process of an unvisited program path provided by the embodiment of the application;

[0026] Figure 6A flow chart of a symbol execution process on an unvisited program path provided by an embodiment of the present application;

[0027] Figure 7 A flow chart of determining a test case value provided by an embodiment of the present application;

[0028] Figure 8 A structural schematic diagram of a hybrid test device of an application provided by an embodiment of the present application;

[0029] Figure 9 A structural schematic diagram of an electronic device provided by an embodiment of the present application. DETAILED DESCRIPTION

[0030] To make the objectives, technical solutions and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are some but not all of the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by a person of ordinary skill in the art without creative work fall within the protection scope of the present application.

[0031] It should be noted that the user information (including but not limited to user equipment information, user personal information, etc.) and data (including but not limited to data for analysis, stored data, displayed data, etc.) involved in the embodiments of the present application are all information and data authorized by the user or authorized by all parties, and the collection, use and processing of the related data need to comply with the relevant laws, regulations and standards of the relevant countries and regions, and provide corresponding operation entrances for the user to choose authorization or refusal.

[0032] Some embodiments of the present application will be described in detail below with reference to the drawings. The following embodiments and features in the embodiments can be combined with each other without conflict between the embodiments. In addition, the step sequence in each method embodiment described below is only an example, not a strict limitation.

[0033] First, the terms involved in the embodiments of the present application will be described.

[0034] Fuzzing is a method of discovering software vulnerabilities by providing unexpected input to target software and monitoring abnormal running results of the target software.

[0035] Mutation refers to generating new test cases by changing existing test cases.

[0036] Symbolic execution is a static analysis technique used to automatically derive input conditions and execution paths of an application and solve input data. Static means that the application itself is not actually executed, but the execution path of the application, i.e. the program path, is analyzed. Modern symbolic execution techniques include concrete symbolic execution, hybrid symbolic execution (Concolic Execution) and the like, wherein hybrid symbolic execution is a technique combining concrete execution and symbolic execution.

[0037] A P4 program is a network packet processing program written in the P4 language.

[0038] A program path refers to an executable path in an application, and the code of the application often contains branches such as if and else. The existence of these branches results in many executable paths, i.e. program paths, in the application. A program path is composed of one or more sequentially executed program branches.

[0039] A cloud gateway is the core hub in the entire cloud, responsible for communicating north-south and east-west traffic, and plays a key role in cloud infrastructure. However, the traditional cloud gateway based on x86 architecture cannot cope with the growing network traffic, and is currently inspired by programmable switching ASIC (Application-Specific Integrated Circuit, ASIC for short). Switching ASIC written in P4 language is replacing the original cloud gateway based on x86 architecture in the cloud. This new type of cloud gateway is called P4 gateway, which can greatly improve the traffic forwarding capability.

[0040] Compared with the packet processing program of the x86 architecture cloud gateway written in C language, the corresponding P4 program exposes more unpredictable vulnerabilities due to the hardware architecture and programming language P4 of the above-mentioned P4 gateway, resulting in testers consuming a lot of effort to test the program. In the actual production process, testers manually write test cases to test the P4 program, which is a heavy task and has low testing efficiency. Therefore, an automated testing system needs to be implemented, which can be deployed on the P4 gateway to ensure its stable operation.

[0041] Fuzz testing or symbolic execution methods can be used to detect possible vulnerabilities in P4 programs.

[0042] As described above, the main idea of fuzz testing is to pre-generate or randomly generate a large number of test cases, execute the test cases by the application program, obtain guidance based on the feedback of the running state of the application program, and then mutate the test cases to generate better test cases to discover the vulnerabilities that have not been exposed. Fuzz testing shows natural scalability when testing large programs, because the test cases are randomly generated, so it can be easily extended to complex application programs. Although the number of program paths of a program increases exponentially with the size of the program, fuzz testing only provides best-effort testing and judges the testing effect by coverage. P4 programs are often large application programs, that is, there are a large number of program paths in the programs. Due to the randomness of fuzz testing and low testing efficiency, it is often difficult to achieve high coverage.

[0043] The method for testing P4 programs by symbolic execution is to use symbolic execution to traverse the program paths in the application program, collect path constraints composed of symbolic variables, and then use a first-order theorem prover (i.e., a solver) to solve the path constraints to generate input and output data packets corresponding to the path constraints. This method has high testing accuracy, but when facing production-level P4 programs, the state space of the program paths may be explosive, and the complexity of collecting and solving the path constraints is too high, so it is often difficult to complete the testing and has insufficient scalability.

[0044] In summary, if symbolic execution is used alone to test P4 programs, all program paths in the P4 programs need to obtain their respective path constraints and be solved to achieve testing, so the coverage is high. However, in the path constraint obtaining and solving phase, the state is explosive. If the application program is particularly complex, that is, the application program contains a large number of program paths, the complexity of obtaining and solving the path constraints is too high, so it is often difficult to complete the testing, which causes the problem of insufficient scalability. If fuzz testing is used alone, because the test cases are randomly generated, it can be extended to complex programs, but this randomness makes it difficult to improve the coverage, that is, the randomly generated test cases are difficult to access all the program paths contained in the application program, so the problem of low coverage is caused.

[0045] In view of this, in the embodiments of the present application, the method of mixing fuzz testing and symbolic execution can compensate for the shortcomings of each other, so that the testing scheme of the application program provided by the embodiments of the present application can be compatible with scalability and coverage.

[0046] It should be noted that although the P4 program in the cloud gateway is taken as an example for description above, the hybrid testing scheme provided by the embodiments of the present application can be applied to any application program including but not limited to the P4 program, and is not limited to the programming language of the application program.

[0047] Figure 1A structural schematic diagram of a hybrid test system of an application program provided by an embodiment of the present application is shown in Figure 1 From the perspective of function decomposition, the hybrid test system can include five service modules: a fuzz test module, a coverage statistics module, a path optimization module, a symbolic execution module, and a test case optimization module.

[0048] In actual application, the hybrid test system can be provided as a service in the cloud and deployed in a computing node (i.e., a cloud server) in the cloud. The service can provide a service interface for users to call, such as inputting a compiled executable file of the application program to be tested or access address information of the executable file to trigger the hybrid test system to test the application program.

[0049] Optionally, when the service is deployed in the computing node, the running environment of the service can be configured in the computing node, such as building a virtual machine or a container group (pod) corresponding to each service in the computing node, and allocating corresponding memory resources and processor resources.

[0050] The hybrid test method for an application program provided by an embodiment of the present application will be described below in combination with the hybrid test system shown in Figure 1

[0051] Figure 2 A flowchart of a hybrid test method for an application program provided by an embodiment of the present application is shown in Figure 2 The method can include the following steps:

[0052] 201. Running a plurality of test cases generated by a fuzz test method in the application program to be tested, and determining the access probabilities of a plurality of unvisited paths of the application program.

[0053] 202. Selecting a target unvisited path from the plurality of unvisited paths according to the access probabilities of the plurality of unvisited paths, wherein the access probability of the target unvisited path is lower than the access probabilities of the unvisited paths that are not selected.

[0054] 203. Performing symbolic execution on the target unvisited path, obtaining test cases generated by the symbolic execution, and continuing to test the application program based on the test cases generated by the symbolic execution.

[0055] In an embodiment of the present application, the application program to be tested can be a P4 program, and the corresponding network device can be a cloud gateway, a router, a switch, etc. Of course, the application program to be tested can also be a large application program written in other programming languages, such as an application program containing program paths greater than a set threshold.

[0056] ​In the embodiment of the present application, a hybrid test method of fuzzy testing and symbolic execution is used to test the application program to be tested to find the vulnerabilities and errors existing therein. The fuzzy testing is mainly used, and generally most of the program paths in the application program can be tested by the fuzzy testing. The symbolic execution is used to compensate for the program paths that are difficult to access by the fuzzy testing, so as to achieve high coverage of the application program testing.

[0057] Therefore, the fuzzy testing method is first used to test the application program. Specifically, the fuzzy testing module providing the fuzzy testing method first needs to generate a plurality of test cases, and run the test cases in the application program to determine a plurality of accessed program paths of the test cases in the application program.

[0058] Generally, the fuzzy testing module is mainly responsible for the generation and mutation of the test cases. In actual application, the fuzzy testing module can collect some normal input data generated in the running process of the application program, and randomly modify the normal input data to obtain a plurality of test cases. The normal input data refers to the output result generated by the application program based on the input data without prompting an exception. It can be understood that the input data often contains a plurality of parameter fields and specific parameter values, and the fuzzy testing module can modify the values of some parameter fields to generate test cases.

[0059] Taking the P4 program as an example, the data packets actually transmitted by the network device running the data packet processing program can be collected as the input of the fuzzy testing method, so that the values of the parameter fields in the data packets are modified to obtain the data packets as test cases.

[0060] Actually, a data packet contains a plurality of fields such as IP address, port number, protocol type, network segment, check bit, etc. The state space of the fields is very large, although the values of some fields can be randomly changed to generate test cases, in order to reduce the too random change of the fields to obtain test cases, the test cases can be generated based on the following optimization strategies:

[0061] (1) In the process of processing the data packet by the data packet processing program, the branch can be triggered based on the hit forwarding table entry. Based on this, taking the collected data packet a as an example, any specific value (generally IP address) contained in the generated forwarding table entry can be used to replace the value of the corresponding parameter field in the data packet a, so as to obtain the data packet b as a test case. In this way, when the data packet b is run, the corresponding program branch can be accessed.

[0062] (2) The data packet contains a protocol type field, and the available values of the field are not unique. The values of the field in the collected data packets are counted, and the specific protocol types are taken as a database. For example, the value of a protocol type is selected from the database to replace the value of the corresponding protocol type in the data packet a, so as to obtain the data packet c as a test case.

[0063] (3) The specific network segment values in the collected data packets are counted to obtain some special network segments, such as network segments with less occurrence frequency or network segments close to the boundary within the set network segment range. For example, a special network segment is selected from the special network segments to replace the value of the corresponding network segment in the data packet a, so as to obtain the data packet d as a test case.

[0064] (4) In order to test some extreme cases, the boundary values of some parameter fields defined in the specification file of the network device can be used to generate test cases. For example, the specification file specifies that the length of a data packet ranges from L1 to L2, and the value of the data packet length field in the collected data packet a is changed to L3, which is not within the range.

[0065] (5) The more important test cases are mutated to obtain new test cases. The mutation is performed in units of parameter fields, that is, one parameter field is modified each time. For example, if the application program runs the test case x and the running state is abnormal, that is, a vulnerability or error is found, the test case x is more important, and the test case y obtained by mutating the test case x can discover more vulnerabilities. For another example, compared with other test cases, the test case z accesses program paths that have not been accessed by other test cases, and the test case z is more important. The new test case obtained by mutating the test case z can access more program paths that have not been accessed and discover more program vulnerabilities.

[0066] After obtaining the multiple test cases corresponding to the application program to be tested through the fuzz testing module, the test cases generated by the fuzz testing method are run in the application program to be tested respectively, and the running state of the application program is monitored to discover vulnerabilities, errors and other abnormal problems in the application program.

[0067] In fact, in order to run these test cases, it is necessary to run in the executable file after the application program is compiled, so the application program needs to be compiled first. In addition, in order to realize the calculation of the access probability of the subsequent program paths, a counting code for counting the access times of the program branches contained in each program path in the application program needs to be inserted in the process of compiling the application program.

[0068] Based on this, the abstract syntax tree of the application program can be determined in the compilation stage of the application program, the abstract syntax tree is converted into an intermediate representation matched with the compiler of the application program, the above counting code is inserted into the intermediate representation, then the intermediate representation inserted with the counting code is compiled to obtain a compiled executable file, and the above multiple test cases are respectively run in the executable file. In actual application, when the application program is a P4 program, the above compiler can be a P4C compiler.

[0069] The above counting code can actually be inserted in front of each program branch, so that each time a test case accesses a program branch during running, the corresponding counter is triggered to add one, thereby updating the access frequency of the program branch.

[0070] In addition, based on the compilation process of the application program, a control flow graph (Control Flow Graph, CFG for short) of the application program can also be generated. Specifically, after the abstract syntax tree of the application program is parsed, the control flow graph corresponding to the application program can be generated according to the abstract syntax tree. According to the control flow graph, each program path contained in the application program can be determined. Among them, the nodes and directed edges in the control flow graph respectively represent each basic block in the application program and the conversion between different basic blocks, and the basic blocks connected by a directed edge constitute a program branch.

[0071] For ease of understanding, Figure 3 A control flow graph is shown, which includes a total of 10 nodes numbered 0-9. In Figure 3 , from the root node to the leaf node, there are a total of 5 complete program paths: the program path L 0-7 from the root node 0 to the leaf node 7, the program path L 0-8 from the root node 0 to the leaf node 8, the program path L 0-9 from the root node 0 to the leaf node 9, the program path L 0-2 from the root node 0 to the leaf node 2, and the program path L 0-6 from the root node 0 to the leaf node 6. Taking the program path L 0-8 as an example, it includes the following program branches: the program branch b 0-1 from the root node 0 to the node 1, the program branch b 1-5 from the node 1 to the node 5, and the program branch b 5-8 from the node 5 to the node 8.

[0072] By generating the control flow graph, the whole program paths contained in the application program can be comprehensively understood. In the process of respectively executing the above multiple test cases in the compiled executable file of the application program, based on the counting code inserted by the patching, the access times of the multiple program branches in the application program after the multiple test cases are run can be obtained, and the program path accessed by each test case can be determined based on the program branches accessed by the test case. Thus, after the test cases are run, on one hand, the multiple accessed program paths accessed by the multiple test cases are obtained, and on the other hand, the access times of the program branches at this time are obtained. Then, according to the multiple accessed program paths and the access times of the multiple program branches, the multiple unaccessed program paths in the application program and the access probabilities of the multiple unaccessed program paths can be determined.

[0073] Therefore, specifically, determining the access probabilities of the multiple unaccessed program paths of the multiple test cases in the application program comprises: determining the multiple accessed program paths of the multiple test cases in the application program, determining the access times of the multiple program branches in the application program after the multiple test cases are run by the counting code inserted in the application program, and determining the multiple unaccessed program paths in the application program and the access probabilities of the multiple unaccessed program paths according to the multiple accessed program paths and the access times of the multiple program branches, wherein the counting code is used to count the access times of the program branches contained in each program path in the application program.

[0074] The reason for finding the unaccessed program paths and determining the access probabilities of the unaccessed program paths is to find the combination point of the fuzz testing and the symbolic execution.

[0075] In the embodiment of the application, in order to combine the fuzz testing and the symbolic execution, the test tasks can be classified: the fuzz testing is used to complete the easily explored program paths, and the symbolic execution is used to complete the program paths difficult to be accessed by the fuzz testing. Therefore, the key problem is how to quantify the difficulty of exploring the program paths.

[0076] It can be understood that if a program path is not accessed in the test of a large number of test cases, the program path is obviously the condition of being difficult to explore by the symbol. Therefore, the value of the access probability of a program path can be approximately equal to the access frequency in the multiple test cases, that is, the ratio between the access times of the program path and the total number of test cases. However, if the access probability of each program path is determined according to the definition, the access times of the program paths need to be directly calculated at the granularity of the program paths, especially in a large application program (the program paths are multiple and long), which is difficult to accurately calculate. Therefore, the embodiment of the application further provides another way of calculating the access probability of the program path.

[0077] In fact, determining the access probability of a program path, i.e. determining the coverage of a program path, is realized by the coverage statistics module in the program. Figure 1

[0078] When solving the access probability of a program path in a large application program, it is often limited by the state explosion problem of the program path. However, it is observed that a program path is composed of program branches, and therefore the access probability solving problem of a program path can be regarded as a continuous Markov process: the access probability of a program path can be approximated as the product of the access probabilities of all program branches in the program path. Thus, it is only necessary to determine the access probability of each program branch at the granularity of a program branch. The access probability of a program branch can be calculated based on the access times of the program branch, which can be obtained by counting the code through the plug-in.

[0079] Specifically, taking any program path L j as an example, the access probability P(L j ) can be calculated according to the following formula:

[0080]

[0081] where b i represents a program branch contained in the program path L j , and P(b i ) represents the access probability of the program branch b i . where t i represents the access times of the program branch b i , and t n represents the cumulative sum of the access times of the program branch adjacent to the program branch b i and the access times of the program branch b i .

[0082] The meaning of the program branch adjacent to a program branch will be described below. Here, it is first necessary to note that, when calculating based on the above formula, if the access times of a program branch are actually zero, a very small constant is used to replace the access times, so as to accurately reflect the size relationship of the access probabilities of different program branches.

[0083] The above describes the calculation method of the access probability of a program path. In the embodiment of the present application, as described above, after a plurality of test cases are run in the application program to complete the fuzz testing, a plurality of accessed program paths exist in the application program, and the main program paths whose access probabilities need to be calculated are the unaccessed program paths existing in the application program at this time. Of course, if it is desired to calculate the access probabilities of the accessed program paths, the access probabilities of the accessed program paths can also be calculated.​

[0084] Based on this, the path optimization module can determine the multiple unvisited program paths existing in the application program at this time according to the multiple visited program paths at this time. The multiple unvisited program paths can be determined based on the control flow graph.

[0085] In fact, an execution tree for recording the visited program paths can be generated, and through the execution tree and the control flow graph, the multiple unvisited program paths can be more conveniently determined. The execution tree is empty in the initial state when the test cases have not been run, and each time a test case is run, the program path visited by the test case is added to the execution tree. After the multiple test cases are all executed, the execution tree at this time and the control flow graph are compared to determine the program paths in the control flow graph other than the program paths contained in the execution tree as the multiple unvisited program paths.

[0086] Then, for any visited program path in the multiple visited program paths, unvisited neighbor nodes corresponding to the multiple nodes in the any visited program path are determined, wherein a node and its neighbor node have the same parent node. Finally, unvisited program paths formed between the root node and each neighbor node are determined, and the root node is the root node in the control flow graph. Each visited program path is traversed one by one, and finally the multiple unvisited program paths are obtained.

[0087] For the convenience of understanding, the following will be described in conjunction with Figure 4 . As shown in the control flow graph in Figure 4 , it is assumed that there are two visited program paths as follows: a program path L 0-8 from the root node 0 to the leaf node 8, and a program path L 0-6 from the root node 0 to the leaf node 6. In the figure, the results are marked with thick solid lines.

[0088] For the program path L 0-8 , which contains the root node 0, the node 1, the node 5, and the node 8, the unvisited neighbor nodes of these nodes are determined one by one, and the determination results are as follows: the root node 0 has no neighbor node; because the node 1 and the node 3 have the same parent node: the root node 0, the node 3 is the neighbor node of the node 1, but since the node 3 has been visited, the node 3 is not an unvisited neighbor node of the node 1; the node 5 and the node 4 have the same parent node: the node 1, and the node 4 has not been marked as visited, so the node 4 is found to be an unvisited neighbor node of the node 5; the node 8 and the node 9 have the same parent node: the node 5, and the node 9 has not been marked as visited, so the node 9 is found to be an unvisited neighbor node of the node 8. Based on the above neighbor node determination results, the unvisited nodes 4 and 9 are found for the program path L 0-8 .

[0089] Similarly, for the program path L 0-6 , which contains the root node 0, the node 3 and the node 6, the unvisited neighbor nodes of these nodes are determined one by one, and the determination results are as follows: the root node 0 has no neighbor node; the node 3 and the node 1 have the same parent node: the node 0, but the node 1 is marked as visited, so there is no unvisited neighbor node corresponding to the node 3; the node 6 and the node 2 have the same parent node: the node 3, and the node 2 is not marked as visited, so the node 2 is the unvisited neighbor node of the node 6 found. Based on the above neighbor node determination results, for the program path L 0-6 , the unvisited node 2 is found.

[0090] Then, the program paths formed between the root node 0 and the above unvisited neighbor nodes are determined as unvisited program paths respectively. Specifically, the program path L 0-4 formed between the root node 0 and the node 4 is an unvisited program path, which contains the program branch b 0-1 and the program branch b 1-4 ; the program path L 0-9 formed between the root node 0 and the node 9 is an unvisited program path, which contains the program branch b 0-1 , the program branch b 1-5 and the program branch b 5-9 ; the program path L 0-2 formed between the root node 0 and the node 2 is an unvisited program path, which contains the program branch b 0-3 and the program branch b 3-2 .

[0091] After obtaining the above multiple unvisited program paths, the access probabilities of each unvisited program path can be calculated based on the calculation formula in the above.

[0092] As shown in Figure 5 , the process of determining the access probabilities of the multiple unvisited program paths can include the following steps:

[0093] 501. For any unvisited program path in the multiple unvisited program paths, determine each program branch contained in the any unvisited program path.

[0094] 502. For a first program branch in each program branch contained in the any unvisited program path, determine each second program branch adjacent to the first program branch, the parent node in each second program branch being the same as the parent node in the first program branch, the first program branch being any one of the program branches contained in the any unvisited program path.

[0095] 503、determining the total access times of the first program branch and the second program branches, and determining the access probability of the first program branch as the ratio of the access times of the first program branch to the total access times.

[0096] 504、determining the access probability of the any unvisited program path according to the access probabilities of the program branches included in the any unvisited program path.

[0097] The parent node in one program branch refers to: the program branch corresponds to the branch from node i to node j, relatively, node i is the parent node of node j, and is referred to as the parent node in the program branch.

[0098] For the convenience of understanding, the determination result of the unvisited program path in the embodiment shown in FIG. 6 is still taken as an example for description. Figure 4

[0099] For the unvisited program path L 0-2 , it is determined that it includes the program branch b 0-3 and the program branch b 3-2 .

[0100] For the program branch b 0-3 , the parent node in the program branch is the root node 0, the program branch with the root node 0 as the parent node further includes the program branch b 0-1 , and therefore the program branch adjacent to the program branch b 0-3 is the program branch b 0-1 . According to the counting result of the access times of the program branch b 0-3 and the program branch b 0-1 , the access probability P1 of the program branch b 0-3 is determined as: the access times of the program branch b 0-3 / (the access times of the program branch b 0-3 + the access times of the program branch b 0-1 .

[0101] For the program branch b 3-2 , the parent node in the program branch is the node 3, the program branch with the node 3 as the parent node further includes the program branch b 3-6 , and therefore the program branch adjacent to the program branch b 3-2 is the program branch b 3-6 . According to the counting result of the access times of the program branch b 3-2 and the program branch b 3-6 , the access probability P2 of the program branch b 3-2 is determined as: the access times of the program branch b 3-2 / (the access times of the program branch b 3-2 + the access times of the program branch b 3-6 .​

[0102] Thus, the access probability of the unvisited program path L 0-2

[0103] Figure 4 The access probability of the unvisited program path L 0-9 0-4 The calculation process of the access probability of the unvisited program path L

[0104] After obtaining the access probabilities of the multiple unvisited program paths, the target unvisited program path for symbolic execution is selected from the multiple unvisited program paths according to the access probabilities of the multiple unvisited program paths, wherein the access probability of the target unvisited program path is lower than the access probability of the unvisited program path that is not selected.

[0105] Specifically, the multiple unvisited program paths can be sorted in the order of the access probability from low to high, and the unvisited program paths with the top K (i.e., the top K positions) access probabilities can be selected as the target unvisited program paths, where K is a preset value, such as 3. Alternatively, the unvisited program paths with the access probability lower than a set threshold value can be selected as the target unvisited program paths.

[0106] Then, the symbolic execution is performed on the target unvisited program path, and the test case satisfying the target unvisited program path can be generated through the symbolic execution. For example, in the case of testing the P4 program, the data packet satisfying the target unvisited program path is generated as the test case. The test case satisfying the target unvisited program path means that the test case can access the target unvisited program path when the test case is run in the application program.

[0107] In fact, after obtaining the test case generated by the symbolic execution, the test case can be directly input into the application program for running, and the running state is monitored to find the abnormality in the application program. Alternatively, the test case can be input into the fuzz testing module, and the test case is processed through the fuzz testing method to generate a new test case, and the new test case is run in the application program. In this way, the combination of the fuzz testing and the symbolic execution is realized.

[0108] ​​Since test cases generated by symbolic execution can access program paths that were previously inaccessible by multiple test cases in the fuzzing phase, new test cases mutated from these symbolic execution test cases will have a higher probability of accessing even more previously unaccessed program paths. Prioritizing the execution of these mutated new test cases after the multiple test cases run in the previous fuzzing phase can accelerate the coverage of more program paths in the application, improving testing efficiency and coverage. Furthermore, implementing symbolic execution only on a small number of unaccessed program paths with low access probability results in low solution complexity, ensuring the scalability of this testing method in complex, large-scale applications.

[0109] For symbolic execution processes performed on unaccessed program paths, embodiments of the present invention also provide an optimization method, which can be specifically derived from... Figure 1 Implemented by the symbolic execution module in, such as Figure 6 As shown, the symbol execution process may include the following steps:

[0110] 601. Identify a second visited program path from multiple visited program paths that has more common prefixes with the program branch of the target unvisited program path to be symbolically executed.

[0111] 602. Determine the first constraint condition corresponding to the program branch with the common prefix in the second visited program path, and the second constraint condition corresponding to the remaining program branch in the target unvisited program path. The remaining program branch refers to the program branch remaining in the target unvisited program path other than the program branch with the common prefix.

[0112] 603. Determine the first constraint and the second constraint to form the path constraint for symbolic execution, and solve the path constraint to obtain the test cases corresponding to the unvisited program path of the target. In the process of solving the path constraint, the parameter values ​​in the test cases that have visited the second visited program path are used to assign values ​​to the corresponding symbolic variables, and the parameter values ​​satisfy the first constraint.

[0113] Assume the target does not access the program path L, as exemplified above. 0-9 This includes program branch b 0-1 Program branch b 1-5 and program branch b 5-9 And multiple accessed program paths L 0-8 and program path L 0-6 In the context, the program path L that shares more common prefixes with the unvisited program path of the target is the visited program path. 0-8 The common prefix program branches that exist in both include program branch b. 0-1 and program branch b 1-5For ease of description, assume that the two common prefix program branches collectively represent a program branch b 0-5 .

[0114] When performing symbolic execution on the program path L 0-9 , the test case corresponding to the accessed program path L 0-8 is used as a specific value of the symbolic execution, and the specific execution and the symbolic execution are combined to quickly generate a path constraint.

[0115] Specifically, in terms of a P4 program, the test case corresponding to the accessed program path L 0-8 is a data packet x, which includes a plurality of parameter fields and a value of each parameter field, i.e., a parameter value. In the symbolic execution process, a corresponding symbolic variable is set, and the parameter value of the corresponding parameter field in the data packet x can be used as an initial specific value to assign to the corresponding symbolic variable. In the process of generating the path constraint, the first constraint condition corresponding to the common prefix program branch b 0-5 and the second constraint condition corresponding to the remaining program branch b 0-9 in the program path L 5-9 are used to constitute the path constraint. Solving the path constraint obtains the test case corresponding to the program path L 0-9 : a data packet y. In the process of solving the path constraint, since the parameter value of the parameter field related to the first constraint condition in the data packet x already satisfies the first constraint condition, the first constraint condition does not need to be solved, and only the parameter value of the parameter field satisfying the second constraint condition needs to be solved, which can accelerate the solving speed.

[0116] For example, assume that the parameter fields involved in the path constraint include a1 and a2, and in the data packet x, a1 = 111 and a2 = 333, where a1 = 111 satisfies the first constraint condition. In the process of solving the path constraint, a1 does not need to be solved, and only a2 satisfying the second constraint condition needs to be solved, such as a2 = 222. Then in the generated data packet y, a1 = 111 and a2 = 222.

[0117] As can be seen, for the symbolic execution of the target unaccessed program path, by introducing the test case of the accessed program path having more common prefix program branches as priori knowledge, the efficiency of the symbolic execution can be improved.

[0118] In order to make the hybrid test system achieve higher coverage as soon as possible, and to make the more valuable test cases be mutated by the fuzzing module to run in the application, the embodiment of the present application further provides a method for evaluating the value of a test case. Through the method, the value of a test case can be calculated after the test case is run, and if the value is high, the test case is mutated preferentially, and the mutated test case is run preferentially, and then the remaining test cases are run.

[0119] In summary, any test case that has been executed is taken as a target test case, and the value of the target test case is determined, which reflects the number of nodes that are not accessed by the target test case. If the value of the target test case meets a set condition, the target test case is mutated by a fuzzing method to generate a new test case, and the application is continuously tested based on the new test case preferentially. The set condition is, for example, that the value is greater than a set threshold.

[0120] In an optional embodiment, as shown in Figure 7 , the test case optimization module determines the value of the target test case that has been executed, including the following steps:

[0121] 701. Determine the first accessed program path corresponding to the target test case in the control flow graph.

[0122] 702. Determine the unvisited neighbor nodes corresponding to the plurality of nodes in the first accessed program path, wherein a node and its neighbor nodes have the same parent node.

[0123] 703. For any neighbor node, if the any neighbor node is a leaf node, determine that the unvisited program path corresponding to the any neighbor node is formed between the root node and the any neighbor node; if the any neighbor node is not a leaf node, determine that the unvisited program path corresponding to the any neighbor node is formed between the root node and the leaf node under the any neighbor node.

[0124] 704. Delete the program paths other than the first accessed program path and the unvisited program paths corresponding to the neighbor nodes in the control flow graph, and delete the accessed nodes in the first accessed program path.

[0125] 705. Determine the value of the target test case according to the number of the remaining nodes.

[0126] For ease of understanding, still taking the control flow graph in Figure 3 as an example, it is assumed that test case A and test case B access program paths L 0-8 and L 0-6First, the conclusion: compared with test case B, test case A is more valuable because test case A has more unvisited nodes, such as node 4, node 7 and node 9 in Figure 4 , while test case B only has unvisited node 2, which means that the new test case based on test case A has a higher probability of accessing new program paths.

[0127] For each test case, the number of unvisited nodes is calculated as its value. Taking test case A as an example, first determine the accessed program path L Figure 3 corresponding to test case A in the control flow graph shown in 0-8 . For each node (root node 0, node 1, node 5, node 8) contained in the accessed program path L 0-8 , the unvisited neighbor nodes corresponding to each node are determined respectively. For details, refer to the relevant description in the foregoing embodiments, which are not repeated here. The determination result is: node 4, node 9. Among them, node 4 is a non-leaf node, and node 9 is a leaf node.

[0128] For leaf node 9, an unvisited program path L 0-9 from root node 0 to leaf node 9 is directly determined.

[0129] And for non-leaf node 4, it needs to continue to explore in the direction of its child nodes until it encounters leaf node 7, thereby determining an unvisited program path L 0-7 corresponding to node 4.

[0130] Then, optionally, in the control flow graph, the program paths other than the accessed program path L 0-8 , the unvisited program path L 0-9 and the unvisited program path L 0-7 are deleted, and each accessed node in the accessed program path L 0-8 , node 0, node 1, node 5, node 8, are deleted. The number of remaining nodes = 3, which is the value of test case A.

[0131] The application of one or more embodiments of the application will be described in detail below. Those skilled in the art can understand that these devices can be configured by using commercially available hardware components through the steps taught by the present application.

[0132] Figure 8 The structure of an application hybrid test device provided by an embodiment of the application is shown in Figure 8 , which comprises a first test module 11, a path determination module 12 and a second test module 13.

[0133] The first test module 11 is configured to run a plurality of test cases generated by the fuzz testing method in the application program to be tested, and determine access probabilities of a plurality of unvisited paths in the application program.

[0134] The path determination module 12 is configured to filter a target unvisited program path from the plurality of unvisited program paths according to the access probabilities of the plurality of unvisited program paths, wherein the access probability of the target unvisited program path is lower than that of an unvisited program path not filtered.

[0135] The second test module 13 is configured to perform symbolic execution on the target unvisited path, obtain test cases generated by the symbolic execution, and continue testing the application program based on the test cases generated by the symbolic execution.

[0136] Figure 8 The apparatus can perform the steps in the foregoing embodiments, and the detailed execution process and technical effects can be referred to the descriptions in the foregoing embodiments, which will not be repeated here.

[0137] In one possible design, the above Figure 8 The apparatus can be implemented as an electronic device. As shown in the figure, Figure 9 The electronic device can include a processor 21, a memory 22, and a communication interface 23. The memory 22 stores executable code. When the executable code is executed by the processor 21, the processor 21 can at least implement the hybrid testing method of the application program provided in the foregoing embodiments.

[0138] In addition, an embodiment of the present application provides a non-transitory machine readable storage medium, which stores executable code. When the executable code is executed by a processor of an electronic device, the processor can at least implement the hybrid testing method of the application program provided in the foregoing embodiments.

[0139] The apparatus embodiments described above are merely schematic, and the network elements illustrated as separate components can or can not be physically separated. Part or all of the modules can be selected according to actual needs to achieve the purposes of the embodiments. Those skilled in the art can understand and implement the same without creative effort.

[0140] Through the description of the above embodiments, those skilled in the art can clearly understand that the embodiments can be implemented by means of the necessary general hardware platform, and of course can also be implemented by means of the combination of hardware and software. Based on such understanding, the above technical solutions can be embodied in the form of a computer product, and the present application can be in the form of a computer program product implemented on one or more computer usable storage media (including but not limited to disk storage, optical storage, etc.) containing computer usable program codes.

[0141] Finally, it should be noted that: the above examples are only used to illustrate the technical solutions of the present application, and not to limit them; although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that: it can still modify the technical solutions recorded in the foregoing embodiments, or make equivalent replacement for part of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A hybrid testing method for an application, characterized in that, include: Run multiple test cases generated by fuzz testing in the application under test to determine the access probability of multiple unvisited paths in the application. Based on the access probability of the multiple unaccessed program paths, a target unaccessed program path is selected from the multiple unaccessed program paths, wherein the access probability of the target unaccessed program path is lower than the access probability of the unaccessed program paths that were not selected. Perform symbolic execution on the unvisited path of the target and obtain the test cases generated by the symbolic execution. The application is further tested based on the test cases generated by the symbol execution.

2. The method according to claim 1, characterized in that, Determining the access probability of the multiple test cases for multiple unaccessed paths in the application includes: Identify the multiple accessed program paths of the test cases within the application. By instrumenting counting code in the application, the number of times multiple program branches in the application are accessed after the multiple test cases are run is determined. The counting code is used to count the number of times program branches contained in each program path in the application are accessed. Based on the access counts of the multiple accessed program paths and the multiple program branches, determine the multiple unaccessed program paths in the application and the access probability of the multiple unaccessed program paths.

3. The method according to claim 2, characterized in that, The process of running multiple test cases generated using fuzz testing methods in the application under test includes: During the compilation phase of the application, the abstract syntax tree of the application is determined; The abstract syntax tree is transformed into an intermediate representation that matches the compiler of the application. The counting code is instrumented in the intermediate representation; The intermediate representation of the counting code described by the instrumentation is compiled to obtain the compiled executable file; The executable file runs the multiple test cases generated by the fuzzing method.

4. The method according to claim 2, characterized in that, The method further includes: Determine the abstract syntax tree of the application; Based on the abstract syntax tree, a control flow graph corresponding to the application is generated, wherein the nodes and directed edges in the control flow graph represent the basic blocks in the application and the transitions between different basic blocks, and the basic blocks connected by a directed edge constitute a program branch. The program paths in the application are determined based on the control flow graph.

5. The method according to claim 4, characterized in that, The process of determining multiple unaccessed program paths in the application includes: For any of the multiple visited program paths, determine each unvisited neighbor node corresponding to multiple nodes in the multiple visited program paths, wherein any node has the same parent node as its neighbor nodes, and the multiple visited program paths are determined based on the control flow graph. Determine the unvisited program paths formed between the root node and each of its neighboring nodes, wherein the root node is the root node in the control flow graph.

6. The method according to claim 5, characterized in that, Determining the access probability of the multiple unaccessed program paths includes: For any unaccessed program path among the multiple unaccessed program paths, determine each program branch contained in that unaccessed program path; For the first program branch in each program branch contained in any unaccessed program path, determine each second program branch adjacent to the first program branch, wherein the parent node in each second program branch is the same as the parent node in the first program branch, and the first program branch is any one of the program branches contained in any unaccessed program path. Determine the total number of accesses to the first program branch and each of the second program branches; The access probability of the first program branch is determined as the ratio of the number of times the first program branch is accessed to the total number of accesses; The access probability of any unvisited program path is determined based on the access probability of each program branch contained in any unvisited program path.

7. The method according to claim 4, characterized in that, The method further includes: Determine the value of the target test cases that have been executed, the value being used to reflect the number of nodes that were not visited according to the target test cases; If the value of the target test case meets the set conditions, the target test case is mutated using the fuzzing method to generate a new test case, and the application is then tested based on the new test case.

8. The method according to claim 7, characterized in that, Determining the value of the target test cases that have been executed includes: Determine the first accessed program path corresponding to the target test case in the control flow graph; Identify each unvisited neighbor node corresponding to multiple nodes in the first visited program path, wherein a node has the same parent node as its neighbor nodes; For any neighboring node, if the neighboring node is a leaf node, then it is determined that an unvisited program path corresponding to the neighboring node is formed between the root node and the neighboring node; if the neighboring node is not a leaf node, then it is determined that an unvisited program path corresponding to the neighboring node is formed between the root node and the leaf node under the neighboring node, wherein the root node is the root node in the control flow graph. In the control flow graph, delete the program paths except for the first visited program path and the unvisited program paths corresponding to each neighbor node, and delete each visited node in the first visited program path. The value of the target test case is determined based on the number of remaining nodes.

9. The method according to claim 2, characterized in that, The test cases generated by obtaining symbolic execution include: From the multiple accessed program paths, determine a second accessed program path that has more common prefixes with the target unaccessed program path; Determine the first constraint condition corresponding to the program branch with the common prefix in the second visited program path, and the second constraint condition corresponding to the remaining program branch in the target unvisited program path, wherein the remaining program branch refers to the program branch remaining in the target unvisited program path other than the program branch with the common prefix. The first constraint and the second constraint constitute a path constraint for symbolic execution; Solve the path constraints to obtain test cases corresponding to the unvisited program paths of the target. In the process of solving the path constraints, assign values ​​to the corresponding symbolic variables with the parameter values ​​from the test cases that have visited the second visited program paths. The parameter values ​​satisfy the first constraint condition.

10. The method according to claim 1, characterized in that, The test cases generated based on the symbolic execution are used to continue testing the application, including: The test cases generated by the symbolic execution are mutated using the fuzzing method to generate new test cases; Run the new test case in the application.

11. The method according to any one of claims 1-10, characterized in that, The application includes the P4 program.

12. The method according to claim 11, characterized in that, The method further includes: The data packets actually transmitted by the network device running the P4 program are input into the fuzzing method so that the fuzzing method modifies the values ​​of the parameter fields in the data packets to obtain data packets as test cases.

13. An electronic device, characterized in that, include: The device includes a memory, a processor, and a communication interface; wherein the memory stores executable code, which, when executed by the processor, causes the processor to perform a hybrid testing method for the application as described in any one of claims 1 to 12.

14. A non-transitory machine-readable storage medium, characterized in that, The non-transitory machine-readable storage medium stores executable code that, when executed by a processor of an electronic device, causes the processor to perform a hybrid testing method for the application as described in any one of claims 1 to 12.

15. A computer program product, characterized in that, include: A computer program, when executed by a processor of an electronic device, causes the processor to perform a hybrid testing method for an application as described in any one of claims 1 to 12.

Images