Test Method and Device for Application

Abstract

Embodiments of this specification provide a method and apparatus for testing an application program. The method includes: obtaining input data that conforms to the input format of the application program to be tested; mutating the input data to generate bad data that conforms to the input format of the application program; inputting the bad data into the application program to be tested so that the application program to be tested runs according to the bad data; collecting each memory address executed by the CPU during the running of the application program to be tested; generating a running path of the application program to be tested corresponding to the bad data according to the memory addresses; and determining the performance of the application program to be tested according to the running path. Embodiments of this specification can more effectively test an application program.

Description

Technical Field

[0001] One or more embodiments of this specification relate to computer technology, and in particular, to a method and apparatus for testing application programs. Background Art

[0002] With the rapid development of the network, various business applications have emerged based on the network. Users only need to download the application program (APP) of the corresponding business application in the terminal device, and then they can implement the corresponding business functions through the APP. For example, watching movies, chatting on WeChat, or purchasing goods.

[0003] In order to ensure the normal operation of the application program, it is necessary to test the application program to discover vulnerabilities or defects in the application program. Currently, when testing an application program, it is usually to obtain the feedback information output by the application program for an input, and determine the vulnerabilities or defects of the application program by verifying the feedback information.

[0004] However, in many business scenarios, it may not be possible to obtain the feedback information sent by an application program during the test process, resulting in the inability to effectively test the application program. Summary of the Invention

[0005] One or more embodiments of this specification describe a method and apparatus for testing application programs, which can more effectively test application programs.

[0006] According to a first aspect, there is provided a method for testing an application program, which includes:

[0007] For the application program to be tested, obtain input data that conforms to the input format of the application program;

[0008] Mutate the input data to generate bad data that conforms to the input format of the application program;

[0009] Input the bad data into the application program to be tested, so that the application program to be tested runs according to the bad data;

[0010] Collect each memory address executed by the CPU during the running of the application program to be tested;

[0011] Generate a running path of the application program to be tested corresponding to the bad data according to the memory addresses;

[0012] Determine the performance of the application program to be tested according to the running path.

[0013] Wherein, the application program to be tested is the TA to be tested in the TEE.

[0014] Among them, inputting the bad data into the application to be tested includes: initiating an SMC instruction through the SMC driver in the REE, causing the CPU to switch to execute the TA to be tested, and the client program CA in the REE passing the bad data into the TA to be tested.

[0015] Among them, collecting each memory address executed by the CPU during the operation of the application to be tested includes:

[0016] When the TA to be tested starts running, make the CPU executing the TA to be tested enter the debug mode and debug the CPU;

[0017] During the debugging process, read and write the memory on the TEE side;

[0018] Traverse the known memory range to find the known memory pages of the target ELF file of the TA to be tested;

[0019] According to the found known memory pages, determine the target address range in the memory corresponding to the TA to be tested;

[0020] Within the target address range, collect each memory address executed by the CPU during the operation of the application to be tested.

[0021] Among them, determining the running path of the application to be tested according to the memory address includes:

[0022] According to the order of each memory address executed by the CPU, sequentially read the instructions executed by the application to be tested from each of these memory addresses;

[0023] Generate the running path of the application to be tested according to the sequentially read instructions.

[0024] Among them, execute at least two rounds: from the step of obtaining the input data that conforms to the input format of the application to the step of generating the running path of the application to be tested corresponding to the bad data, until all the running paths of the application to be tested are traversed;

[0025] Mutating the input data includes: determining the mutation method of the input data in this round according to the running path of the application to be tested generated in the previous round, and mutating the input data according to the determined mutation method;

[0026] Determining the performance of the application to be tested includes: determining the performance of the application to be tested according to at least two running paths obtained in at least two rounds.

[0027] According to the second aspect, a test device for an application is provided. Among them, it includes:

[0028] An input data acquisition module, configured to obtain input data conforming to the input format of the application under test for the application under test;

[0029] A mutation module, configured to mutate the input data to generate bad data conforming to the input format of the application;

[0030] A bad data input module, configured to input the bad data into the application under test, so that the application under test runs according to the bad data;

[0031] An address acquisition module, configured to acquire each memory address executed by the CPU during the running of the application under test;

[0032] A running path generation module, configured to generate a running path of the application under test corresponding to the bad data according to the memory addresses;

[0033] A performance acquisition module, configured to determine the performance of the application under test according to the running path.

[0034] Wherein, the address acquisition module is configured to execute:

[0035] When the application under test TA starts running, make the CPU executing the application under test TA enter the debug mode and debug the CPU;

[0036] During the debugging process, read and write the memory on the TEE side;

[0037] Traverse the known memory range to find the known memory pages of the target ELF file of the application under test TA;

[0038] According to the found known memory pages, determine the target address range in the memory corresponding to the application under test TA;

[0039] Within the target address range, acquire each memory address executed by the CPU during the running of the application under test.

[0040] Wherein, the running path generation module is configured to execute:

[0041] According to the order of each memory address executed by the CPU, sequentially read the instructions executed by the application under test from each memory address;

[0042] According to the sequentially read instructions, generate a running path of the application under test.

[0043] According to a third aspect, a computing device is provided, including a memory and a processor. Executable code is stored in the memory. When the processor executes the executable code, the method described in any embodiment of this specification is implemented.

[0044] When testing a to-be-tested application program, the method and device provided in the embodiments of this specification do not need to obtain the feedback information regarding the testing process sent by the application program during the testing process. That is to say, there is no need to perform testing based on this feedback information. Instead, first, each memory address executed by the CPU during the running process of the to-be-tested application program is obtained, and then the running path of the to-be-tested application program is generated based on each memory address, that is, the execution process of the to-be-tested application program for the current input. In this way, according to such a running path / execution process, after multiple rounds of testing, the execution processes of the to-be-tested application program under various inputs can be obtained, and thus the performance of the to-be-tested application program can be determined. It can be seen that the method of the embodiments of this specification can more effectively test the to-be-tested application program. Description of the Drawings

[0045] To more clearly illustrate the technical solutions in the embodiments of this specification or the prior art, the following will briefly introduce the drawings required for use in the description of the embodiments or the prior art. Obviously, the following drawings are some embodiments of this specification. For those of ordinary skill in the art, without creative efforts, other drawings can also be obtained based on these drawings.

[0046] Figure 1 It is a flowchart of a method for testing an application program in an embodiment of this specification.

[0047] Figure 2 It is a schematic diagram of the composition of a testing system applied to a to-be-tested TA in an embodiment of this specification.

[0048] Figure 3 It is a flowchart of a method for collecting each memory address executed by the CPU during the running process of a to-be-tested TA in an embodiment of this specification.

[0049] Figure 4 It is a schematic diagram of the structure of a testing device for an application program in an embodiment of this specification. Detailed Embodiments

[0050] As described above, in many business scenarios, it may not be possible to obtain the feedback information sent by an application during the testing process, resulting in the inability to effectively test the application. For example, for a trusted application (TA) set in a trusted execution environment (TEE), since the TA runs inside the TEE, even if an input for testing is sent to the TA, it is impossible to obtain the feedback information of the TA for this input during the testing process. Therefore, the application cannot be effectively tested by using the feedback information method.

[0051] The solution provided in this specification will be described below with reference to the accompanying drawings.

[0052] Figure 1 It is a flowchart of a method for testing an application in an embodiment of this specification. The execution subject of this method is a testing device for the application. It can be understood that this method can also be executed by any device, equipment, platform, or device cluster with computing and processing capabilities. Refer to Figure 1 This method includes:

[0053] Step 101: For the application to be tested, obtain input data that conforms to the input format of the application.

[0054] Step 103: Mutate the input data to generate bad data that conforms to the input format of the application.

[0055] Step 105: Input the generated bad data into the application to be tested, so that the application to be tested runs according to the bad data.

[0056] Step 107: Collect each memory address executed by the CPU during the running process of the application to be tested.

[0057] Step 109: Generate the running path of the application to be tested for the current bad data according to the memory addresses.

[0058] Step 111: Determine the performance of the application to be tested according to the running path.

[0059] According to the above Figure 1As can be seen from the process shown, when testing the application to be tested, there is no need to obtain the feedback information sent by the application during the testing process for the testing process. That is to say, there is no need to perform testing based on this feedback information. Instead, first, obtain each memory address executed by the CPU during the operation of the application to be tested, and then generate the running path of the application to be tested based on the data in each memory address, that is, the execution process of the application to be tested for the current input. In this way, according to this running path / execution process, after multiple rounds of testing, various execution processes of the application to be tested under various inputs can be obtained, and thus the performance of the application to be tested can be determined. It can be seen that the method of the embodiment of this specification can more effectively test the application to be tested.

[0060] The following will describe each step in Figure 1 in combination with specific embodiments.

[0061] First, for step 101:

[0062] For the application to be tested, obtain input data that conforms to the input format of the application.

[0063] For various applications, the formats of their input data may be different. Therefore, it is necessary to obtain corresponding legal input data according to the application to be tested currently.

[0064] Next, for step 103: Mutate the input data to generate bad data that conforms to the input format of the application.

[0065] In the embodiment of this specification, by providing unexpected input, that is, bad data, to the application to be tested and monitoring the abnormal results, software vulnerabilities of the application to be tested can be discovered.

[0066] There are various ways to mutate the input data. For example, replace a part of the bytes in the legal input data, or add or subtract a part of the bytes in the legal input data, etc.

[0067] Next, for step 105: Input the bad data into the application to be tested so that the application to be tested runs according to the bad data.

[0068] In an embodiment of this specification, the application to be tested may be the TA to be tested in the TEE.

[0069] It can be understood that because the TA to be tested is in the TEE environment, the external test program cannot directly access the TA. Therefore, refer to Figure 2, in this step 105, an SMC instruction can be initiated through the SMC driver in the REE (Rich Execution Environment), causing the CPU to switch to execute the TA under test, and the client program (CA) in the REE passes the bad data into the TA under test.

[0070] Next, for step 107: Collect each memory address executed by the CPU during the operation of the application under test.

[0071] When the application under test is the TA under test in the TEE, in order to further improve the collection efficiency, in an embodiment of this specification, the target address range corresponding to the TA under test in the memory can be determined first, and then each memory address executed by the CPU is collected within this target address range, thus avoiding blind collection among numerous memory addresses. At this time, refer to Figure 3 , the specific implementation process of this step 107 includes:

[0072] Step 1071: When the TA under test starts running, make the CPU executing the TA under test enter the debug mode and debug this CPU.

[0073] In an embodiment of this specification, the processes of step 1071 to step 1079 can be implemented through the CoreSight component.

[0074] For an ARM multi-core SoC, the CTI (CrossTrigger Interface) in the CoreSight component can be triggered on one CPU to make another CPU enter the debug mode, so as to debug it. Therefore, in this step 1071, when the TA under test starts running, the CPU executing the TA under test can be made to enter the debug mode and this CPU can be debugged.

[0075] Step 1073: During the debugging process, read and write the memory on the TEE side.

[0076] In the existing technology, it is impossible to read and write the memory on the TEE side. In the embodiment of this specification, when the CPU executing the TA under test enters the debug mode, the memory on the TEE side can be read and written during the debugging process.

[0077] Step 1075: Traverse the known memory range to find the known memory pages of the target ELF file corresponding to the TA under test.

[0078] Step 1077: According to the found known memory page, determine the target address range in the memory corresponding to the TA under test.

[0079] Step 1079: Within the target address range, collect each memory address executed by the CPU during the operation of the application under test.

[0080] See Figure 2 , in this step 1079, each memory address executed by the CPU during the operation of the TA under test can be collected through the ETM (Embedded Trace Macrocell) in the CoreSight component. Specifically, the ETM component can set filtering options through configuration registers to collect the memory addresses executed by a specific CPU, that is, the CPU running the application under test.

[0081] Next, for step 109:

[0082] Generate the running path of the application under test for the current input of bad data based on the collected memory addresses executed by the CPU.

[0083] In an embodiment of this specification, the specific implementation process of step 109 includes: sequentially read the instructions executed by the application under test from each of these memory addresses according to the order of the memory addresses executed by the CPU; generate the running path of the application under test for the current bad data based on the sequentially read instructions.

[0084] For example, the application under test includes 10 pairs of "if else" programming languages. That is to say, there are a total of 100 execution flow branches, that is, 100 running paths. When a bad data is input in step 105, the application under test runs for this bad data and may execute 10 of these flow branches. In step 109, a running path for the current bad data is generated, and this running path includes 10 flow branches. This kind of running path reflects the running situation of the application under test for the current bad data. Therefore, the performance of the application under test in this case can be tested.

[0085] Next, for step 111: Determine the performance of the application under test based on the generated running path.

[0086] As mentioned above, for a kind of bad data, in step 109, a running path of the application under test can be generated. Usually, this running path can only cover a part of all the paths of the application under test. Therefore, in order to better complete the test and perform more path coverage or even full path coverage, in an embodiment of this specification, a fuzz testing method can be adopted to execute at least two rounds of the above Figure 1The process from step 101 to step 109 until all the running paths of the application under test are traversed. In each round of testing, based on the running path of the application under test generated in the previous round, determine the mutation method for the input data in this round, and mutate the input data according to the determined mutation method; finally, comprehensively determine the performance of the application under test based on at least two running paths obtained in at least two rounds.

[0087] For example, in the previous round, the mutation method was: replace a part of the bytes in the input data to obtain bad data. Then, in this round, the mutation method can continue to be: replace another part of the bytes in the input data, and observe the running path generated in this round. If the running path generated in this round is roughly the same as the running path generated in the previous round, it means that the mutation method needs to be changed. For example, in the next round, the mutation method can no longer be the way of replacing bytes, but instead add some bytes to the input data to obtain bad data; of course, if the running path generated in this round is very different from the running path generated in the previous round, the way of replacing bytes can be continued in the next round for mutation.

[0088] After obtaining multiple running paths through multiple rounds of testing, the full path of the application under test can be covered, so as to test whether the application under test crashes and whether there are execution errors, etc. when facing bad data for various inputs.

[0089] In an embodiment of this specification, a test device for an application is provided. Refer to Figure 4 This device includes:

[0090] An input data acquisition module 401, configured to obtain input data that conforms to the input format of the application under test for the application under test;

[0091] A mutation module 402, configured to mutate the input data to generate bad data that conforms to the input format of the application under test;

[0092] A bad data input module 403, configured to input the bad data into the application under test so that the application under test runs according to the bad data;

[0093] An address acquisition module 404, configured to acquire each memory address executed by the CPU during the running of the application under test;

[0094] A running path generation module 405, configured to generate the running path of the application under test for the bad data according to the memory addresses;

[0095] A performance acquisition module 406, configured to determine the performance of the application under test according to the running path.

[0096] In one embodiment of the device in this specification, the application under test is the TA under test in the TEE;

[0097] Correspondingly, the bad data input module 403 is configured to execute: initiate an SMC instruction through the SMC driver in the REE, so that the CPU switches to execute the TA under test, and the client program CA in the REE passes the bad data into the TA under test.

[0098] In one embodiment of the device in this specification, the address acquisition module 404 is configured to execute:

[0099] When the TA under test starts running, make the CPU executing the TA under test enter the debug mode and debug the CPU;

[0100] During the debugging process, read and write the memory on the TEE side;

[0101] Traverse the known memory ranges to find the known memory pages of the target ELF file of the TA under test;

[0102] According to the found known memory page, determine the target address range in the memory corresponding to the TA under test;

[0103] Within the target address range, collect each memory address executed by the CPU during the running of the application under test.

[0104] In one embodiment of the device in this specification, the running path generation module 405 is configured to execute:

[0105] According to the order of each memory address executed by the CPU, sequentially read the instructions executed by the application under test from each of these memory addresses;

[0106] Generate the running path of the application under test according to the sequentially read instructions.

[0107] In one embodiment of the device in this specification, at least two rounds of tests are performed, and the performance acquisition module 406 is configured to execute: determine the performance of the application under test according to at least two running paths obtained in at least two rounds of tests;

[0108] The mutation module 402 is configured to execute: determine the mutation method of the input data in this round according to the running path of the application under test generated in the previous round, and mutate the input data according to the determined mutation method.

[0109] One embodiment of this specification provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed on a computer, the computer is made to execute the method in any one of the embodiments in this specification.

[0110] One embodiment of this specification provides a computing device, including a memory and a processor. An executable code is stored in the memory. When the processor executes the executable code, the method in any one of the embodiments in this specification is implemented.

[0111] It can be understood that the structure illustrated in the embodiments of this specification does not constitute a specific limitation on the devices in the embodiments of this specification. In other embodiments of this specification, the above-mentioned device may include more or fewer components than those illustrated, or combine certain components, or split certain components, or have different component arrangements. The illustrated components can be implemented in hardware, software, or a combination of software and hardware.

[0112] Regarding the information interaction, execution process, etc. among the various modules within the above-mentioned device and system, since they are based on the same concept as the method embodiments of this specification, the specific content can be referred to the descriptions in the method embodiments of this specification and will not be elaborated here.

[0113] The various embodiments in this specification are all described in a progressive manner. The same or similar parts among the various embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the device embodiments, since they are basically similar to the method embodiments, the descriptions are relatively simple, and the relevant parts can be referred to the partial descriptions of the method embodiments.

[0114] Those skilled in the art should be able to realize that in the above one or more examples, the functions described in the present invention can be implemented by hardware, software, add-ons, or any combination thereof. When implemented using software, these functions can be stored in a computer-readable medium or transmitted as one or more instructions or codes on a computer-readable medium.

[0115] The above-mentioned specific implementation manners further elaborate the purpose, technical solution, and beneficial effects of the present invention. It should be understood that the above is only the specific implementation manner of the present invention and is not used to limit the protection scope of the present invention. Any modification, equivalent replacement, improvement, etc. made on the basis of the technical solution of the present invention shall be included in the protection scope of the present invention.

Claims

1. A test method for an application, wherein, Including: For the application to be tested, obtain input data that conforms to the input format of the application; Mutate the input data to generate bad data that conforms to the input format of the application; Input the bad data into the application to be tested, so that the application to be tested runs according to the bad data; Collect each memory address executed by the CPU during the operation of the application to be tested; Generate the running path of the application to be tested for the bad data according to the memory addresses; Determine the performance of the application to be tested according to the running path; Wherein, the application to be tested is the TA to be tested in the TEE; Wherein, the collection of each memory address executed by the CPU during the operation of the application to be tested includes: When the TA to be tested starts running, make the CPU executing the TA to be tested enter the debug mode and debug the CPU; During the debugging process, read and write the memory on the TEE side; Traverse the known memory range to find the known memory pages of the target ELF file of the TA to be tested; According to the found known memory page, determine the target address range in the memory corresponding to the TA to be tested; Within the target address range, collect each memory address executed by the CPU during the operation of the application to be tested.

2. The method according to claim 1, wherein Inputting the bad data into the application to be tested includes: initiating an SMC instruction through the SMC driver in the REE, causing the CPU to switch to execute the TA to be tested, and the client program CA in the REE passes the bad data into the TA to be tested.

3. The method according to claim 1, wherein The determining the running path of the application to be tested according to the memory address includes: According to the order of each memory address executed by the CPU, sequentially read the instructions executed by the application to be tested from each memory address; Generate the running path of the application to be tested according to the sequentially read instructions.

4. The method according to claim 1, wherein, Execute at least two rounds: from the step of obtaining the input data that conforms to the input format of the application to the step of generating the running path of the application to be tested corresponding to the bad data, until all running paths of the application to be tested are traversed; The mutating the input data includes: determining the mutation method of the input data in this round according to the running path of the application to be tested generated in the previous round, and mutating the input data according to the determined mutation method; The determining the performance of the application to be tested includes: determining the performance of the application to be tested according to at least two running paths obtained in at least two rounds.

5. Test device for an application, wherein, Including: An input data acquisition module configured to obtain input data that conforms to the input format of the application to be tested for the application to be tested; A mutation module configured to mutate the input data to generate bad data that conforms to the input format of the application; A bad data input module configured to input the bad data into the application to be tested, so that the application to be tested runs according to the bad data; An address acquisition module, configured to acquire each memory address executed by the CPU during the operation of the application under test. A running path generation module, configured to generate a running path of the application under test corresponding to the bad data according to the memory addresses. A performance acquisition module, configured to determine the performance of the application under test according to the running path. Wherein, the application under test is a TA under test in the TEE. Wherein, the address acquisition module is configured to execute: When the TA under test starts to run, cause the CPU executing the TA under test to enter the debug mode and debug the CPU. During the debugging process, read and write the memory on the TEE side. Traverse the known memory range to find the known memory pages of the target ELF file of the TA under test. According to the found known memory pages, determine the target address range in the memory corresponding to the TA under test. In the target address range, acquire each memory address executed by the CPU during the operation of the application under test.

6. The device according to claim 5, wherein, The running path generation module is configured to execute: According to the order of each memory address executed by the CPU, sequentially read the instructions executed by the application under test from each memory address. Generate a running path of the application under test according to the sequentially read instructions.

7. A computing device, including a memory and a processor, wherein executable code is stored in the memory, and when the processor executes the executable code, the method according to any one of claims 1-4 is implemented.

Images