Method and device for discovering defects in program testing, medium and equipment

Abstract

The embodiment of the invention provides a method and device for discovering defects in program testing, a medium and equipment. A specific embodiment of the method comprises the following steps: acquiring crash data of running crash generated when an online version of a tested program runs online; obtaining test data of a target unit test case which fails to be executed in the offline test of the current version of the tested program; and determining whether a program unit corresponding to the target unit test case has a defect or not according to the test data and the crash data. According to the method, comprehensive analysis can be performed based on the online crash data and the offline test data, so that the defects existing in the tested program can be accurately screened out.

Description

Technical Field

[0001] The embodiments of the present disclosure relate to the field of automated testing technology, and more particularly to a method and device for defect discovery in program testing. Background Art

[0002] With the development of computer technology, software programs cover a wider range of industries, and programs with various functions are developed. In the process of iterative program development, testing the program code is a very important link. Code testing can effectively ensure the correctness of program operation. How to find defects in the program during program testing is a problem that the testing industry is very concerned about. Summary of the invention

[0003] The embodiments of the present disclosure describe a method and apparatus for defect discovery in program testing, which can be based on comprehensive analysis of online crash data and offline test data, thereby accurately screening out defects in the tested program.

[0004] According to a first aspect, a method for defect discovery in program testing is provided, including: obtaining crash data of an operation crash generated when an online version of a tested program is running online; obtaining test data of a target unit test case in which the current version of the tested program fails to execute in an offline test; and determining whether a program unit corresponding to the target unit test case has a defect based on the test data and the crash data.

[0005] In one embodiment, the method further includes: performing offline analysis on the test data to determine whether the program unit corresponding to the target unit test case has defects. Thus, the offline test data is analyzed offline to screen out defects.

[0006] In one embodiment, the test data includes a stack trace; and the offline analysis of the test data to determine whether the program unit corresponding to the target unit test case has a defect includes: matching the stack trace of the target unit test case with a first rule in a preset first rule set, wherein the first rule in the first rule set is determined based on analyzing the stack traces of multiple program units with abnormalities; in response to determining that the stack trace of the target unit test case matches the first rule in the first rule set, determining that the program unit corresponding to the target unit test case has a defect. Thus, through the pre-created first rule set, it is possible to quickly determine that the program unit corresponding to the target unit test case whose stack trace matches the first rule has a defect.

[0007] In one embodiment, the test data includes a stack trace; and the offline analysis of the test data to determine whether the program unit corresponding to the target unit test case has defects includes: matching the stack trace of the target unit test case with a second rule in a preset second rule set, wherein the second rule in the second rule set is determined based on analyzing the stack traces of multiple program units that do not have exceptions; in response to determining that the stack trace of the target unit test case matches the second rule in the second rule set, determining that the program unit corresponding to the target unit test case does not have defects. Thus, through the pre-created second rule set, it can be quickly determined that the program unit corresponding to the target unit test case whose stack trace matches the second rule does not have defects.

[0008] In one embodiment, the above-mentioned tested program is developed iteratively, and a new version is generated each time through iteration. The above-mentioned target unit test case is automatically generated by the test tool based on the code of the test case generation version; and the above-mentioned offline analysis of the above-mentioned test data to determine whether the program unit corresponding to the above-mentioned target unit test case has defects includes: determining whether the above-mentioned target unit test case fails to execute in the code of the above-mentioned test case generation version; in response to determining that the above-mentioned target unit test case fails to execute in the code of the above-mentioned test case generation version, determining to filter the above-mentioned target unit test case, and no longer judging whether the program unit corresponding to the above-mentioned target unit test case has defects. Thus, the target unit test cases that fail to execute in the code of the test case generation version can be filtered out, thereby reducing unnecessary waste of computing resources.

[0009] In one embodiment, the above-mentioned offline analysis of the above-mentioned test data to determine whether the program unit corresponding to the above-mentioned target unit test case has defects includes: in response to determining that the above-mentioned target unit test case is successfully executed in the code of the above-mentioned test case generation version, determining the file changes between the above-mentioned test case generation version and the above-mentioned current version, and the first call chain of the first change file; in response to determining that the code file corresponding to the above-mentioned target unit test case is related to the above-mentioned first change file, or is on the above-mentioned first call chain, determining that the program unit corresponding to the above-mentioned target unit test case has defects. In this way, based on the changes between the test case generation version and the current version, defective program units can be screened out.

[0010] In one embodiment, the method further includes: in response to determining that the program unit corresponding to the target unit test case has defects, grading the defects, thereby achieving priority grading of the screened defects.

[0011] In one embodiment, the above-mentioned defect classification includes: in response to determining that the program unit corresponding to the above-mentioned target unit test case has a defect according to the above-mentioned test data and the above-mentioned crash data, determining the defect as a first priority defect; in response to determining that the code block corresponding to the code line reporting an error in the source code of the program unit corresponding to the defect has been modified in the above-mentioned current version, determining the defect as a first priority defect. In this way, the defect classification can be achieved, thereby classifying the defects into different priorities.

[0012] In one embodiment, the above-mentioned defect classification also includes: in response to determining that the code block corresponding to the code line reporting an error in the source code of the program unit corresponding to the defect has not been modified in the above-mentioned current version, determining the file changes between the first version and the above-mentioned current version, and the second call chain of the second change file, wherein the above-mentioned current version is a version iterated based on the above-mentioned first version; in response to determining that the code file corresponding to the above-mentioned target unit test case is related to the above-mentioned second change file, or is on the above-mentioned second call chain, determining that the defect is a second priority defect, wherein the priority of the first priority defect is higher than that of the second priority defect. In this way, the classification of defects can be achieved, thereby dividing the defects into different priorities.

[0013] In one embodiment, the test data includes the tested class, the tested exception type, and the tested method; the crash data includes the crashed class, the crashed exception type, the crashed method, and the crash identifier; and the determination of whether the program unit corresponding to the target unit test case has a defect based on the test data and the crash data includes: in response to determining that the crash data matches the test data, determining that the crash corresponding to the matched crash data is a target crash; according to the crash identifier of the target crash, obtaining the stack trace corresponding to the target crash; matching the stack trace corresponding to the target crash with the stack trace corresponding to the target unit test case; in response to determining that the match is made, determining that the program unit corresponding to the target unit test case has a defect. Thus, defects can be determined based on crash data, test data, and stack traces, thereby making the determined defects more accurate.

[0014] In one embodiment, the above determination that the crash data matches the above test data includes: in response to determining that the class of the above test and the exception type of the above test are respectively the same as the class of the above crash and the exception type of the above crash, further determining whether the method of the above test is the same as the method of the above crash; if the method of the above test is the same as the method of the above crash, determining that the crash data matches the above test data. In this way, it is possible to quickly determine whether the crash data matches the test data.

[0015] According to a second aspect, a device for defect discovery in program testing is provided, including: a first acquisition unit, configured to acquire crash data of an operation crash generated when an online version of the tested program is running online; a second acquisition unit, configured to acquire test data of a target unit test case that fails to execute in an offline test of the current version of the tested program; and a determination unit, configured to determine whether a program unit corresponding to the target unit test case has a defect based on the test data and the crash data.

[0016] According to a third aspect, a computer program product is provided, comprising a computer program, wherein when the computer program is executed by a processor, the method as described above in any one of the first aspect is implemented.

[0017] According to a fourth aspect, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed in a computer, the computer is caused to execute any one of the methods in the first aspect.

[0018] According to a fifth aspect, an electronic device is provided, comprising a memory and a processor, wherein the memory stores executable code, and when the processor executes the executable code, any one of the methods in the first aspect is implemented.

[0019] According to the method and device for defect discovery in program testing provided by the embodiments of the present disclosure, crash data of the operation crash generated when the online version of the tested program is running online can be obtained, and test data of the target unit test case that failed to execute in the linear test of the current version of the tested program can also be obtained. Afterwards, based on the obtained test data and crash data, it is determined whether the program unit corresponding to the target unit test case has defects. In this way, it is achieved to accurately screen out defects in the tested program based on comprehensive analysis of online crash data and offline test data. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] Figure 1 A schematic diagram showing an application scenario in which the embodiment of the present disclosure can be applied;

[0021] Figure 2 A schematic flow chart of a method for finding defects in program testing according to an embodiment is shown;

[0022] Figure 3 A schematic flow chart of a method for finding defects in program testing according to yet another embodiment is shown;

[0023] Figure 4 A schematic diagram showing an example of a branch graph of a version control system;

[0024] Figure 5A schematic block diagram of a device for finding defects in program testing according to an embodiment is shown;

[0025] Figure 6 A schematic diagram of the structure of an electronic device suitable for implementing the embodiments of the present application is shown. DETAILED DESCRIPTION

[0026] It is understandable that before using the technical solutions disclosed in the embodiments of the present disclosure, the types, scope of use, usage scenarios, etc. of the personal information involved in the present disclosure should be informed to the user and the user's authorization should be obtained in an appropriate manner in accordance with relevant laws and regulations.

[0027] For example, in response to receiving an active request from a user, a prompt message is sent to the user to clearly prompt the user that the operation requested to be performed will require obtaining and using the user's personal information. Thus, the user can autonomously choose whether to provide personal information to software or hardware such as an electronic device, application, server, or storage medium that performs the operation of the technical solution of the present disclosure according to the prompt message.

[0028] As an optional but non-limiting implementation, in response to receiving an active request from the user, the prompt information may be sent to the user in the form of a pop-up window, in which the prompt information may be presented in text form. In addition, the pop-up window may also carry a selection control for the user to choose "agree" or "disagree" to provide personal information to the electronic device.

[0029] It is understandable that the above notification and the process of obtaining user authorization are merely illustrative and do not constitute a limitation on the implementation of the present disclosure. Other methods that meet the relevant laws and regulations may also be applied to the implementation of the present disclosure.

[0030] The technical solution provided by the present disclosure is further described in detail below in conjunction with the accompanying drawings and embodiments. It is to be understood that the specific embodiments described herein are only used to explain the relevant inventions, rather than to limit the inventions. It should also be noted that, for ease of description, only the parts related to the relevant inventions are shown in the accompanying drawings. It should be noted that, in the absence of conflict, the embodiments of the present disclosure and the features in the embodiments can be combined with each other.

[0031] The disclosed embodiments provide a method for defect discovery in program testing, which can combine online crash data and offline test data for comprehensive analysis, thereby accurately screening out defects in the tested program.

[0032] Figure 1 FIG. 1 is a schematic diagram showing an application scenario in which the embodiment of the present disclosure can be applied. Figure 1As shown, when the online version of the tested program is running online, APM (Application Performance Monitor, application performance monitoring software) can be used to capture the crash data 101 of the operation crash (crash) generated when the tested program is running online. For example, crash data within a period of time (for example, one week, one month, etc.) can be obtained. In this example, the tested program is developed in an iterative development manner, and a new version can be generated for each iteration. For the current version of the tested program, multiple unit test cases can be used for offline testing, and the unit test cases that failed to execute in the offline test are determined as target unit test cases. Get the test data 102 of the target unit test case. Afterwards, according to the test data 102 of the target unit test case and the crash data 101 of the online operation crash, determine whether the program unit corresponding to the target unit test case has defects. Thus, the online crash data and the offline test data are comprehensively analyzed to accurately screen out the defects in the tested program.

[0033] Continue to see Figure 2 , Figure 2 The flowchart of a method for finding defects in program testing according to an embodiment is shown. The method can be executed by any device, equipment, platform, or device cluster with computing and processing capabilities. Figure 2 As shown, the method for discovering defects in program testing may include the following steps:

[0034] Step 201 : Obtain crash data of a running crash generated when the online version of the tested program is running online.

[0035] In this embodiment, various application performance monitoring software can be used to capture the crash data of the operation crash generated when the online version of the tested program is running online. Here, the crash may refer to the phenomenon that an error or anomaly occurs when the program is running, resulting in failure to work normally. As an example, the crash data may include various data related to the crash, such as the crash class (crash_class), the crash method (crash_method), the crash exception type (crash_exception), the crash time, etc. In this way, the crash data can be obtained from the application performance monitoring software.

[0036] Step 202: Acquire test data of target unit test cases that failed to be executed in offline testing of the current version of the tested program.

[0037] In this embodiment, the program under test can be developed in an iterative development manner, and a new version can be generated for each iteration. For the current version of the program under test, unit test cases can be used for offline testing. Here, the current version of the program under test and the online version of the program under test can be the same or different. In this example, unit testing can refer to testing the smallest testable unit in the program. Unit test cases can refer to test cases for testing program units in the program. For example, unit test cases can be written by testers or automatically generated by intelligent unit test case automatic generation tools. During offline testing, multiple unit test cases can be used to test the current version of the program under test. Unit test cases are executed based on the program code of the current version of the program under test, and unit test cases that fail to execute are used as target unit test cases. In this way, test data of the target unit test case can be obtained. As an example, test data can include various data related to the test, such as the class of the test (test_class), the method of the test (test_method), the exception type of the test (test_exception), etc.

[0038] Step 203: Determine whether the program unit corresponding to the target unit test case has defects based on the test data and the crash data.

[0039] In this embodiment, it is possible to determine whether the program unit corresponding to the target unit test case has defects based on the online crash data and the offline test data. For example, for crash data of multiple crashes generated by online operation within a period of time (e.g., within a week, within a month, etc.), it is possible to query in the online crash data whether there is crash data that matches the test data of the target unit test case (e.g., with a high degree of similarity). If there is, it can be inferred that the program unit corresponding to the target unit test case has problems in both online operation and offline testing, and therefore, the program unit corresponding to the target unit test case may have defects.

[0040] In some implementations, the test data of the target unit test case may include the tested class, the tested exception type, the tested method, the tested stack trace, etc. The crash data of the operation crash generated when the online version is running online may include the crashed class, the crashed method, the crash exception type, the crash identifier, etc. Here, the crash identifier can be used to uniquely identify a run crash. Based on this, the above step 203 can be specifically performed as follows:

[0041] S1 , in response to determining that crash data matches test data, determining a crash corresponding to the matched crash data as a target crash.

[0042] In this implementation, if the crash data of a certain crash matches the test data of the target unit test case, the crash corresponding to the matched crash data is determined to be the target crash. In this example, the crash data of a certain crash matches the test data of the target unit test case, which may mean that the crash class, crash method, and crash exception type in the crash data of the crash are respectively the same as the test class, test method, and test exception type in the test data.

[0043] Optionally, you can verify that the crash data matches the test data by:

[0044] 1) Determine whether the test class and the test exception type in the test data of the target unit test case are respectively the same as the crash class and the crash exception type in the crash data of each crash.

[0045] If they are not the same, it is determined that the crash data of the crash does not match the test data of the target unit test case, and no subsequent judgment is required, thereby saving time and calculation amount; if they are the same, it is further determined whether the test method in the test data of the target unit test case is the same as the crash method in the crash data of the crash.

[0046] 2) If the method of the test in the test data of the target unit test case is the same as the method of the crash in the crash data of the crash, then it is determined that the crash data of the crash matches the test data of the target unit test case. Through this implementation, it is possible to quickly determine whether the crash data matches the test data.

[0047] S2, obtaining a stack trace corresponding to the target crash according to the crash identifier of the target crash.

[0048] In this implementation, the application performance monitoring software may associate and store the crash identifier of each captured target crash with the stack trace corresponding to the target crash. Here, the stack trace corresponding to the crash may refer to the description of the active stack frame information at the time when the crash occurs. In this way, the stack trace corresponding to the target crash may be obtained from the application performance monitoring software according to the crash identifier of the target crash.

[0049] S3, matches the stack trace corresponding to the target crash with the stack trace corresponding to the target unit test case.

[0050] In this implementation, first, the similarity between the stack trace corresponding to the target crash and the stack trace corresponding to the target unit test case can be calculated, for example, cosine similarity, Euclidean distance, etc. Then, it is determined whether the calculated similarity is greater than a preset threshold. If so, it is determined that the stack trace corresponding to the target crash matches the stack trace corresponding to the target unit test case; if not, it is determined that they do not match.

[0051] S4, in response to determining that there is a match, determining that a program unit corresponding to the target unit test case has a defect.

[0052] In this implementation, if the stack trace corresponding to the target crash matches the stack trace corresponding to the target unit test case, it can be determined that the program unit corresponding to the target unit test case has a defect. Through this implementation, defects can be determined based on crash data, test data and stack traces, making the determined defects more accurate.

[0053] Reviewing the above process, in the above embodiment of the present disclosure, the crash data of the operation crash generated when the online version of the tested program is running online can be obtained, and the test data of the target unit test case that failed to execute in the linear test of the current version of the tested program can also be obtained. Afterwards, based on the obtained test data and crash data, it is determined whether the program unit corresponding to the target unit test case has defects. In this way, it is achieved to accurately screen out the defects in the tested program based on the comprehensive analysis of online crash data and offline test data.

[0054] Further references Figure 3 , Figure 3 A flowchart of a method for finding defects in program testing according to another embodiment is shown. The method for finding defects in program testing may include the following steps:

[0055] Step 301 : Obtain crash data of a running crash generated when the online version of the tested program is running online.

[0056] In this embodiment, the specific implementation method of step 301 is the same as Figure 2 The process is similar to step 201 in , and will not be described again here.

[0057] Step 302: Acquire test data of target unit test cases that failed to be executed in offline testing of the current version of the tested program.

[0058] In this embodiment, the specific implementation method of step 302 is the same as Figure 2 The process is similar to step 202 in , and will not be described again here.

[0059] Step 303: Determine whether a program unit corresponding to the target unit test case has a defect based on the test data and the crash data.

[0060] In this embodiment, the specific implementation method of step 303 is the same as Figure 2 The process is similar to step 203 in the embodiment and will not be described again here.

[0061] Step 304: perform offline analysis on the test data to determine whether the program unit corresponding to the target unit test case has defects.

[0062] In this embodiment, one or more offline analysis strategies can be pre-set, and the offline analysis strategy can be used to determine whether the program unit corresponding to the unit test case has defects. In this way, it can be determined whether the program unit corresponding to the target unit test case has defects based on the offline analysis strategy and the test data of the target unit test case obtained in step 302.

[0063] In some implementations, the test data of the target unit test case may also include a stack trace. And the above step 304 may include:

[0064] a) Matching the stack trace of the target unit test case with a first rule in a preset first rule set.

[0065] In this implementation, the first rule in the first rule set may be determined based on analyzing stack traces of multiple program units that have abnormalities. For example, the first rule may be determined by manual analysis. The first rule may specify which keywords are included in the stack trace to determine that the program unit corresponding to the stack trace has a defect.

[0066] b) In response to determining that the stack trace of the target unit test case matches a first rule in the first rule set, determining that a program unit corresponding to the target unit test case has a defect.

[0067] In this implementation, if the stack trace of the target unit test case matches a first rule in the first rule set, it can be determined that the program unit corresponding to the target unit test case has a defect. In this implementation, through the pre-created first rule set, it can be quickly determined that the program unit corresponding to the target unit test case whose stack trace matches the first rule has a defect.

[0068] In some other implementations, the above step 304 may also include:

[0069] 1) Matching the stack trace of the target unit test case with a second rule in a preset second rule set.

[0070] In this implementation, the second rule in the second rule set may be determined based on analyzing stack traces of multiple program units that do not have exceptions. For example, the second rule may be determined by manual analysis. The second rule may specify which keywords are included in the stack trace to determine that the corresponding unit of the stack trace has a defect.

[0071] 2) In response to determining that the stack trace of the target unit test case matches a second rule in the second rule set, determining that a program unit corresponding to the target unit test case does not have a defect.

[0072] In this implementation, if the stack trace of the target unit test case matches a second rule in the second rule set, it can be determined that the program unit corresponding to the target unit test case does not have a defect. In this implementation, through the pre-created second rule set, it can be quickly determined that the program unit corresponding to the target unit test case whose stack trace matches the second rule does not have a defect.

[0073] In other implementations, the program under test may be developed in an iterative development manner, and a new version may be generated in each iteration. Thus, multiple versions of the program under test may be obtained. As an example, a unit test case may be automatically generated by a test tool based on a certain version of the code. In other words, the target unit test case may be automatically generated by the test tool based on the code of the test case generation version.

[0074] In practice, when using iterative development to develop programs, a version control system can be used for version management. Generally, various version control systems can support branches in some form, and a branch represents an independent development line. Using branches can separate from the main development line and develop branches without affecting the operation of the main line. Figure 4 As shown, Figure 4 A schematic diagram showing an example of a branch graph of a version control system. Figure 4 The branch diagram shown includes multiple nodes, each of which can represent a code commit, and each code commit can represent the generation of a new version of the code. The line connecting the nodes can represent a branch. During the program development process, there will be a main branch. When different developers develop new requirements, they can pull out a requirement branch based on the main branch and develop functions in the requirement branch. After the development is completed, the requirement branch can be merged back to the main branch. Figure 4 In the example, the branch formed by the nodes in the dotted box 401 can represent the trunk branch. In this example, multiple unit test cases can be generated based on the version of the code corresponding to node 4011 on the trunk branch. Afterwards, the generated unit test cases can also be executed based on the version of the code corresponding to node 4011. Here, the version corresponding to node 4011 can be called the test case generation version. In this example, the requirement branch can be pulled out through node 4012 on the trunk branch, and the version corresponding to node 4012 can be called the first version. For example, the branch formed by the nodes in the dotted box 402 can represent the requirement branch. In this example, the version corresponding to node 4021 on the requirement branch can be called the current version. Multiple unit test cases can be generated based on the version of the code corresponding to node 4011, or the program can be executed based on the version of node 4021, and the unit test cases that failed to execute can be used as target unit test cases. It can be understood that Figure 4The number of nodes, the number of branches, the number of nodes contained in the branches, etc. shown in the figure are only for illustration, and are not intended to limit the number of nodes, the number of branches, the number of nodes contained in the branches, etc. In practice, any number of nodes, branches, etc. may be set according to actual development needs, and each branch may also include nodes of the number of tasks.

[0075] Based on this, the above step 304 may include:

[0076] i) Determine whether the target unit test case fails to execute in the code of the test case generation version.

[0077] In this implementation, after the unit test cases are automatically generated based on the code of the test case generation version, each unit test case can also be executed based on the code of the test case generation version, and the execution results can include execution success and execution failure.

[0078] ii) In response to determining that the target unit test case fails to execute in the code of the test case generation version, determine to filter the target unit test case and no longer determine whether the program unit corresponding to the target unit test case has a defect.

[0079] In this implementation, if the target unit test case fails to execute in the code of the test case generation version, it means that there may be a problem when the target unit test case is generated and it cannot be executed normally. Therefore, the failure to execute in the code of the current version of the program being tested may not be due to a problem with the current version of the code. Therefore, in this case, the target unit test case can be filtered, and no further judgment is made as to whether the program unit corresponding to the target unit test case has defects, thereby reducing unnecessary waste of computing resources. Through this implementation, the target unit test cases that fail to execute in the code of the test case generation version can be filtered out, thereby reducing unnecessary waste of computing resources.

[0080] In some implementations, the above step 304 may also include:

[0081] ① In response to determining that the target unit test case is successfully executed in the code of the test case generation version, determine the file changes between the test case generation version and the current version, and the first call chain of the first changed file.

[0082] In this implementation, if the target unit test case is successfully executed in the code of the test case generation version, it is necessary to further determine the file changes between the test case generation version and the current version to determine the first changed file that has changed between the two. In practice, the difference between the two versions can be obtained through the version control system. Afterwards, the first call chain of the first change file can also be determined. Here, the first call chain can represent the calling relationship of the first change file, including which files are called and which files are called. For example, suppose class People calls classBook.

[0083] ② In response to determining that the code file corresponding to the target unit test case is related to the first change file, or is on the first call chain, it is determined that the program unit corresponding to the target unit test case has a defect.

[0084] In this implementation, if it is determined that the code file corresponding to the target unit test case is related to the first change file, for example, the same, or partially the same, it can be determined that the program unit corresponding to the target unit test case has a defect. For example, assuming that the code file corresponding to the target unit test case is class AA, and the first change file is also classAA, it can be indicated that the direct modification of the code file class AA causes the execution failure of the test case of class AA. Therefore, it can be inferred that the code unit corresponding to the target unit test case has a defect. In addition, if it is determined that the code file corresponding to the target unit test case is on the first call chain corresponding to the first change file, it is determined that the program unit corresponding to the target unit test case has a defect. For example, assuming that the code file corresponding to the target unit test case is class 10, the first change file is class 20, and class 10 calls class20, it can be indicated that the code file corresponding to the target unit test case is on the call chain of the first change file, which can indicate that the execution failure of class 10 may be caused by the modification of class 20, so it can be inferred that the code unit corresponding to the target unit test case has a defect. Through this implementation, defective program units can be screened out based on the changes between the test case generation version and the current version.

[0085] Step 305 , in response to determining that a program unit corresponding to the target unit test case has a defect, the defect is graded.

[0086] In this embodiment, when it is determined that a program unit corresponding to a target unit test case has a defect, the defect may be graded to determine the priority of the defect.

[0087] In some implementations, the above step 305 may include:

[0088] (1) In response to determining that a program unit corresponding to a target unit test case is defective based on the test data and the crash data, the defect is determined as a first priority defect.

[0089] In this implementation, if a defect is determined based on both online crash data and offline test data, it can mean that the defect already exists in the online operation, so the defect can be rated as a first priority defect. Here, the first priority defect is a high priority defect.

[0090] (2) In response to determining that a code block corresponding to a code line reporting an error in the source code of a program unit corresponding to the defect has been modified in the current version, the defect is determined to be a first priority defect.

[0091] In this implementation, if the code block corresponding to the code line reporting an error in the source code of the program unit corresponding to a defect is modified in the current version, it can be inferred that the defect is caused by the direct source code being modified. Therefore, the defect can be classified as a first priority defect.

[0092] (3) In response to determining that a code block corresponding to a code line reporting an error in the source code of a program unit corresponding to the defect has not been modified in the current version, determining a file change between the first version and the current version, and a second call chain of a second change file.

[0093] In this implementation, if the code block corresponding to the code line where the source code of the program unit corresponding to a defect reports an error has not been modified in the current version, it is possible to further determine the file changes between the first version and the current version to determine the second change file that has changed between the two, and the second call chain of the second change file. Among them, the second call chain can represent the call relationship of the second change file. Here, the node corresponding to the first version can be the node of the requirement branch where the current version is pulled out, and the current version can be a version iterated based on the first version.

[0094] (4) In response to determining that the code file corresponding to the target unit test case is related to the second change file, or is in the second call chain, determining that the defect is a second priority defect.

[0095] In this implementation, if a defect is not caused by the modification of the source code, the change of the code file can be further determined to determine the defect level. Here, the priority of the first priority defect is higher than the second priority defect.

[0096] In addition, defects that do not meet the first priority defect and the second priority defect can be determined as third priority defects. The priority of the third priority defect is lower than that of the second priority defect. This implementation method can implement defect grading, thereby classifying defects into different priorities.

[0097] Figure 3 In the illustrated embodiment, in addition to comprehensively analyzing and screening defects based on online crash data and offline test data, offline analysis and screening of defects can also be performed on the test data, and the screened defects can be prioritized.

[0098] According to another embodiment, a device for finding defects in program testing is provided. The device for finding defects in program testing can be deployed on any device, equipment, platform, equipment cluster, etc. with computing and processing capabilities.

[0099] Figure 5 A schematic block diagram of an apparatus for defect discovery in program testing according to an embodiment is shown. Figure 5 The device shown is used to perform Figure 2 As shown in the method. Figure 5 As shown, the device 500 for defect discovery in program testing includes: a first acquisition unit 501, configured to obtain crash data of an operation crash generated when the online version of the tested program is running online; a second acquisition unit 502, configured to obtain test data of a target unit test case that failed to execute in an offline test of the current version of the tested program; and a determination unit 503, configured to determine whether a program unit corresponding to the target unit test case has a defect based on the test data and the crash data.

[0100] In some optional implementations of this embodiment, the device 500 also includes: an analysis unit (not shown in the figure), configured to perform offline analysis on the above test data to determine whether there is a defect in the program unit corresponding to the above target unit test case.

[0101] In some optional implementations of the present embodiment, the test data includes a stack trace; and the analysis unit is further configured to match the stack trace of the target unit test case with a first rule in a preset first rule set, wherein the first rule in the first rule set is determined based on analyzing the stack traces of multiple program units in which exceptions occur; in response to determining that the stack trace of the target unit test case matches the first rule in the first rule set, it is determined that a program unit corresponding to the target unit test case has a defect.

[0102] In some optional implementations of the present embodiment, the test data includes a stack trace; and the analysis unit is further configured to match the stack trace of the target unit test case with a second rule in a preset second rule set, wherein the second rule in the second rule set is determined based on analyzing the stack traces of a plurality of program units in which no exceptions occur; in response to determining that the stack trace of the target unit test case matches the second rule in the second rule set, it is determined that there is no defect in the program unit corresponding to the target unit test case.

[0103] In some optional implementations of the present embodiment, the above-mentioned tested program is developed iteratively, and a new version is generated in each iteration; the above-mentioned target unit test case is automatically generated by the testing tool based on the code of the test case generation version; and the analysis unit is further configured to determine whether the above-mentioned target unit test case fails to execute in the code of the above-mentioned test case generation version; in response to determining that the above-mentioned target unit test case fails to execute in the code of the above-mentioned test case generation version, determine to filter the above-mentioned target unit test case, and no longer judge whether the program unit corresponding to the above-mentioned target unit test case has defects.

[0104] In some optional implementations of this embodiment, the analysis unit is further configured to, in response to determining that the target unit test case is successfully executed in the code of the test case generation version, determine the file changes between the test case generation version and the current version, and the first call chain of the first change file; in response to determining that the code file corresponding to the target unit test case is related to the first change file, or is on the first call chain, determine that the program unit corresponding to the target unit test case has a defect.

[0105] In some optional implementations of this embodiment, the apparatus 500 further includes: a grading unit configured to, in response to determining that a program unit corresponding to the target unit test case has a defect, grade the defect.

[0106] In some optional implementations of the present embodiment, the grading unit (not shown in the figure) is further configured to, in response to determining that a program unit corresponding to the target unit test case has a defect based on the test data and the crash data, determine the defect as a first priority defect; in response to determining that a code block corresponding to a code line reporting an error in the source code of the program unit corresponding to the defect has been modified in the current version, determine the defect as a first priority defect.

[0107] In some optional implementations of the present embodiment, the grading unit is further configured to, in response to determining that the code block corresponding to the code line reporting an error in the source code of the program unit corresponding to the defect has not been modified in the above-mentioned current version, determine the file changes between the first version and the above-mentioned current version, and a second call chain of the second change file, wherein the above-mentioned current version is a version iterated based on the above-mentioned first version; in response to determining that the code file corresponding to the above-mentioned target unit test case is related to the above-mentioned second change file, or is on the above-mentioned second call chain, determine that the defect is a second priority defect, wherein the priority of the first priority defect is higher than that of the second priority defect.

[0108] In some optional implementations of the present embodiment, the test data includes the tested class, the tested exception type, and the tested method; the crash data includes the crashed class, the crashed exception type, the crashed method, and the crash identifier; and the determination unit 503 is further configured to, in response to determining that the crash data matches the test data, determine that the crash corresponding to the matching crash data is a target crash; based on the crash identifier of the target crash, obtain the stack trace corresponding to the target crash; match the stack trace corresponding to the target crash with the stack trace corresponding to the target unit test case; and in response to determining that the match is found, determine that a program unit corresponding to the target unit test case has a defect.

[0109] In some optional implementations of the present embodiment, the above-mentioned determination that the above-mentioned crash data matches the above-mentioned test data includes: in response to determining that the class of the above-mentioned test and the exception type of the above-mentioned test are respectively the same as the class of the above-mentioned crash and the exception type of the above-mentioned crash, further judging whether the method of the above-mentioned test is the same as the method of the above-mentioned crash; if the method of the above-mentioned test is the same as the method of the above-mentioned crash, determining that the above-mentioned crash data matches the above-mentioned test data.

[0110] The above device embodiments correspond to the method embodiments. For specific descriptions, please refer to the description of the method embodiments, which will not be repeated here. The device embodiments are obtained based on the corresponding method embodiments and have the same technical effects as the corresponding method embodiments. For specific descriptions, please refer to the corresponding method embodiments.

[0111] According to another embodiment, there is also provided a computer-readable storage medium having a computer program stored thereon, which, when executed in a computer, causes the computer to execute Figure 2 The method described.

[0112] According to another embodiment, there is also provided an electronic device, comprising a memory and a processor, wherein the memory stores an executable code, and when the processor executes the executable code, Figure 2 The method described.

[0113] The foregoing describes specific embodiments of the present disclosure, and other embodiments are within the scope of the appended claims. In some cases, the actions or steps recorded in the claims may be performed in an order different from that in the embodiments, and the desired results may still be achieved. In addition, the processes depicted in the accompanying drawings do not necessarily have to be performed in the specific order or sequential order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible, or may be advantageous.

[0114] Reference below Figure 6 , which shows a schematic structural diagram of an electronic device 600 suitable for implementing an embodiment of the present application. Figure 6 The electronic device shown is merely an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.

[0115] like Figure 6 As shown, the electronic device 600 may include a processing device (e.g., a central processing unit, a graphics processing unit, etc.) 601, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 602 or a program loaded from a storage device 608 into a random access memory (RAM) 603. In the RAM 603, various programs and data required for the operation of the electronic device 600 are also stored. The processing device 601, the ROM 602, and the RAM 603 are connected to each other via a bus 604. An input / output (I / O) interface 605 is also connected to the bus 604.

[0116] Typically, the following devices may be connected to the I / O interface 605: an input device 606 including, for example, a touch screen, a touch pad, a keyboard, a mouse, etc.; an output device 607 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; a storage device 608 including, for example, a magnetic tape, a hard disk, etc.; and a communication device 609. The communication device 609 may allow the electronic device 600 to communicate with other devices wirelessly or by wire to exchange data. Figure 6 The electronic device 600 is shown with various devices, but it should be understood that it is not required to implement or possess all the devices shown. More or fewer devices may be implemented or possessed instead. Figure 6 Each block shown in the figure may represent one device, or may represent multiple devices as required.

[0117] In particular, according to an embodiment of the present application, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present application includes a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from the network through the communication device 609, or installed from the storage device 608, or installed from the ROM 602. When the computer program is executed by the processing device 601, the above-mentioned functions defined in the method of the embodiment of the present application are executed.

[0118] The embodiment of the present disclosure further provides a computer-readable storage medium on which a computer program is stored. When the computer program is executed in a computer, the computer is caused to execute the method provided by the present disclosure.

[0119] It should be noted that the computer-readable medium described in the embodiments of the present disclosure may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the embodiments of the present disclosure, the computer-readable storage medium may be any tangible medium containing or storing a program, which may be used by or in combination with an instruction execution system, device or device. In the embodiments of the present disclosure, the computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, in which a computer-readable program code is carried. This propagated data signal may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. The computer readable signal medium may also be any computer readable medium other than a computer readable storage medium, which may send, propagate or transmit a program for use by or in conjunction with an instruction execution system, apparatus or device. The program code contained on the computer readable medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, RF (Radio Frequency), etc., or any suitable combination of the above.

[0120] The computer-readable medium may be included in the electronic device; or it may exist independently without being installed in the electronic device. The computer-readable medium carries one or more programs. When the one or more programs are executed by the electronic device, the electronic device: obtains crash data of the online version of the tested program generated when the online version is running online; obtains test data of the target unit test case that failed to execute in the offline test of the current version of the tested program; and determines whether the program unit corresponding to the target unit test case has defects based on the test data and the crash data.

[0121] Computer program code for performing the operations of the disclosed embodiments may be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages, such as Java, Smalltalk, C++, and conventional procedural programming languages, such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as an independent software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or electronic device. In the case of a remote computer, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0122] Each embodiment in the present disclosure is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the storage medium and computing device embodiments, since they are basically similar to the method embodiments, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiments.

[0123] Those skilled in the art should be aware that in one or more of the above examples, the functions described in the embodiments of the present invention may be implemented using hardware, software, firmware, or any combination thereof. When implemented using software, these functions may be stored in a computer-readable medium or transmitted as one or more instructions or codes on a computer-readable medium.

[0124] The specific implementation methods described above further describe the purpose, technical solutions and beneficial effects of the embodiments of the present invention in detail. It should be understood that the above description is only a specific implementation method of the embodiments of the present invention and is not intended to limit the scope of protection of the present invention. Any modification, equivalent replacement, improvement, etc. made on the basis of the technical solution of the present invention shall be included in the scope of protection of the present invention.

Claims

1. A method for discovering defects in program testing, comprising: Obtain crash data of operation crashes generated when the online version of the tested program is running online; Obtaining test data of target unit test cases that failed to be executed in offline testing of the current version of the tested program; It is determined whether a program unit corresponding to the target unit test case has a defect according to the test data and the crash data.

2. The method according to claim 1, wherein: The method further comprises: The test data is analyzed offline to determine whether a program unit corresponding to the target unit test case has a defect.

3. The method according to claim 2, wherein: The test data includes a stack trace; and the offline analysis of the test data to determine whether a program unit corresponding to the target unit test case has a defect includes: Matching the stack trace of the target unit test case with a first rule in a preset first rule set, wherein the first rule in the first rule set is determined based on analyzing stack traces of a plurality of program units in which exceptions occur; In response to determining that the stack trace of the target unit test case matches a first rule in the first rule set, it is determined that a program unit corresponding to the target unit test case has a defect.

4. The method according to claim 2, wherein: The test data includes a stack trace; and the offline analysis of the test data to determine whether a program unit corresponding to the target unit test case has a defect includes: Matching the stack trace of the target unit test case with a second rule in a preset second rule set, wherein the second rule in the second rule set is determined based on analyzing the stack traces of a plurality of program units that do not have exceptions; In response to determining that the stack trace of the target unit test case matches a second rule in the second rule set, it is determined that a program unit corresponding to the target unit test case does not have a defect.

5. The method according to claim 2, wherein: The tested program is developed iteratively, and a new version is generated in each iteration. The target unit test case is automatically generated by the test tool based on the code of the test case generation version; as well as The offline analysis of the test data to determine whether the program unit corresponding to the target unit test case has defects includes: Determine whether the target unit test case fails to execute in the code of the test case generation version; In response to determining that the target unit test case fails to execute in the code of the test case generation version, it is determined to filter the target unit test case and no longer determine whether a program unit corresponding to the target unit test case has a defect.

6. The method according to claim 5, wherein: The offline analysis of the test data to determine whether the program unit corresponding to the target unit test case has defects includes: In response to determining that the target unit test case is successfully executed in the code of the test case generation version, determining file changes between the test case generation version and the current version, and a first call chain of a first changed file; In response to determining that the code file corresponding to the target unit test case is related to the first change file, or is on the first call chain, it is determined that a program unit corresponding to the target unit test case has a defect.

7. The method according to any one of claims 1 to 6, wherein: The method further comprises: In response to determining that a program unit corresponding to the target unit test case has a defect, the defect is graded.

8. The method according to claim 7, wherein: The defects are classified, including: In response to determining that a program unit corresponding to the target unit test case has a defect according to the test data and the crash data, determining the defect as a first priority defect; In response to determining that a code block corresponding to a code line reporting an error in the source code of a program unit corresponding to the defect is modified in the current version, the defect is determined to be a first priority defect.

9. The method according to claim 8, wherein: The defect classification further includes: In response to determining that a code block corresponding to a code line reporting an error in the source code of a program unit corresponding to the defect has not been modified in the current version, determining a file change between a first version and the current version, and a second call chain of a second change file, wherein the current version is a version iterated based on the first version; In response to determining that the code file corresponding to the target unit test case is related to the second change file, or is on the second call chain, determining that the defect is a second priority defect, wherein the priority of the first priority defect is higher than that of the second priority defect.

10. The method according to claim 1, wherein: The test data includes the tested class, the tested exception type, and the tested method; the crash data includes the crashed class, the crash exception type, the crashed method, and the crash identifier; as well as The determining, according to the test data and the crash data, whether a program unit corresponding to the target unit test case has a defect comprises: In response to determining that the crash data matches the test data, determining a crash corresponding to the matched crash data as a target crash; Obtaining a stack trace corresponding to the target crash according to the crash identifier of the target crash; Matching the stack trace corresponding to the target crash with the stack trace corresponding to the target unit test case; In response to determining that there is a match, it is determined that a program unit corresponding to the target unit test case has a defect.

11. The method according to claim 10, wherein: The determining that the crash data matches the test data includes: In response to determining that the tested class and the tested exception type are respectively the same as the crashed class and the crashed exception type, further determining whether the tested method is the same as the crashed method; If the method of the test is the same as the method of the crash, it is determined that the crash data matches the test data.

12. A device for finding defects in program testing, comprising: A first acquisition unit is configured to acquire crash data of a running crash generated when the online version of the tested program is running online; A second acquisition unit is configured to acquire test data of target unit test cases that failed to be executed in offline testing of the current version of the tested program; The determination unit is configured to determine whether a program unit corresponding to the target unit test case has a defect based on the test data and the crash data.

13. A computer-readable storage medium having a computer program stored thereon, which, when executed in a computer, causes the computer to execute the method according to any one of claims 1 to 11.

14. An electronic device comprising a memory and a processor, wherein the memory stores executable code, and when the processor executes the executable code, the method according to any one of claims 1 to 11 is implemented.

Images