Efficiency evaluation method and device for test case, equipment and storage medium

Abstract

The invention discloses an effectiveness evaluation method and device for a test case, equipment and a storage medium. The method comprises the following steps: acquiring relevant parameters of a test case during execution, wherein the relevant parameters comprise defect relevant parameters, test cost relevant parameters and an environment scene; analyzing the defect related parameters to obtain a defect capture rate of the test case during execution; analyzing the related parameters of the test cost to obtain the test cost of the test case during execution; and determining a test efficiency value of the test case based on the defect capture rate, the test cost and the environment adaptation coefficient corresponding to the environment scene. According to the method, the related parameters of the test case during execution are obtained and analyzed, so that the test efficiency value of the test case can be obtained, and the efficiency of the test case is determined.

Description

Technical Field

[0001] The embodiments of the present invention relate to the field of test case technology, and in particular to a method, apparatus, device and storage medium for evaluating the performance of test cases. Background Technology

[0002] In software testing, performance evaluation primarily targets test cases, test processes, test tools, or test teams. Its core objective is to assess their ability to discover defects and ensure software quality within limited resources (time, manpower, and cost). Current performance evaluation technologies are generally based on static evaluation. Static performance evaluation is an assessment of the potential performance (such as effectiveness, efficiency, and quality) of an object by analyzing its inherent attributes, design documents, structural features, and other static information, without relying on the actual execution process. It does not involve dynamic operational data but rather judges whether the object has the potential to achieve the expected goals based on "design-level" information.

[0003] However, this method of performance evaluation only looks at individual indicators such as the number of defects, and cannot comprehensively and quantitatively reflect the test performance. Summary of the Invention

[0004] This invention provides a method, apparatus, device, and storage medium for evaluating the performance of test cases, in order to solve the problem that existing performance evaluation methods cannot quantitatively reflect test performance from multiple perspectives.

[0005] According to one aspect of the present invention, a method for evaluating the performance of test cases is provided, the method comprising:

[0006] Obtain relevant parameters when the test cases are executed, including defect-related parameters, test cost-related parameters, and environmental scenarios;

[0007] The defect-related parameters are analyzed to obtain the defect capture rate of the test cases during execution.

[0008] The test cost-related parameters are analyzed to obtain the test cost of the test case during execution.

[0009] The test performance value of the test case is determined based on the defect capture rate, the test cost, and the environment adaptation coefficient corresponding to the environment scenario.

[0010] According to another aspect of the present invention, a device for evaluating the performance of test cases is provided, the device comprising:

[0011] The acquisition module is used to acquire relevant parameters of the test cases during execution, including defect-related parameters, test cost-related parameters, and environmental scenarios.

[0012] The first analysis module is used to analyze the defect-related parameters to obtain the defect capture rate of the test cases during execution.

[0013] The second analysis module is used to analyze the test cost-related parameters to obtain the test cost when the test case is executed;

[0014] The determination module is used to determine the test performance value of the test case based on the defect capture rate, the test cost, and the environment adaptation coefficient corresponding to the environment scenario.

[0015] According to another aspect of the present invention, an electronic device is provided, the electronic device comprising: at least one processor; and

[0016] A memory communicatively connected to the at least one processor; wherein,

[0017] The memory stores a computer program that can be executed by the at least one processor, the computer program being executed by the at least one processor to enable the at least one processor to perform the performance evaluation method for test cases according to any embodiment of the present invention.

[0018] According to another aspect of the present invention, a computer-readable storage medium is provided, the computer-readable storage medium storing computer instructions for causing a processor to execute and implement the performance evaluation method for test cases according to any embodiment of the present invention.

[0019] This invention discloses a method, apparatus, device, and storage medium for evaluating the performance of test cases. The method includes: acquiring relevant parameters of a test case during execution, including defect-related parameters, test cost-related parameters, and an environmental scenario; analyzing the defect-related parameters to obtain the defect capture rate of the test case during execution; analyzing the test cost-related parameters to obtain the test cost of the test case during execution; and determining the test performance value of the test case based on the defect capture rate, the test cost, and the environmental adaptation coefficient corresponding to the environmental scenario. This method, by acquiring and analyzing relevant parameters of a test case during execution, can obtain the test performance value of the test case, thereby determining the performance of the test case and solving the problem that existing performance evaluation methods cannot comprehensively quantify and reflect test performance.

[0020] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of the present invention, nor is it intended to limit the scope of the invention. Other features of the invention will become readily apparent from the following description. Attached Figure Description

[0021] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0022] Figure 1 This is a flowchart illustrating a method for evaluating the effectiveness of test cases according to Embodiment 1 of the present invention.

[0023] Figure 2 This is a schematic diagram of the structure of a test case performance evaluation device provided in Embodiment 2 of the present invention;

[0024] Figure 3 This is a schematic diagram of the structure of an electronic device according to an embodiment of the present invention. Detailed Implementation

[0025] To enable those skilled in the art to better understand the present invention, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are merely some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention. It should be understood that the various steps described in the method embodiments of the present invention can be performed in different orders and / or in parallel. Furthermore, the method embodiments may include additional steps and / or omit the steps shown. The scope of the present invention is not limited in this respect.

[0026] The term "comprising" and its variations as used herein are open-ended inclusions, meaning "including but not limited to". The term "based on" means "at least partially based on". The term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one additional embodiment"; the term "some embodiments" means "at least some embodiments". Definitions of other terms will be given in the description below.

[0027] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. Furthermore, any variations of the terms "comprising" and "having," etc., are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0028] It should be noted that the terms "a" and "a plurality of" used in this invention are illustrative rather than restrictive. Those skilled in the art should understand that, unless otherwise expressly indicated in the context, they should be understood as "one or more".

[0029] The names of the messages or information exchanged between the multiple devices in the embodiments of the present invention are for illustrative purposes only and are not intended to limit the scope of these messages or information.

[0030] Example 1

[0031] Figure 1 This is a flowchart illustrating a test case performance evaluation method provided in Embodiment 1 of the present invention. This method is applicable to situations where test cases are evaluated for performance during or after execution. The method can be executed by a test case performance evaluation device, which can be implemented by software and / or hardware and is generally integrated into an electronic device. In this embodiment, the electronic device includes, but is not limited to, devices such as computers.

[0032] like Figure 1 As shown in Embodiment 1 of the present invention, a method for evaluating the performance of test cases includes the following steps:

[0033] S110. Obtain relevant parameters when the test case is executed, including defect-related parameters, test cost-related parameters, and environmental scenarios.

[0034] In this context, a test case can be a standardized set of execution steps, input data, expected results, and execution environment designed during software testing to verify whether a software system meets specific requirements. The test cases in this embodiment can be software-specific or hardware-specific; this embodiment does not limit the scope. Relevant parameters can be data generated when the test cases are executed, and may include defect-related parameters, test cost-related parameters, and environmental scenarios. Defect-related parameters can refer to parameters related to defects discovered by the test cases, test cost-related parameters can be parameters related to the cost of executing the test cases, and environmental scenarios can refer to the execution environment of the test cases.

[0035] In this embodiment, relevant parameters of the test case during execution can be obtained.

[0036] S120. Analyze the defect-related parameters to obtain the defect capture rate of the test case during execution.

[0037] Among them, the defect capture rate can refer to the percentage of defects actually found by a test case in a single test run relative to its historical average defect detection capability, and can be used to measure the rate at which test cases find defects.

[0038] In this embodiment, defect-related parameters can be analyzed to obtain the defect capture rate during test case execution. For example, the ratio of the number of defects discovered this time to the number of defects discovered in the past can be used as the defect capture rate, or the defect capture rate can be determined by combining the number of defects and the importance of each defect.

[0039] In one embodiment, the defect-related parameters include the defects discovered by the test case and the defect level corresponding to the defects. Accordingly, the step of analyzing the defect-related parameters to obtain the defect capture rate of the test case during execution includes: determining the number of valid defects discovered this time based on the defects and the defect level corresponding to the defects; and determining the defect capture rate of the test case this time based on the number of valid defects and the historical average number of defects discovered by the test case.

[0040] Among these, the number of valid defects refers to the number of defects that, after confirmation, are indeed present in the software product and require fixing. Defect level can be a standard for prioritizing discovered defects in software testing, used to clarify the degree of impact of defects on software functionality, user experience, system stability, etc. The historical average number of defects found refers to the average number of defects found per unit of time (e.g., daily, weekly), per unit of test case, or per unit of functional module in similar projects, at the same stage, or with the same testing type in the past.

[0041] In this embodiment, the defect capture rate of a test case can be determined based on the number of valid defects, defect level, and the historical average number of defects found in the test case. By considering the level of defects of different levels, the capture capability can be avoided by simply measuring the quantity, which is more in line with the actual test value.

[0042] In one embodiment, determining the number of valid defects discovered this time based on the defect and the defect level corresponding to the defect includes: calculating the number of valid defects discovered this time based on the defect level and quantity corresponding to each discovered defect and the total number of defects; the defect level includes level zero, level one, level two, level three and level four.

[0043] The defect levels can be divided into Level 0 (P0), Level 1 (P1), Level 2 (P2), Level 3 (P0), and Level 4 (P5).

[0044] In this embodiment, the number of valid defects discovered can be calculated based on the defect level and quantity corresponding to each discovered defect, as well as the total number of defects. For example, Table 1 below is a defect level classification table, defining the severity and coefficient of different defect levels. The number of valid defects discovered is calculated as follows: Valid Defect 1 * Defect Coefficient + Defect 2 * Defect Coefficient + ... + Defect N * Defect Coefficient) / N. For instance, if two defects are discovered with severity levels P1 and P2 respectively, the number of valid defects discovered is (3 + 1) / 2 = 2.

[0045] Table 1 Defect Level Classification Table

[0046]

[0047]

[0048] S130. Analyze the test cost-related parameters to obtain the test cost of the test case during execution.

[0049] In this embodiment, test cost-related parameters can be analyzed to obtain the test cost when the test cases are executed. For example, the various costs of time, manpower, and resources required in the process of executing the test cases can be determined to determine the test cost.

[0050] In one embodiment, the test cost-related parameters include time cost indicators, resource cost indicators, and maintenance cost indicators. Correspondingly, analyzing these parameters to obtain the test cost of the test case during execution includes: determining the time cost required for the test case to execute based on the time cost indicator; determining the resource cost required for the test case to execute based on the resource cost indicator; determining the maintenance cost required for the test case to execute based on the maintenance cost indicator; and inputting the time cost, resource cost, and maintenance cost into a multidimensional cost model to obtain the test cost of the test case during execution.

[0051] Time cost can refer to the time spent executing test cases (such as manual operation time and automated script execution time), while resource cost can refer to the consumption of resources such as hardware (servers, equipment), software (tools), and manpower (test personnel's working hours) during execution. Maintenance cost can refer to the cost of modifying and updating test cases due to changes in requirements or version iterations (such as the time invested in redesigning steps and adjusting expected results).

[0052] In this embodiment, test cost-related parameters are divided into time cost indicators, resource cost indicators, and maintenance cost indicators. These indicators can be analyzed separately to obtain the time cost, resource cost, and maintenance cost required for the execution of test cases. The time cost, resource cost, and maintenance cost are then input into a multidimensional cost model to obtain the test cost of the test cases during execution. The multidimensional cost model in this embodiment can be preset. For example, the multidimensional cost model can be set as: log(1 + time cost + resource cost) + maintenance cost weight factor × maintenance cost, where the maintenance cost weight factor can be set based on actual conditions, for example, it can be set to 0.5. This model uses a logarithmic function to suppress the influence of extreme values.

[0053] In one embodiment, the time cost metric includes the execution time of the test case and the cost of the machine required to execute the test case; the resource cost metric includes the memory occupied by the test case, the test duration, and the cost of the cloud server; and the maintenance cost includes the number of lines of code for the test case and the maintenance complexity factor.

[0054] In this embodiment, the time cost metric may include the execution time of the test case and the cost of the machine required to execute the test case; the resource cost metric may include the memory occupied by the test case, the test duration, and the cost of the cloud server; and the maintenance cost may include the number of lines of code in the test case and the maintenance complexity factor. The maintenance complexity factor can be set according to the difficulty of maintenance.

[0055] S140. Based on the defect capture rate, the test cost, and the environment adaptation coefficient corresponding to the environment scenario, determine the test performance value of the test case.

[0056] Among them, the environment adaptation coefficient can refer to the weight of the environment scenario. The test performance value can refer to...

[0057] In this embodiment, the test performance value of test cases can be calculated based on the defect capture rate, test cost, and the environment adaptation coefficient corresponding to the environment scenario.

[0058] This invention provides a method for evaluating the performance of test cases, comprising: acquiring relevant parameters of the test cases during execution, including defect-related parameters, test cost-related parameters, and environmental scenarios; analyzing the defect-related parameters to obtain the defect capture rate of the test cases during execution; analyzing the test cost-related parameters to obtain the test cost of the test cases during execution; and determining the test performance value of the test cases based on the defect capture rate, the test cost, and the environmental adaptation coefficient corresponding to the environmental scenarios. This method, by acquiring and analyzing relevant parameters of the test cases during execution, can obtain the test performance value of the test cases, thereby determining the performance of the test cases and solving the problem that existing performance evaluation methods cannot comprehensively quantify and reflect test performance.

[0059] Based on the above embodiments, modified embodiments of the above embodiments are proposed. It should be noted that, in order to keep the description brief, only the differences from the above embodiments are described in the modified embodiments.

[0060] In one embodiment, the environment scenarios include a local development environment, a testing environment, a pre-release environment, and a production environment.

[0061] In this context, the local development environment can refer to the environment for developing test cases. The testing environment can be the environment for testing test cases. For example, a testing environment could be a Continuous Integration (CI) / Continuous Delivery (CD) pipeline. CI refers to the development team frequently merging code into a shared repository, automatically executing build and testing processes after each merge to quickly identify and resolve code integration issues, ensuring code quality. CD, based on continuous integration, refers to automatically deploying code to a test or pre-production environment, ensuring the software is ready for release at any time, although final deployment to the production environment may still require manual intervention. The pre-release environment refers to the final verification environment before the software is officially released to the production environment. Its configuration, data, and operating conditions are as consistent as possible with the production environment, used to simulate final testing under production scenarios to ensure the software runs correctly in a real environment. The production environment refers to the final, user-facing operating environment of the software, containing real user data, business traffic, and formal server configurations. It is the carrier on which the software provides actual services, directly impacting user experience and business continuity.

[0062] In this embodiment, the environment scenarios may include local development environment, testing environment, pre-release environment and production environment. Different environment scenarios can be set with different environment adaptation coefficients, so as to better evaluate the performance of test cases in different environments.

[0063] In one embodiment, determining the test performance value of the test case based on the defect capture rate, the test cost, and the environment adaptation coefficient corresponding to the environment scenario includes: taking the product of the defect capture rate, the reciprocal of the test cost, and the environment adaptation coefficient corresponding to the environment scenario as the test performance value of the test case.

[0064] In this embodiment, the product of the defect capture rate, the reciprocal of the test cost, and the environment adaptation coefficient corresponding to the environmental scenario can be used as the test performance value of the test case. For example, the test performance value = (defect capture rate / test cost) × environment adaptation coefficient.

[0065] Based on the technical solutions of the above embodiments, this invention provides several specific implementation methods.

[0066] As a specific implementation method of this embodiment, a quantitative basis for dynamic priority adjustment can be generated by integrating defect capture efficiency, multi-dimensional testing cost, and environmental weight factors. The key feature is the use of a logarithmic cost compression function and a rolling time window mechanism. The formula for calculating the defect capture rate is as follows:

[0067] Defect capture rate = Number of valid defects found this time / (Average number of defects found in the last 10 times + 1) × 100%;

[0068] The formula uses the historical average number of defects found over the last 10 runs as an example, where +1 is added to prevent division by zero errors. For instance, if test case A found an average of 2 defects over the last 10 runs, and this execution found 3 valid defects, then the defect capture rate = 3 / (2+1) × 100% = 100%.

[0069] Table 2 Parameter Table for Multidimensional Cost Model

[0070]

[0071]

[0072] As shown in Table 2, time cost can be calculated by execution time and the hourly rate of the machine used; resource cost can be calculated by memory usage, test duration, and shipping cost; and maintenance cost can be calculated by the number of test case lines and the maintenance complexity factor. Test cost = log(1 + time cost + resource cost) + 0.5 × maintenance cost.

[0073] For the environmental adaptability coefficient, a dynamic adjustment factor of the environment can be considered.

[0074] Table 3 Environmental Adaptability Coefficients for Environmental Scenarios

[0075] Environment Scene coefficient value illustrate Production Environment 1.5 Discovering production defects is of higher value Pre-release environment 1.2 CI / CD pipeline 1.0 benchmark value Develop local environment 0.7 Early defects have lower costs

[0076] As shown in Table 3, the coefficients differ for different environmental scenarios.

[0077] After obtaining the relevant parameters of the test cases, the performance value can be calculated. For example, for test case X, if:

[0078] Two defects were found (severity levels P1 and P2 respectively);

[0079] Historical average defects: 1.5;

[0080] Execution time: 3 minutes (machine hour price: 0.5 yuan / minute);

[0081] Resource usage: 4GB memory × 0.1 yuan / GB / minute;

[0082] Operating environment: Pre-release environment;

[0083] Then we can calculate:

[0084] Defect capture rate = 2 / (1.5+1) = 80%;

[0085] Time cost = 3 × 0.5 = 1.5 yuan;

[0086] Resource cost = 4GB × 3 minutes × 0.1 = 1.2 yuan;

[0087] Test cost = log(1 + 1.5 + 1.2) + 0 = log(3.7) ≈ 1.3;

[0088] Environmental factor = 1.2;

[0089] Final efficiency value = 80% / 1.3 * 1.2 ≈ 73.8%.

[0090] Compared with traditional methods, the method in this embodiment obtains indicators that include defect value, cost, and environmental weighting. It can perform dynamic rolling calculations through a time window mechanism, and obtain the performance value of test cases in real time. This performance value can be used to directly guide the allocation of test resources, providing a quantifiable basis for subsequent dynamic adjustment of test case priorities.

[0091] Example 2

[0092] Figure 2 This is a schematic diagram of a test case performance evaluation device provided in Embodiment 2 of the present invention. The device is applicable to evaluating the performance of test cases during or after their execution. The device can be implemented by software and / or hardware and is generally integrated into an electronic device.

[0093] like Figure 2 As shown, the device includes:

[0094] The acquisition module 210 is used to acquire relevant parameters of the test cases during execution, including defect-related parameters, test cost-related parameters, and environmental scenarios.

[0095] The first analysis module 220 is used to analyze the defect-related parameters to obtain the defect capture rate of the test case during execution.

[0096] The second analysis module 230 is used to analyze the test cost-related parameters to obtain the test cost of the test case during execution.

[0097] The determination module 240 is used to determine the test performance value of the test case based on the defect capture rate, the test cost, and the environment adaptation coefficient corresponding to the environment scenario.

[0098] This embodiment provides a test case performance evaluation device, comprising: an acquisition module for acquiring relevant parameters of the test case during execution, the relevant parameters including defect-related parameters, test cost-related parameters, and environmental scenarios; a first analysis module for analyzing the defect-related parameters to obtain the defect capture rate of the test case during execution; a second analysis module for analyzing the test cost-related parameters to obtain the test cost of the test case during execution; and a determination module for determining the test performance value of the test case based on the defect capture rate, the test cost, and the environmental adaptation coefficient corresponding to the environmental scenarios. By acquiring and analyzing the relevant parameters of the test case during execution, the test performance value of the test case can be obtained, thereby determining the performance of the test case and solving the problem that existing performance evaluation methods cannot comprehensively quantify test performance.

[0099] Furthermore, the defect-related parameters include the defects discovered by the test cases and the corresponding defect levels. Accordingly, the first analysis module 220 includes:

[0100] The number of valid defects discovered in this instance is determined based on the defects and the corresponding defect levels.

[0101] Based on the number of valid defects and the historical average number of defects found for the test case, the defect capture rate of the test case in this test is determined.

[0102] Furthermore, determining the number of valid defects discovered this time based on the defect and the defect level corresponding to the defect includes:

[0103] Based on the defect level, quantity, and total number of defects for each discovered defect, calculate the number of valid defects discovered this time;

[0104] The defect levels include Level 0, Level 1, Level 2, Level 3, and Level 4.

[0105] Furthermore, the test cost-related parameters include time cost indicators, resource cost indicators, and maintenance cost indicators; correspondingly, the second analysis module 230 includes:

[0106] Based on the time cost metric, determine the time cost required to execute the test case;

[0107] Based on the resource cost metric, determine the resource cost required to execute the test case;

[0108] Based on the maintenance cost metric, determine the maintenance cost required to execute the test case;

[0109] By inputting the time cost, the resource cost, and the maintenance cost into a multidimensional cost model, the test cost of the test case during execution is obtained.

[0110] Furthermore, the time cost metric includes the execution time of the test case and the cost of the machine required to execute the test case; the resource cost metric includes the memory occupied by the test case, the test duration, and the cost of the cloud server; and the maintenance cost includes the number of lines of code for the test case and the maintenance complexity factor.

[0111] Furthermore, the environmental scenarios include local development environment, testing environment, pre-release environment, and production environment.

[0112] Furthermore, module 240 is defined as including:

[0113] The product of the defect capture rate, the reciprocal of the test cost, and the environment adaptation coefficient corresponding to the environment scenario is used as the test performance value of the test case.

[0114] The above-mentioned test case performance evaluation device can execute the test case performance evaluation method provided in any embodiment of the present invention, and has the corresponding functional modules and beneficial effects of the execution method.

[0115] Example 3

[0116] Figure 3 A schematic diagram of an electronic device 10 that can be used to implement embodiments of the present invention is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices (e.g., helmets, glasses, watches, etc.), and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the invention described and / or claimed herein.

[0117] like Figure 3As shown, the electronic device 10 includes at least one processor 11 and a memory, such as a read-only memory (ROM) 12 or a random access memory (RAM) 13, communicatively connected to the at least one processor 11. The memory stores computer programs executable by the at least one processor. The processor 11 can perform various appropriate actions and processes based on the computer program stored in the ROM 12 or loaded from storage unit 18 into the RAM 13. The RAM 13 may also store various programs and data required for the operation of the electronic device 10. The processor 11, ROM 12, and RAM 13 are interconnected via a bus 14. An input / output (I / O) interface 15 is also connected to the bus 14.

[0118] Multiple components in electronic device 10 are connected to I / O interface 15, including: input unit 16, such as keyboard, mouse, etc.; output unit 17, such as various types of displays, speakers, etc.; storage unit 18, such as disk, optical disk, etc.; and communication unit 19, such as network card, modem, wireless transceiver, etc. Communication unit 19 allows electronic device 10 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0119] Processor 11 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of processor 11 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. Processor 11 performs the various methods and processes described above, such as performance evaluation methods for test cases.

[0120] In some embodiments, the test case performance evaluation method may be implemented as a computer program tangibly contained in a computer-readable storage medium, such as storage unit 18. In some embodiments, part or all of the computer program may be loaded and / or installed on electronic device 10 via ROM 12 and / or communication unit 19. When the computer program is loaded into RAM 13 and executed by processor 11, one or more steps of the test case performance evaluation method described above may be performed. Alternatively, in other embodiments, processor 11 may be configured to execute the test case performance evaluation method by any other suitable means (e.g., by means of firmware).

[0121] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), payload-programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0122] Computer programs used to implement the methods of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, such that when executed by the processor, the computer programs cause the functions / operations specified in the flowcharts and / or block diagrams to be performed. The computer programs may be executed entirely on a machine, partially on a machine, or as a standalone software package, partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0123] In the context of this invention, a computer-readable storage medium can be a tangible medium that may contain or store a computer program for use by or in conjunction with an instruction execution system, apparatus, or device. A computer-readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination thereof. Alternatively, a computer-readable storage medium may be a machine-readable signal medium. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0124] To provide interaction with a user, the systems and techniques described herein can be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the electronic device. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0125] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as data servers), or computing systems that include middleware components (e.g., application servers), or computing systems that include frontend components (e.g., user computers with graphical user interfaces or web browsers through which users can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., communication networks). Examples of communication networks include local area networks (LANs), wide area networks (WANs), blockchain networks, and the Internet.

[0126] A computing system can include clients and servers. Clients and servers are generally located far apart and typically interact through communication networks. The client-server relationship is created by computer programs running on the respective computers and having a client-server relationship with each other. The server can be a cloud server, also known as a cloud computing server or cloud host, which is a hosting product within the cloud computing service system to address the shortcomings of traditional physical hosts and VPS services, such as high management difficulty and weak business scalability.

[0127] It should be understood that the various forms of processes shown above can be used, with steps reordered, added, or deleted. For example, the steps described in this invention can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution of this invention can be achieved, and this is not limited herein.

[0128] The specific embodiments described above do not constitute a limitation on the scope of protection of this invention. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this invention should be included within the scope of protection of this invention.

Claims

1. A method for evaluating the effectiveness of test cases, characterized in that, The method includes: Obtain relevant parameters when the test cases are executed, including defect-related parameters, test cost-related parameters, and environmental scenarios; The defect-related parameters are analyzed to obtain the defect capture rate of the test cases during execution. The test cost-related parameters are analyzed to obtain the test cost of the test case during execution. The test performance value of the test case is determined based on the defect capture rate, the test cost, and the environment adaptation coefficient corresponding to the environment scenario.

2. The method according to claim 1, characterized in that, The defect-related parameters include the defects discovered by the test cases and the corresponding defect levels. Accordingly, the analysis of the defect-related parameters to obtain the defect capture rate of the test cases during execution includes: The number of valid defects discovered in this instance is determined based on the defects and the corresponding defect levels. Based on the number of valid defects and the historical average number of defects found for the test case, the defect capture rate of the test case in this test is determined.

3. The method according to claim 2, characterized in that, The determination of the number of valid defects discovered this time based on the defect and the defect level corresponding to the defect includes: Based on the defect level, quantity, and total number of defects for each discovered defect, calculate the number of valid defects discovered this time; The defect levels include Level 0, Level 1, Level 2, Level 3, and Level 4.

4. The method according to claim 1, characterized in that, The test cost-related parameters include time cost indicators, resource cost indicators, and maintenance cost indicators; correspondingly, the analysis of these test cost-related parameters to obtain the test cost of the test case during execution includes: Based on the time cost metric, determine the time cost required to execute the test case; Based on the resource cost metric, determine the resource cost required to execute the test case; Based on the maintenance cost metric, determine the maintenance cost required to execute the test case; By inputting the time cost, the resource cost, and the maintenance cost into a multidimensional cost model, the test cost of the test case during execution is obtained.

5. The method according to claim 4, characterized in that, The time cost metric includes the execution time of the test case and the cost of the machine required to execute the test case. The resource cost metric includes the memory occupied by the test case, the test duration, and the cost of the cloud server. The maintenance cost includes the number of lines of code for the test case and the maintenance complexity factor.

6. The method according to claim 1, characterized in that, The environmental scenarios include local development environment, testing environment, pre-release environment, and production environment.

7. The method according to claim 1, characterized in that, The step of determining the test performance value of the test case based on the defect capture rate, the test cost, and the environment adaptability coefficient corresponding to the environment scenario includes: The product of the defect capture rate, the reciprocal of the test cost, and the environment adaptation coefficient corresponding to the environment scenario is used as the test performance value of the test case.

8. A device for evaluating the effectiveness of test cases, characterized in that, The device includes: The acquisition module is used to acquire relevant parameters of the test cases during execution, including defect-related parameters, test cost-related parameters, and environmental scenarios. The first analysis module is used to analyze the defect-related parameters to obtain the defect capture rate of the test cases during execution. The second analysis module is used to analyze the test cost-related parameters to obtain the test cost when the test case is executed; The determination module is used to determine the test performance value of the test case based on the defect capture rate, the test cost, and the environment adaptation coefficient corresponding to the environment scenario.

9. An electronic device, characterized in that, The device includes: At least one processor; and A memory communicatively connected to the at least one processor; wherein, The memory stores a computer program that can be executed by the at least one processor, the computer program being executed by the at least one processor to enable the at least one processor to perform the performance evaluation method of the test cases according to any one of claims 1-7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer instructions that cause a processor to execute the performance evaluation method of the test cases according to any one of claims 1-7.

Images