Intelligent mechanism for evaluating performance of test engineers

Abstract

Embodiments of the present disclosure relate to an intelligent mechanism for evaluating performance of a test engineer. Methods, systems, and non-transitory processor-readable storage media for a resource assessment system are provided herein. An example method includes obtaining defect data obtained in response to executing software on a plurality of tester systems. The resource evaluation system determines a performance weighting metric associated with the defect detection resource and the defect data. Using the performance weighted metric, the resource evaluation system determines a defect range delta percentage associated with the defect detection resource and the defect data. The resource assessment system then determines a defect detection resource score based on the defect range delta percentage.

Description

Technical Field

[0001] The field relates generally to evaluating defect detection performance, and more particularly to evaluating defect detection performance in information processing systems. Background Art

[0002] Customers demand high-quality software, and comprehensive test coverage is an integral part of software quality. Therefore, comprehensive test coverage of the software provided by test engineers is critical to the success of the software project. Therefore, accurate evaluation of the test engineers who perform the testing is crucial. Summary of the invention

[0003] The illustrative embodiments provide techniques for implementing a resource assessment system in a storage system. For example, in the illustrative embodiments, the resource assessment system obtains defect data in response to executing software on a plurality of tester systems. The resource assessment system determines a performance weighted metric associated with a defect detection resource and the defect data. Using the performance weighted metric, the resource assessment system determines a defect coverage delta percentage associated with the defect detection resource and the defect data. The resource assessment system determines a defect detection resource score based on the defect coverage delta percentage.

[0004] In other embodiments, other types of processing devices may be used.These and other illustrative embodiments include, but are not limited to, apparatus, systems, methods, and processor-readable storage media. BRIEF DESCRIPTION OF THE DRAWINGS

[0005] Figure 1 An information processing system including a resource assessment system in an illustrative embodiment is shown;

[0006] Figure 2 A flow chart showing the process of the resource assessment system in an illustrative embodiment; and

[0007] Figure 3 and Figure 4 An example of a processing platform that may be used to implement at least a portion of an embodiment of a resource assessment system is shown. DETAILED DESCRIPTION

[0008] Illustrative embodiments will be described herein with reference to an exemplary computer network and associated computers, servers, network devices, or other types of processing devices. However, it should be understood that these and other embodiments are not limited to use with the specific illustrative network and device configurations shown. Therefore, the term "computer network" as used herein is intended to be interpreted broadly to include, for example, any system including multiple networked processing devices.

[0009] The following describes techniques for implementing a resource assessment system that can be used to provide, among other things, an assessment of defect detection resources. The resource assessment system acquires defect data in response to executing software on a plurality of tester systems. The resource assessment system determines a performance weighted metric associated with the defect detection resources and the defect data. Using the performance weighted metric, the resource assessment system determines a defect coverage delta percentage associated with the defect detection resources and the defect data. The resource assessment system determines a defect detection resource score based on the defect coverage delta percentage.

[0010] Every team needs an efficient way to evaluate the performance of team members, not only the time spent on different types of tasks, but also the professional ability to handle the assigned tasks. For example, performance evaluation can include team member productivity and the breadth and depth of team member knowledge. The breadth of skills and depth of knowledge of team members are the technical strength of the company and the core competitiveness of team members, teams and companies.

[0011] Conventional techniques for assessing resources do not fully assess team member performance. Conventional techniques for assessing resources do not assess the breadth and depth of team member knowledge. Conventional techniques for assessing resources do not assess resource performance based on multiple skill levels. Conventional techniques for assessing resources do not assess resource performance using both skill level and productivity. Conventional techniques for assessing resources do not provide accurate resource assessments. Conventional techniques for assessing resources do not provide the ability to use historical data for various skill levels. Conventional techniques for assessing resources do not improve the applicability and feasibility of assessing resource performance by using historical data for various skill levels.

[0012] In contrast, in at least some implementations of the current technology described herein, in response to executing software on a plurality of tester systems, resource assessment is optimized by acquiring defect data. A resource assessment system determines a performance weighted metric associated with defect detection resources and defect data. Using the performance weighted metric, the resource assessment system determines a defect coverage delta percentage associated with the defect detection resources and defect data. The resource assessment system determines a defect detection resource score based on the defect coverage delta percentage.

[0013] Therefore, an object of the current technology is to provide a method and system for providing a resource evaluation system that can evaluate the defect detection performance of resources that are executing test cases / scripts. Another object is to comprehensively evaluate the performance of team members. Another object is to evaluate the breadth and depth of team member knowledge. Another object is to evaluate the performance of resources based on multiple skill levels. Another object is to use both skill level and productivity to evaluate the performance of resources. Another object is to provide accurate resource evaluation. Another object is to provide the ability to use historical data of various skill levels. Yet another object is to improve the applicability and feasibility of evaluating resource performance by using historical data of various skill levels.

[0014] In at least some implementations of the current technology described herein, use of a resource assessment system can provide one or more of the following advantages: comprehensively assessing team member performance, assessing the breadth and depth of team member knowledge, assessing the performance of resources based on multiple skill levels, assessing resource performance using both skill level and productivity, providing accurate resource assessments, providing the ability to use historical data for various skill levels, and improving the applicability and feasibility of assessing resource performance by using historical data for various skill levels.

[0015] Compared to conventional techniques, in at least some implementations of the current techniques described herein, in response to executing software on a plurality of tester systems, resource assessment is optimized by acquiring defect data. A resource assessment system determines a performance weighted metric associated with defect detection resources and defect data. Using the performance weighted metric, the resource assessment system determines a defect coverage delta percentage associated with the defect detection resources and defect data. The resource assessment system determines a defect detection resource score based on the defect coverage delta percentage.

[0016] In an example embodiment of the current technology, software is executed on multiple tester systems. An issue tracking system obtains defect data generated by the software execution and associates the defect data with defect detection resources. The issue tracking system then stores the defect data in a defect database.

[0017] In an example embodiment of the present technology, the performance weighted metric includes a defect priority associated with the defect data.

[0018] In an example embodiment of the present technology, a defect priority is associated with a defect detection resource, wherein the defect detection resource generates the defect data.

[0019] In an example embodiment of the present technology, the performance weighted metric includes a priority weight associated with a defect priority associated with the defect data.

[0020] In an example embodiment of the present technology, the performance weighted metric includes a plurality of priority weights, each priority weight being associated with a respective defect priority of a plurality of defect priorities associated with the defect data, wherein the plurality of priority weights are equal to one when aggregated.

[0021] In an example embodiment of the present technology, the performance weighted metric includes a plurality of detected defects associated with a defect priority associated with the defect data, wherein the plurality of detected defects are weighted with a priority weight associated with the defect priority associated with the defect data.

[0022] In an example embodiment of the present technology, the performance weighted metric includes aggregating a plurality of weighted detected defects, wherein each defect in the plurality of weighted detected defects is associated with a respective defect priority and weighted with a respective priority weight.

[0023] In an example embodiment of the present technology, a resource assessment system calculates a percentage difference between a performance weighted metric and a historical performance weighted metric associated with a plurality of defect detection resources.

[0024] In an example embodiment of the present technology, the historical performance weighted metric includes a number of detected defects associated with a defect detection resource.

[0025] In an example embodiment of the present technology, the historical performance weighted metric includes a plurality of detected defects associated with a second defect detection resource, wherein the second defect detection resource is associated with a defect priority associated with the defect data.

[0026] In an example embodiment of the present technology, the historical performance weighted metric includes a plurality of detected defects associated with a second defect detection resource, wherein the second defect detection resource is associated with a second defect priority level associated with the defect data.

[0027] In an example embodiment of the present technology, the resource assessment system applies an impact factor to a second defect priority level associated with the defect data.

[0028] In an example embodiment of the present technology, a resource assessment system determines a test case percentage associated with a defect detection resource, wherein the test case percentage indicates a percentage of test cases executed by the defect detection resource relative to a plurality of defect detection resources.

[0029] In an example embodiment of the present technology, the resource assessment system normalizes the defect coverage variance percentage and the test case percentage.

[0030] In an example embodiment of the present technology, the resource assessment system weights the defect coverage variance percentage using the defect weight and weights the test case percentage using the test case weight.

[0031] In an example embodiment of the present technology, defect weights and test case weights are equal to one when aggregated.

[0032] In an example embodiment of the present technology, a resource assessment system adjusts at least one of a defect weight and a test case weight.

[0033] Figure 1 A computer network (also referred to herein as an information processing system) 100 configured in accordance with an illustrative embodiment is shown. The computer network 100 includes a defect database 101, a resource assessment system 105, and a tester system 102-N. The defect database 101, the resource assessment system 105, and the tester system 102-N are coupled to a network 104, where in this embodiment, the network 104 is assumed to represent a sub-network or other related portion of a larger computer network 100. Therefore, elements 100 and 104 are both referred to herein as examples of a "network," but in the example of Figure 1 In the context of the embodiments of the present invention, the latter is assumed to be a component of the former. Also coupled to the network 104 is a resource assessment system 105 which may reside on a storage system. Such a storage system may include any of a variety of different types of storage devices, including network attached storage (NAS), storage area network (SAN), direct attached storage (DAS) and distributed DAS, as well as combinations of these and other storage types, including software defined storage devices.

[0034] Each of the tester systems 102-N may include, for example, a server, and / or portions of one or more server systems, and devices such as mobile phones, laptop computers, tablet computers, desktop computers, or other types of computing devices. Such devices are examples of devices more generally referred to herein as "processing devices." Some of these processing devices are also often referred to herein as "computers."

[0035] In some embodiments, the tester system 102-N includes respective computers associated with a particular company, organization, or other enterprise. In addition, at least a portion of the computer network 100 may also be collectively referred to herein as comprising an "enterprise network." As will be appreciated by those skilled in the art, many other operational scenarios involving a variety of different types and arrangements of processing devices and networks are possible.

[0036] Furthermore, it should be understood that in the context of this document and elsewhere, the term "user" is intended to be broadly interpreted to include, for example, humans, hardware, software or firmware entities, and various combinations of such entities.

[0037] The network 104 is assumed to comprise a portion of a global computer network such as the Internet, but other types of networks may be part of the computer network 100, including a wide area network (WAN), a local area network (LAN), a satellite network, a telephone or wired network, a cellular network, a wireless network such as a Wi-Fi or WiMAX network, or various portions or combinations of these and other types of networks. Thus, in some embodiments, the computer network 100 comprises a combination of multiple different types of networks, each of which includes processing devices configured to communicate using the Internet Protocol (IP) or other related communication protocols.

[0038] Also associated with the resource assessment system 105 are one or more input-output devices, which illustratively include any combination of keyboards, displays, or other types of input-output devices. For example, such input-output devices can be used to support one or more user interfaces to the resource assessment system 105, as well as to support communication between the resource assessment system 105 and other related systems and devices not explicitly shown. For example, a dashboard can be provided for the user to view the execution progress of the resource assessment system 105. One or more input-output devices can also be associated with any one of the tester systems 102-N.

[0039] In addition, assuming Figure 1 The resource assessment system 105 in the embodiment is implemented using at least one processing device. Each such processing device generally includes at least one processor and associated memory, and implements one or more functional modules for controlling certain features of the resource assessment system 105.

[0040] More specifically, the resource assessment system 105 in this embodiment may include a processor coupled to a memory and a network interface.

[0041] A processor illustratively includes a microprocessor, a microcontroller, an application specific integrated circuit (ASIC), a field programmable gate array (FPGA), or other type of processing circuitry, as well as portions or combinations of such circuitry elements.

[0042] Memory illustratively includes any combination of random access memory (RAM), read-only memory (ROM), or other types of memory. The memory and other memories disclosed herein may be viewed as examples of what is more generally referred to as a "processor-readable storage medium" that stores executable computer program code or other types of software programs.

[0043] One or more embodiments include articles of manufacture, such as computer-readable storage media. Examples of articles of manufacture include, but are not limited to, storage devices (such as storage disks, storage arrays, or integrated circuits including memory), and various other types of computer program products. The term "article of manufacture" as used herein should be understood to exclude transient propagating signals. These and other references to "disk" herein generally refer to storage devices, including solid-state drives (SSDs), and therefore should not be considered to be limited in any way to rotating magnetic media.

[0044] The network interface allows the resource assessment system 105 to communicate with the defect database 101 and the tester systems 102 -N over the network 104 , and illustratively includes one or more conventional transceivers.

[0045] The resource assessment system 105 may be implemented at least in part in the form of software stored in a memory and executed by a processor, and may reside in any processing device. The resource assessment system 105 may be a stand-alone plug-in that may be included within a processing device.

[0046] It should be understood that Figure 1 The specific set of components shown for the resource assessment system 105 and the tester system 102-N of the computer network 100 involving the defect database 101 is presented by way of illustrative example only, and in some other embodiments, additional or alternative components may be used. Thus, another embodiment includes additional or alternative systems, devices and other network entities, and different arrangements of modules and other components. For example, in at least one embodiment, one or more of the resource assessment systems 105 may be on the same processing platform and / or be part of the same processing platform.

[0047] References such as Figure 2 The flowchart of FIG. 1 describes an exemplary process of the resource assessment system 105 in the computer network 100 in more detail.

[0048] Figure 2 is a flow chart of a process for executing resource assessment system 105 in an illustrative embodiment. It should be appreciated that this particular process is merely one example and that additional or alternative processes may be executed in other embodiments.

[0049] In an example embodiment, software is executed on multiple tester systems 102-N. An issue tracking system, such as a system that allows for bug tracking (i.e., defect tracking and collection) and agile project management, is running in the background to collect data from the software executed on the multiple tester systems 102-N. The issue tracking system obtains defect data generated by the software execution and associates the defect data with defect detection resources. The issue tracking system then stores the defect data in a defect database.

[0050] At 200, the resource assessment system 105 obtains defect data obtained in response to executing software on multiple tester systems. For example, the resource assessment system 105 can obtain the defect data directly from the defect database 101. In an example embodiment, the defect data can be transmitted to the resource assessment system 105 when the defect data is generated and / or when the defect data is stored in the defect database 101. In an example embodiment, the resource assessment system 105 can continue to run in the background while the software is executed on the multiple tester systems 102-N to obtain the defect data.

[0051] At 202, the resource assessment system 105 determines a performance weighted metric associated with the defect detection resource and the defect data. In an example embodiment, the performance weighted metric (i.e., C 错误 ) can be defined as:

[0052] C 错误 =ω P00 ·C P00 +ω P01 ·C P01 +ω P02 ·C P02

[0053] In an example embodiment, the performance weighted metric includes a defect priority associated with the defect data. In the above equation, the defect priority is represented by "P00", "P01", and "P02". In an example embodiment, "P00" is the highest defect priority, "P01" is the second highest defect priority, and "P02" is the next highest defect priority.

[0054] In an example embodiment, defect priorities are associated with defect detection resources, where the defect detection resources generate defect data. For example, when running a test case on the tester system 102-N, a defect detection resource such as a test engineer can generate / call / trigger defects (i.e., "errors") during testing (through their skills). The defect detection resource can manually test the tester system 102-N and / or automatically test the tester system 102-N using a test case / script. For example, a highly skilled defect detection resource can execute a test case / script to test the tester system 102-N and can trigger seven "P01" errors because the highly skilled tester uses different parameter settings (e.g., using different I / O tools, different cycle durations, different feature toggles, etc.) to execute the test case and script when executing the test case / script. A medium-skilled defect detection resource can execute the same test case / script on the tester system 102-N and only trigger one "P02" error because the medium-skilled defect detection resource only uses some common parameter settings in the test case / script to run the test case / script. In this example, the highly skilled defect detection resources demonstrated better "bug hunting" capabilities compared to the averagely skilled defect detection resources.

[0055] In the example embodiment, in the above equation, C P00 represents the error count (i.e., defect data) associated with the “P00” defect priority level, C P01 represents the error count associated with the "P01" defect priority, and C P02 Represents the error count associated with the 'P02' defect priority level.

[0056] In an example embodiment, the performance weighted metric includes a priority weight associated with a defect priority associated with the defect data. In the above equation, the priority weight is ω P00 ,ω P01 and ω P02 In an example embodiment, the performance weighted metric includes a plurality of priority weights, each priority weight being associated with a corresponding defect priority in a plurality of defect priorities, the plurality of defect priorities being associated with the defect data. In the above formula, ω P00 is the priority weight associated with the “P00” defect priority, ω P01 is the priority weight associated with the “P01” defect priority, ω P02 is the priority weight associated with the "P02" defect priority. In the example embodiment, multiple priority weights are equal to one when aggregated. In other words, ω P00 +ω P01 +ω P02 =1.

[0057] Thus, in an example embodiment, the performance weighted metric includes a number of detected defects associated with a defect priority associated with the defect data, wherein the number of detected defects is weighted with a priority weight associated with the defect priority associated with the defect data. In other words, the number of errors C associated with the priority weight "P00" is P00 is the priority weight ω P00 Weighted, we get ω P00 ·C P00 The number of errors C associated with priority weight "P01" P01 is the priority weight ω P01 Weighted, we get ω P01 ·C P01 The number of errors C associated with priority weight "P02" P02 is the priority weight ω P02 Weighted, we get ω P02 ·C P02 In addition, the performance weighted metric (C 错误 ) comprises aggregating a plurality of weighted pluralities of detected defects, wherein each defect in the plurality of weighted pluralities of detected defects is associated with a respective defect priority and weighted with a respective priority weight:

[0058] C 错误 =ω P00 ·C P00 +ω P01 ·C P01 +ω P02 ·C P02

[0059] In an example embodiment, the priority weights are adjusted to achieve the estimated requirements of the defect detection resources.

[0060] The following table lists the number of errors (i.e., defect data) found by tester A, tester B, tester C, and tester D according to the defect priority of "P00", "P01", and "P02". In the example embodiment, the priority weight of "P00" is 0.5, the priority weight of "P01" is 0.3, and the priority weight of "P02" is 0.2, resulting in a performance weighted metric equal to 0.5*C P00 +0.3*C P01 +0.2*C P02 By calculating the performance weighted metric, the results show that the defect detection resource “Tester A” scores higher than the other defect detection resources (Tester B, Tester C, and Tester D).

[0061] Error Count Tester A Tester B Tester C Tester D P00 23 44 3 44 P01 68 20 99 12 P02 12 2 6 12 Performance Weighting 34.3 28.4 32.4 28 measure √

[0062] At 204, using the performance weighted metrics, the resource assessment system 105 determines a defect coverage delta percentage associated with the defect detection resource and the defect data. In an example embodiment, some test cases / scripts may generate a higher amount of defect data (i.e., a greater number of errors) than other test cases / scripts. In an example embodiment, the defect coverage delta percentage compares the amount of defect data generated by the defect detection resource to the amount of defect data that the particular defect detection resource has historically generated using the same test case / script range over a given historical period.

[0063] In an example embodiment, the resource assessment system 105 calculates a percentage difference between the performance weighted metric and a historical performance weighted metric associated with a plurality of defect detection resources. In an example embodiment, the historical performance weighted metric includes a plurality of detected defects associated with the defect detection resources. In an example embodiment, the defect coverage difference is defined as:

[0064] Δ 错误 =C 错误，现在 -C 错误，历史

[0065] Therefore, the percentage increase and / or decrease in the defect range difference (i.e., the defect range difference percentage) is:

[0066]

[0067] The following table illustrates the calculated defect coverage variance percentages:

[0068] Tester A Tester B Tester C Tester D Historical performance weighted metrics 36 12 66 19 Performance Weighted Metrics 34.3 28.4 32.4 28 Difference / reduction: Δ -1.7 16.4 -33.6 9 <![CDATA[Percentage: Δ% 错误 > -4.72％ 136.67％ -50.91％ 47.37％

[0069] In this example embodiment, tester A has the highest performance weighted metric, but tester A's productivity is reduced by 1.7% and 4.72% relative to the historical defect data, while tester B has an overall lower performance weighted metric, but tester B's productivity is increased by 16.4% and 136.6% relative to the historical defect data. In the example embodiment, this data indicates that tester B is very skilled at generating defect data (i.e., "bug hunting").

[0070] In an example embodiment, the historical performance weighted metric includes a plurality of detected defects associated with a second defect detection resource, wherein the second defect detection resource is associated with a defect priority associated with the defect data. In an example embodiment, when the historical defect data associated with the "first" defect detection resource is not available, the performance weighted metric of the "first" defect detection resource may be calculated using the historical defect data of the second defect detection resource. For example, tester A generated defect data using 10 test cases associated with the same test domain.

[0071] In an example embodiment, there may be different skill levels, such as:

[0072] Level 0: Beginner

[0073] Level 1: Familiarity

[0074] Level 2: Mastery

[0075] Level 3: Expert

[0076] In this example scenario, Tester A's skill level for all 10 test cases is "Level 1: Familiar".

[0077] In this example embodiment, the performance weighted metric for tester A is calculated using the following equation:

[0078] C 错误，现在 =ω P00 ·C P00 +ω P01 ·C P01 +ω P02

[0079] If tester A did not execute these 10 test cases, there would be no historical defect data in the defect database 101 for calculating the historical performance weighted metric. However, there may be historical defect data for 10 test cases in the defect database 101, where tester B executed the 10 test cases and tester B also has a skill level of "Level 1: Familiar" for all 10 test cases. In this example scenario, the resource assessment system 105 uses the historical defect data associated with tester B (i.e., the second defect detection resource) to calculate the performance weighted metric for tester A (i.e., the "first" defect detection resource).

[0080] In an example embodiment, the historical performance weighted metric includes a plurality of detected defects associated with a second defect detection resource, wherein the second defect detection resource is associated with a second defect priority level associated with the defect data. If tester A did not execute the 10 test cases, there would be no historical defect data in the defect database 101 for calculating the historical performance weighted metric. In an example embodiment, there may also be no historical defect data in the defect database 101 for any other tester that also has the skill level "Level 1: Familiar". However, there may be historical defect data for 10 test cases in the defect database 101, wherein tester C executed the 10 test cases and tester C has the skill level "Level 2: Proficient" for all 10 test cases. In this example scenario, the resource evaluation system 105 uses the historical defect data associated with tester C (i.e., the second defect detection resource) associated with the second defect priority level (i.e., the skill level "Level 2: Proficient") to calculate the performance weighted metric for tester A (i.e., the "first" defect detection resource).

[0081] In an example embodiment, when determining the defect coverage delta percentage, the impact factor is applied to the second defect priority associated with the defect data. For example, tester A (skill level is "Level 1: Familiar") has historical defect data of 100 errors. 错误，现在 The value will be calculated as:

[0082] C 错误，现在 =ω P00 ·C P00 +ω P01 ·C P01 +ω P02

[0083] In the example embodiment, 50 errors are detected by the test cases / scripts associated with the skill level "Level 1: Familiar", and there is historical defect data, so C 错误，现在 The value will be calculated as:

[0084] C 错误，level1 =ω P00，level1 ·C P00，level1 +ω P01，level1 ·C P01，level1 +ω P02，level1 ·C P02，level1

[0085] However, if 50 errors are detected by test cases / scripts associated with skill level "Level 0: Novice", and there is no historical defect data for any defect detection resource with skill level "Level 0: Novice", the resource assessment system 105 applies the impact factor when determining the defect coverage difference percentage.

[0086]

[0087] This yields the following equation:

[0088] C 错误，历史 =C 错误，level1 +C 历史，level0

[0089] C 错误，历史 Used to calculate the defect range difference percentage:

[0090]

[0091] At 206, the resource assessment system 105 determines a defect detection resource score based on the defect coverage delta percentage. In an example embodiment, the defect detection resource score combines the effort that the defect detection resource expends on different kinds of tasks. In an example embodiment, a defect detection resource may be assigned a certain number of test cases / scripts to be executed. The test case / script may be denoted by "TC", and the number of test cases / scripts (i.e., "count") may be denoted by C. tc In other words, “m” can be associated with a test engineer in a test engineer team.

[0092] In the exemplary embodiment, resource assessment system 105 uses defect weight ω Δ Weight the defect range difference percentage and use the test case weight ω tc The test case percentages are weighted. In an example embodiment, the defect weight and the test case weight are equal to one when aggregated; Δ +ω tc =1.

[0093] In an example embodiment, the resource assessment system 105 adjusts at least one of the defect weight and the test case weight. In an example scenario, the defect weight may be set to 0.5, and the test case weight may be set to 0.5. Each of these weights may be adjusted (e.g., 0.7 / 0.3 instead of 0.5 / 0.5) depending on which of the defect weight and the test case weight should have a greater impact when calculating the defect detection resource score.

[0094] In an example embodiment, the defect detection resource score (i.e., the "score" in the equation below) may be represented by the following equation:

[0095] Score = ω Δ Δ% 错误 ′+ω tc ·C% tc '

[0096] In an example embodiment, the resource assessment system 105 determines a test case percentage associated with a defect detection resource, where the test case percentage indicates a percentage of test cases executed by the defect detection resource relative to a plurality of defect detection resources. This can be expressed as:

[0097]

[0098] In an example embodiment, the resource assessment system 105 normalizes the defect coverage delta percentage and the test case percentage. For example, the resource assessment system 105 uses Z-score normalization to normalize Δ% 错误 and C% tc Normalize.

[0099] and

[0100] and

[0101] In an example embodiment, ω Δ and ω tc Each is 0.5. Resource evaluation system 105 calculates defect detection resource scores as follows:

[0102] Tester A Tester B Tester C Tester D Historical performance weighted metrics 36 12 66 19 Performance Weighted Metrics 34.3 28.4 32.4 28 <![CDATA[Δ% 错误 ]]> -4.72％ 136.67％ -50.91％ 47.37％ <![CDATA[Δ% 错误 ′]]> -0.46 1.30 -1.03 0.19 Assigned use case count 88 100 75 94 <![CDATA[C% tc ]]> 24.65％ 28.01％ 21.01％ 26.33％ <![CDATA[C% tc ′]]> -0.12 1.01 -1.33 0.44 Defect Detection Resource Score -0.29 1.15 -1.18 0.32

[0103] The above calculations show that tester B has the best performance. Tester B is not only very skilled in generating defects, but also has a high productivity.

[0104] In an example embodiment, the issue tracking system collects defect data, including test case / script assignments and test case / script executions by defect detection resources, and stores the defect data in a defect database 101 .

[0105] For each defect detection resource, the resource evaluation system 105 obtains a skill level for each assigned test case / script. In an exemplary embodiment, the defect database 101 stores information associated with each defect detection resource. Each defect detection resource is associated with a skill level associated with a domain, where the test case / script belongs to the corresponding domain. In an exemplary embodiment, the skill level of defect detection resource i on domain j is represented as L 测试器i，域j The skill level score of defect detection resource i on test case / script j is denoted as L 测试器i，TCj As mentioned above, in the example embodiment, there are four skill levels:

[0106] Level 0: Novice

[0107] Level 1: Familiar

[0108] Level 2: Mastery

[0109] Level 3: Expert

[0110] In an example embodiment, each defect detection resource is skilled in one or more domains as follows:

[0111]

[0112] According to the above table, in the example embodiment, when tester A is assigned 10 test cases / scripts, 5 of which are associated with the data path, 3 are associated with the control path, and 2 are associated with the GUI, the resource evaluation system 105 calculates the defect detection resource score of tester A on each test case / script. In the example embodiment, different domains may require different skill levels to detect defect data. For example, the data path domain may be a domain that is easy to detect defects, and therefore, defect detection resources may detect a higher number of "errors", but this does not necessarily mean that these defect detection resources have a higher "error hunting" capability.

[0113] In an exemplary embodiment, for each defect detection resource assigned to each test case / script, if there is historical defect data (associated with the same skill level) in the defect database 101, the defect coverage difference percentage is calculated using the existing historical defect data. If there is no historical defect data associated with the same skill level in the defect database 101, but there is historical defect data related to a different skill level, or there is historical defect data associated with multiple skill levels, the impact factor (i.e., IF leveli ) to calculate the defect range difference percentage.

[0114] Level 0 Level 1 Level 2 Level 3 Impact Factor (IF) 0.25 0.5 0.75 1

[0115] In an example embodiment, if there is no historical defect data associated with skill level 0, the error count C of skill level 0 is calculated using the historical defect data associated with skill level 1, skill level 2, and skill level 3. bug , as long as historical deficit data exists for at least one skill level.

[0116]

[0117] In the example embodiment, the newly calculated C 错误，level0 This is then used to calculate the defect detection resource score as described above:

[0118] Score = ω Δ Δ% 错误 ′+ωtc ·C% tc '

[0119] In an example embodiment, if there is no historical defect data for at least one skill level, the value will be zero and the test case / script is most likely a newly designed test case / script with no historical defect data.

[0120] In example embodiments, the resource assessment system 105 can be extended to assess tasks other than defect detection. For example, the resource assessment system 105 can be extended to assess the design of test cases / scripts, the coding of test cases / scripts, etc. In these example scenarios, priority attributes are defined based on the product that the resource assessment system 105 is assessing, and historical data is used to obtain the final performance score.

[0121] Therefore, combined with Figure 2 The specific processing operations and other functions described in the flowcharts are presented only by way of illustrative examples and should not be interpreted as limiting the scope of the present disclosure in any way. For example, in some other embodiments, the order of the process steps can be changed, or some steps can be performed simultaneously with each other instead of serially.

[0122] The illustrative embodiments described above provide significant advantages over traditional methods. For example, some embodiments are configured to significantly improve the assessment of defect detection resources. These and other embodiments can effectively improve the assessment of defect detection resources to achieve the ultimate goal of providing a comprehensive assessment. The embodiments disclosed herein comprehensively assess the performance of team members. The embodiments disclosed herein assess the breadth and depth of team member knowledge. The embodiments disclosed herein assess the performance of resources based on multiple skill levels. The embodiments disclosed herein use both skill levels and productivity to assess the performance of resources. The embodiments disclosed herein provide accurate resource assessments. The embodiments disclosed herein provide the ability to use historical data of various skill levels. The embodiments disclosed herein improve the applicability and feasibility of assessing resource performance by using historical data of various skill levels.

[0123] It should be understood that the specific advantages described above and elsewhere herein are associated with specific illustrative embodiments and need not be present in other embodiments. In addition, the specific types of information processing system features and functions shown in the drawings and described above are only exemplary, and many other arrangements can be used in other embodiments.

[0124] As previously mentioned, at least part of the information handling system 100 can be implemented using one or more processing platforms. Given such a processing platform includes at least one processing device, which includes a processor coupled to a memory. In some embodiments, the processor and memory include corresponding processors and memory elements of a virtual machine or container provided using one or more underlying physical machines. The term "processing device" used herein is intended to be broadly interpreted as including various different arrangements of virtual instances of physical processors, memory and other device components and such components. For example, in some embodiments, a "processing device" may include one or more virtual processors or execute across one or more virtual processors. Therefore, a processing device can be physical or virtual and can be executed on one or more physical or virtual processors. It should also be noted that a given virtual device can be mapped to a portion of a physical device.

[0125] Some illustrative embodiments of a processing platform for implementing at least a portion of an information processing system include a cloud infrastructure that includes virtual machines implemented using a hypervisor running on a physical infrastructure. The cloud infrastructure also includes a set of applications running on corresponding virtual machines in the virtual machines under the control of the hypervisor. Multiple hypervisors can also be used, each hypervisor providing a set of virtual machines using at least one underlying physical machine. Different sets of virtual machines provided by one or more hypervisors can be used to configure multiple instances of various components of the system.

[0126] These and other types of cloud infrastructures can be used to provide an environment also referred to herein as a multi-tenant environment. One or more system components, or portions thereof, are illustratively implemented for use by tenants of such a multi-tenant environment.

[0127] As previously mentioned, the cloud infrastructure disclosed herein may include a cloud-based system. In an illustrative embodiment, a virtual machine provided in such a system may be used to implement at least part of a computer system.

[0128] In some embodiments, the cloud infrastructure additionally or alternatively includes multiple containers implemented using a container host device. For example, as described herein, a given container of the cloud infrastructure illustratively includes a Docker container or other type of Linux container (LXC). The container runs on a virtual machine in a multi-tenant environment, although other arrangements are possible. Containers are used to implement various different types of functions within the information processing system 100. For example, containers can be used to implement corresponding processing devices that provide computing and / or storage services for a cloud-based system. Similarly, containers can be used in conjunction with other virtualization infrastructures, such as virtual machines implemented using a hypervisor.

[0129] Now refer to Figure 3 and Figure 4 Illustrative embodiments of the processing platforms are described in further detail. Although described in the context of information handling system 100, in other embodiments, these platforms may also be used to implement at least portions of other information handling systems.

[0130] Figure 3 An example processing platform including a cloud infrastructure 300 is shown. The cloud infrastructure 300 includes a combination of physical and virtual processing resources for implementing at least a portion of the information processing system 100. The cloud infrastructure 300 includes a plurality of virtual machines (VMs) and / or container sets 302-1, 302-2, ..., 302-L implemented using a virtualization infrastructure 304. The virtualization infrastructure 304 runs on a physical infrastructure 305 and illustratively includes one or more hypervisors and / or operating system-level virtualization infrastructure. The operating system-level virtualization infrastructure illustratively includes a kernel control group of a Linux operating system or other type of operating system.

[0131] The cloud infrastructure 300 also includes application sets 310-1, 310-2, ..., 310-L running on corresponding VM / container sets in the VM / container sets 302-1, 302-2, ..., 302-L under the control of the virtualization infrastructure 304. The virtual machine / container set 302 includes a corresponding VM, a corresponding set of one or more containers, or a corresponding set of one or more containers running in a VM. Figure 3 In some implementations of the embodiments, the VM / container set 302 includes respective VMs implemented using a virtualization infrastructure 304 including at least one hypervisor.

[0132] A hypervisor platform may be used to implement a hypervisor within the virtualized infrastructure 304, wherein the hypervisor platform has an associated virtual infrastructure management system. The underlying physical machines include one or more distributed processing platforms including one or more storage systems.

[0133] exist Figure 3 In other implementations of the embodiment, the VM / container set 302 includes corresponding containers implemented using a virtualization infrastructure 304 that provides operating system-level virtualization functionality, such as support for Docker containers running on a bare metal host or Docker containers running on a VM. The containers are illustratively implemented using corresponding kernel control groups of the operating system.

[0134] As described above, one or more of the processing modules or other components of information handling system 100 may each run on a computer, server, storage device, or other processing platform element. Such given elements are considered examples of what is more generally referred to herein as a "processing device." Figure 3 The cloud infrastructure 300 shown may represent at least a portion of a processing platform. Another example of such a processing platform is Figure 4 A processing platform 400 is shown.

[0135] The processing platform 400 in this embodiment comprises a part of the information processing system 100 and includes a plurality of processing devices, represented as 402 - 1 , 402 - 2 , 402 - 3 , . . . , 402 -K, communicating with each other via a network 404 .

[0136] Network 404 includes any type of network including, for example, a global computer network (such as the Internet), a WAN, a LAN, a satellite network, a telephone or wired network, a cellular network, a wireless network (such as a Wi-Fi or WiMAX network), or various portions or combinations of these and other types of networks.

[0137] The processing device 402 - 1 in the processing platform 400 includes a processor 410 coupled to a memory 412 .

[0138] Processor 410 includes a microprocessor, microcontroller, application specific integrated circuit (ASIC), field programmable gate array (FPGA) or other type of processing circuitry, as well as portions or combinations of such circuitry elements.

[0139] The memory 412 includes any combination of random access memory (RAM), read-only memory (ROM), or other types of memory. The memory 412 and other memories disclosed herein should be considered as an illustrative example of an object more generally referred to as a "processor-readable storage medium" that stores executable program code of one or more software programs.

[0140] Articles of manufacture including such processor-readable storage media are considered illustrative embodiments. For example, a given such article of manufacture includes a storage array, a storage disk, or an integrated circuit containing RAM, ROM or other electronic memory, or any of a variety of other types of computer program products. The term "article of manufacture" as used herein should be understood to exclude transient propagation signals. Many other types of computer program products including processor-readable storage media can be used.

[0141] Also included in the processing device 402-1 is a network interface circuitry 414 for interfacing the processing device with the network 404 and other system components, and may include a conventional transceiver.

[0142] It is assumed that the other processing devices 402 of the processing platform 400 are configured in a manner similar to that shown in the figure for the processing device 402-1.

[0143] Again, the specific processing platform 400 shown in the figure is presented only by way of example, and the information processing system 100 may include additional or alternative processing platforms, as well as any combination of many different processing platforms. Each such platform includes one or more computers, servers, storage devices or other processing devices.

[0144] For example, other processing platforms for implementing the illustrative embodiments may include different types of virtualization infrastructures to replace or supplement virtualization infrastructures including virtual machines. Such virtualization infrastructures illustratively include container-based virtualization infrastructures configured to provide Docker containers or other types of LXC.

[0145] As another example, in some embodiments, portions of a given processing platform may include a converged infrastructure.

[0146] It should therefore be understood that in other embodiments, different arrangements of additional or alternative elements may be used.At least a subset of these elements may be implemented together on a common processing platform, or each such element may be implemented on a separate processing platform.

[0147] Furthermore, many other arrangements of computers, servers, storage products or devices, or other components are possible in information handling system 100. Such components may communicate with other elements of information handling system 100 via any type of network or other communications medium.

[0148] For example, in an exemplary embodiment, specific types of storage products that can be used to implement a given storage system of a distributed processing system include all-flash and hybrid flash arrays, extended all-flash arrays, extended NAS clusters, or other types of storage arrays. In illustrative embodiments, a combination of multiple of these and other storage products can also be used to implement a given storage system.

[0149] It should be emphasized again that the above embodiments are presented for illustrative purposes only. Many variations and other alternative embodiments may be used. In addition, in other embodiments, the specific configurations of the systems and device elements schematically shown in the figures and the associated processing operations may be changed. Thus, for example, the specific types of processing devices, modules, systems and resources deployed in a given embodiment and their corresponding configurations may vary. In addition, the various assumptions made in the process of describing the illustrative embodiments above should also be regarded as exemplary, rather than requirements or limitations of the present disclosure. Many other alternative embodiments within the scope of the appended claims will be clear to those skilled in the art.

Claims

1. A method comprising: acquiring, by a resource assessment system, defect data acquired in response to executing software on a plurality of tester systems; determining, by the resource assessment system, a performance weighted metric associated with a defect detection resource and the defect data; determining a defect coverage delta percentage associated with the defect detection resource and the defect data using the performance weighted metric; as well as A defect detection resource score is determined based on the defect coverage delta percentage, wherein the method is implemented by at least one processing device including a processor coupled to a memory.

2. The method according to claim 1, wherein obtaining defect data by the resource assessment system comprises: executing the software on a plurality of tester systems; obtaining defect data generated by execution of the software; associating the defect data with the defect detection resource; as well as The defect data is stored in a defect database. The method of claim 1 , wherein the performance weighted metric comprises a defect priority associated with the defect data. 4 . The method of claim 3 , wherein the defect priority is associated with the defect detection resource, wherein the defect detection resource generated the defect data. 5 . The method of claim 1 , wherein the performance weighted metric comprises a priority weight associated with a defect priority associated with the defect data.

6. The method of claim 1, wherein the performance weighted metric comprises a plurality of priority weights, each priority weight being associated with a respective defect priority of a plurality of defect priorities associated with the defect data, wherein the plurality of priority weights when aggregated equal one.

7. The method of claim 1, wherein the performance weighted metric comprises a plurality of detected defects associated with defect priorities associated with the defect data, wherein the plurality of detected defects are weighted using priority weights associated with the defect priorities associated with the defect data.

8. The method of claim 7, wherein the performance weighted metric comprises aggregating a plurality of weighted detected defects, wherein each defect in the plurality of weighted detected defects is associated with a respective defect priority and is weighted with a respective priority weight.

9. The method of claim 1, wherein determining the defect range difference percentage comprises: A percentage difference between the weighted performance metric and historical weighted performance metrics associated with a plurality of defect detection resources is calculated.

10. The method of claim 9, wherein the historical performance weighted metric comprises a number of detected defects associated with the defect detection resource.

11. The method of claim 9, wherein the historical performance weighted metric comprises a number of detected defects associated with a second defect detection resource, wherein the second defect detection resource is associated with a defect priority associated with the defect data.

12. The method of claim 9, wherein the historical performance weighted metric comprises a plurality of detected defects associated with a second defect detection resource, wherein the second defect detection resource is associated with a second defect priority level associated with the defect data.

13. The method of claim 12, wherein an impact factor is applied to the second defect priority level associated with the defect data when determining the defect coverage delta percentage.

14. The method of claim 1, wherein determining a defect detection resource score comprises: A test case percentage associated with the defect detection resource is determined, wherein the test case percentage indicates a percentage of test cases executed by the defect detection resource relative to a plurality of defect detection resources.

15. The method of claim 1, wherein determining a defect detection resource score comprises: The defect range difference percentage and the test case percentage are normalized.

16. The method according to claim 15, further comprising: The defect range difference percentage is weighted using a defect weight; as well as The test case percentage is weighted using a test case weight.

17. The method of claim 16, wherein the defect weight and the test case weight are equal to one when aggregated.

18. The method according to claim 16, further comprising: At least one of the defect weight and the test case weight is adjusted.

19. A system comprising: at least one processing device including a processor coupled to a memory; The at least one processing device is configured to: acquiring, by a resource assessment system, defect data acquired in response to executing software on a plurality of tester systems; determining, by the resource assessment system, a performance weighted metric associated with a defect detection resource and the defect data; determining a defect coverage delta percentage associated with the defect detection resource and the defect data using the performance weighted metric; as well as A defect detection resource score is determined based on the defect coverage delta percentage.

20. A computer program product comprising a non-transitory processor-readable storage medium having stored therein program code of one or more software programs, wherein the program code, when executed by at least one processing device, causes the at least one processing device to: acquiring, by a resource assessment system, defect data acquired in response to executing software on a plurality of tester systems; determining, by the resource assessment system, a performance weighted metric associated with a defect detection resource and the defect data; determining a defect coverage delta percentage associated with the defect detection resource and the defect data using the performance weighted metric; as well as A defect detection resource score is determined based on the defect coverage delta percentage.

Images