Test case management system for large software system

Abstract

The embodiment of the invention relates to a test case management system for a large software system. Methods, systems, and non-transitory processor-readable storage media for test case management systems are provided herein. An example method includes identifying, by a test case management system, a namespace for including test characteristics required to provide test coverage in a software test lifecycle system. The test case management system creates a test case characteristic matrix data set for the test cases in the test case group. The test case management system performs a similarity analysis between the test characteristics in the namespace and the test cases in the group of test cases using the test case characteristic matrix dataset to identify a subset of the test cases that provide at least a portion of the test coverage. The software test lifecycle system executes a subset of the test cases on the test system to provide at least a portion of the test overlay.

Description

Technical Field

[0001] The field relates generally to optimizing test coverage, and more particularly to optimizing test coverage in information processing systems. Background Art

[0002] Customers demand high-quality software, and adequate test coverage is an integral part of software quality. Therefore, comprehensive and balanced test coverage of the software, while selectively selecting a subset of the large number of available test cases to provide balanced test coverage, is critical to the success of a software project. Summary of the invention

[0003] The illustrative embodiments provide techniques for implementing a test case management system in a storage system. For example, the illustrative embodiments identify a namespace for including test features required to provide test coverage in a software testing lifecycle system by a test case management system. The test case management system creates a test case feature matrix data set for test cases in a test case group. The test case management system uses the test case feature matrix data set to perform a similarity analysis between the test features in the namespace and the test cases in the test case group to identify a subset of test cases that provide at least a portion of test coverage. The software testing lifecycle system executes a subset of test cases on a test system to provide at least a portion of test coverage. Other types of processing devices may be used in other embodiments. These and other illustrative embodiments include, but are not limited to, apparatus, systems, methods, and processor-readable storage media. BRIEF DESCRIPTION OF THE DRAWINGS

[0004] Figure 1 An information processing system that includes a test case management system in an illustrative embodiment is shown.

[0005] Figure 2 A flow diagram of a process for a test case management system in an illustrative embodiment is shown.

[0006] Figure 3 and 4 An example of a processing platform that may be utilized to implement at least a portion of an embodiment of a test case management system is shown. DETAILED DESCRIPTION

[0007] Illustrative embodiments will be described herein with reference to an exemplary computer network and associated computers, servers, network devices or other types of processing devices. However, it should be understood that these and other embodiments are not limited to use with the specific illustrative network and device configurations shown. Therefore, the term "computer network" as used herein is intended to be interpreted broadly so as to cover, for example, any system including multiple networked processing devices.

[0008] Described below are techniques for implementing a test case management system that, among other things, can be used to efficiently use available test cases to provide test coverage by identifying, by the test case management system, a namespace that includes test features required to provide test coverage in a software testing lifecycle system. The test case management system creates a test case feature matrix data set for test cases in a test case group. The test case management system uses the test case feature matrix data set to perform a similarity analysis between the test features in the namespace and the test cases in the test case group to identify a subset of test cases that provide at least a portion of the test coverage. The software testing lifecycle system executes the subset of test cases on a test system to provide at least a portion of the test coverage.

[0009] The productivity of software development processes is constantly increasing due to customer demands for effective functionality and higher and higher quality. Feature coverage of the tested software is critical to the success of software projects. Software projects are constantly expanding, new features that need to be tested are constantly being developed, and functionality is added to previous software versions. Engineering teams develop new test cases to test new functionality and rerun regression test cases. This means that new test cases that provide additional feature coverage are often needed during the software testing life cycle (STLC), where STLC is the series of activities that occur during software testing.

[0010] The result of the new and regression test cases is the Definition of Done (DoD) criteria for integrating the feature code into the integration code branch. Even after completing this integration step, a very large number of system and integration tests are required to ensure quality for complex software systems, especially large storage systems. Even when making code changes to one of the subcomponents that supports a new feature, there may be dozens of new test cases and hundreds of regression test cases that need to be executed to verify the code changes. Finally, the new test cases are automated and added to the regression test group. As the number of test cases increases, the duration required to complete the execution and the consumption of other resources (i.e., machines, expensive destination hardware, storage systems, etc.) also increase.

[0011] When new features are introduced, typically new test cases are developed to test the new features. Over time, as the total number of test cases increases, the likelihood of duplication of testing effort within a large library of test cases also increases. It would be desirable to identify similar existing test cases that test the new features. Identified similar test cases can be enhanced to test the new features, used as a basis for creating new test cases, and / or similar test cases can be merged together. For example, existing test cases can identify existing configurations, steps, and / or functions that aid in test design and / or automation.

[0012] Manually reviewing a large library of test cases to identify a subset of these similar test cases that provide test feature coverage for a large number of test features often takes too much time and may not even be comprehensive enough to provide the necessary feature coverage.

[0013] Conventional techniques for identifying test cases that provide similar test feature coverage do not provide an automated process for outputting test cases that provide feature coverage from a very large number of test cases. Conventional techniques do not provide a customizable process that is also scalable. Conventional techniques do not provide a system that allows a user to adjust the weights of test features to increase and / or decrease the importance of specific features. Conventional techniques do not allow a user to specify how many similar test cases are output as a result. Conventional techniques do not provide a system for analyzing the relevance of a test case relative to other test cases to reduce overlapping labor, and reduce duplication of testing labor. Conventional techniques do not provide a reference to global feature coverage for a group of test cases.

[0014] Differently, in at least some implementations of the current technology described herein, feature coverage is optimized by efficiently using available test cases by identifying, by a test case management system, a namespace that includes test features required to provide test coverage in a software testing lifecycle system. The test case management system creates a test case feature matrix data set for test cases in a test case group. The test case management system uses the test case feature matrix data set to perform a similarity analysis between the test features in the namespace and the test cases in the test case group to identify a subset of test cases that provide at least a portion of test coverage. The software testing lifecycle system executes the subset of test cases on a test system to provide at least a portion of test coverage.

[0015] Therefore, an object of the current technology is to provide a method and system for providing a test case management system that delivers an automated process that outputs test cases that provide test coverage from a very large number of test cases. Another object is to provide a scalable and customizable process. Another object is to provide a system that allows a user to adjust the weight of a test feature to increase and / or decrease the importance of a specific feature. Another object is to allow a user to specify how many similar test cases should be output as a result. Another object is to provide a system that analyzes the relevance of a test case relative to other test cases to reduce overlapping labor and reduce duplication of testing labor. Another object is to provide a reference to global feature coverage of a group of test cases. Yet another object is to provide a system in which the scalability of the number of features in a namespace and the number of test cases in a test group reduces interference from other factors and promotes the guidance of specific test features as opposed to filtering the entire pool of test cases.

[0016] In at least some implementations of the current technology described herein, use of a test case management system may provide one or more of the following advantages: providing an automated process for outputting test cases that provide feature coverage from a very large number of test cases, providing a scalable and customizable process, providing a system that allows a user to adjust the weights of test features to increase and / or decrease the importance of a particular feature, allowing a user to specify how many similar test cases should be output as a result, providing a system that analyzes the relevance of a test case relative to other test cases to reduce overlapping labor and reduce duplication of testing labor, providing a reference to global feature coverage for a group of test cases. And providing a system in which the scalability of the number of features in a namespace and the number of test cases in a test group reduces interference from other factors and facilitates targeting of specific test features as opposed to filtering the entire pool of test cases.

[0017] In contrast to conventional techniques, in at least some implementations of the current techniques described herein, feature coverage is optimized by efficiently using available test cases by identifying, by a test case management system, namespaces for including test features required to provide test coverage in a software testing lifecycle system. The test case management system creates a test case feature matrix data set for test cases in a test case group. The test case management system uses the test case feature matrix data set to perform a similarity analysis between the test features in the namespace and the test cases in the test case group to identify a subset of test cases that provide at least a portion of test coverage. The software testing lifecycle system executes the subset of test cases on a test system to provide at least a portion of test coverage.

[0018] In an example embodiment of the present technology, a test case management system receives a request for identifying a subset of test cases that provide test coverage in a software testing lifecycle system.

[0019] In an example embodiment of the present technology, a test case management system identifies a plurality of necessary test characteristics required to provide test coverage in a software testing lifecycle system.

[0020] In an example embodiment of the present technology, the test case management system identifies at least one of: a new namespace and an existing namespace.

[0021] In an example embodiment of the current technology, a test case management system identifies a feature pool that includes multiple test features tested by multiple test cases, identifies test features required to provide test coverage from the feature pool, and adds the test features required to provide test coverage from the feature pool to a namespace.

[0022] In an example embodiment of the present technology, the test case management system initializes a weight for at least one of the test characteristics in a namespace.

[0023] In an example embodiment of the present technology, the test case management system adjusts at least one of the initialized weights in the namespace.

[0024] In an example embodiment of the current technology, the test case management system performs at least one of the following: increasing the weight of at least one of the initialized weights of test characteristics that are more relevant to providing test coverage, and decreasing the weight of at least one of the initialized weights of test characteristics that are less relevant to providing test coverage.

[0025] In an example embodiment of the current technology, the test case management system sets a weight for at least one of the test characteristics in a namespace as a default weight.

[0026] In an example embodiment of the present technology, a test case management system creates a feature mapping library for test cases in a test case group, wherein the feature mapping library identifies feature coverage provided by each of the test cases.

[0027] In an example embodiment of the present technology, a test case management system generates a test case property matrix data set using a property mapping library.

[0028] In an example embodiment of the present technology, a test case management system assigns a binary representation for each feature to each test case, wherein the binary representation indicates whether each test case provides feature coverage for each feature in a namespace.

[0029] In an example embodiment of the present technology, the test case management system assigns a random number of test cases from among a plurality of test cases to a test case group.

[0030] In an example embodiment of the present technology, the test case management system calculates the number of elements in each row of a test case property matrix data set, wherein the number of elements indicates a property represented by a corresponding test case.

[0031] In an example embodiment of the present technology, the test case management system determines that the number of elements in each row of the test case property matrix data does not meet a threshold, and determines that insufficient property coverage exists in the test cases in the test case group.

[0032] In an example embodiment of the present technology, the test case management system determines that the number of elements in each row of the test case feature matrix data is equal to zero, determines that feature coverage does not exist in the test cases in the test case group, and performs at least one of adding an additional test case to the test case group and replacing at least one of the test cases in the test case group with another at least one test case.

[0033] In an example embodiment of the present technology, a test case management system identifies a number of test cases that should be in a subset of test cases that provide at least a portion of test coverage.

[0034] In an example embodiment of the current technology, the test case management system sorts the results of the similarity analysis to identify the most relevant test cases in the test case group, selects the top most relevant test cases as a subset of test cases, and outputs the subset of test cases.

[0035] In an example embodiment of the present technology, the test case management system identifies an index associated with each of the top most relevant test cases and maps the index in the test case group to the corresponding test case to identify each test case in the subset of test cases.

[0036] In an example embodiment of the present technology, a test case management system provides a subset of test cases to a software testing lifecycle system for execution of the subset of test cases.

[0037] Figure 1 A computer network (also referred to herein as an information processing system) 100 configured in accordance with an illustrative embodiment is shown. Computer network 100 includes software testing lifecycle system 101, test case management system 105, and test systems 102-N. Software testing lifecycle system 101, test case management system 105, and test systems 102-N are coupled to network 104, where network 104 is assumed in this embodiment to represent a sub-network or other associated portion of a larger computer network 100. Therefore, elements 100 and 104 are both referred to herein as examples of "networks," but in the context of FIG. Figure 1The latter is assumed to be an integral part of the former in the context of the embodiments. Also coupled to the network 104 is a test case management system 105 which may reside on a storage system. Such a storage system may include any of a variety of storage of different types, including network-attached storage (NAS), storage area network (SAN), direct-attached storage (DAS), and distributed DAS, as well as combinations of these and other storage types, including software-defined storage.

[0038] Each of the test systems 102-N may include, for example, a server and / or portions of one or more server systems, as well as devices such as mobile phones, laptop computers, tablet computers, desktop computers, or other types of computing devices. Such devices are examples of what are more generally referred to herein as "processing devices." Some of these processing devices are also often referred to herein as "computers."

[0039] In some embodiments, the test system 102-N includes respective computers associated with a particular company, organization, or other enterprise. In addition, at least a portion of the computer network 100 may also be collectively referred to herein as comprising an "enterprise network." As will be appreciated by those skilled in the art, many other operational scenarios involving various different types and arrangements of processing devices and networks are possible.

[0040] It should further be understood that the term "user" in this context and elsewhere herein is intended to be broadly interpreted to cover, for example, humans, hardware, software or firmware entities, and various combinations of such entities.

[0041] The network 104 is assumed to comprise a portion of a global computer network such as the Internet, although other types of networks may be part of the computer network 100, including a wide area network (WAN), a local area network (LAN), a satellite network, a telephone or wired network, a cellular network, a wireless network such as a Wi-Fi or WiMAX network, or various portions or combinations of these and other types of networks. Thus, in some embodiments, the computer network 100 comprises a combination of multiple different types of networks, each including processing devices configured to communicate using the Internet Protocol (IP) or other associated communication protocols.

[0042] Also associated with the test case management system 105 are one or more input-output devices, which illustratively include any combination of keyboards, displays, or other types of input-output devices. For example, such input-output devices can be used to support one or more user interfaces to the test case management system 105, as well as to support communication between the test case management system 105 and other associated systems and devices not explicitly shown. For example, a panel can be provided for a user to view the execution progress of the test case management system 105. One or more input-output devices can also be associated with any of the test systems 102-N.

[0043] Additionally, assuming Figure 1 The test case management system 105 in the embodiment is implemented using at least one processing device. Each such processing device generally includes at least one processor and associated memory, and implements one or more functional modules for controlling certain features of the test case management system 105.

[0044] More specifically, the test case management system 105 in this embodiment may include a processor coupled to a memory and a network interface.

[0045] The processor illustratively includes a microprocessor, a microcontroller, an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other type of processing circuit, as well as portions or combinations of such circuit elements.

[0046] The memory illustratively includes any combination of random access memory (RAM), read-only memory (ROM), or other types of memory. The memory and other memories disclosed herein may be considered as examples of what is more generally referred to as a "processor-readable storage medium" that stores executable computer program code or other types of software programs.

[0047] One or more embodiments include an article of manufacture, such as a computer-readable storage medium. Examples of articles of manufacture include, but are not limited to, storage devices such as storage disks, storage arrays, or integrated circuits including memory, as well as a wide variety of other types of computer program products. The term "article of manufacture" as used herein should be understood to exclude transient propagating signals. These and other references to "disk" herein are intended to refer generally to storage devices, including solid state disks (SSDs), and therefore should not be considered to be limited in any way to rotating magnetic media.

[0048] The network interface allows the test case management system 105 to communicate with the software testing lifecycle system 101 , and the test systems 102 -N via the network 104 , and illustratively includes one or more conventional transceivers.

[0049] The test case management system 105 may be implemented at least in part in the form of software stored in a memory and executed by a processor, and may reside in any processing device. The test case management system 105 may be a stand-alone plug-in that may be included within a processing device.

[0050] It should be understood that Figure 1 The specific set of elements of the software testing lifecycle system 101 involving the computer network 100 and the test system 102-N shown in the for test case management system 105 is presented only as an illustrative example, and additional or alternative elements may be used in other embodiments. Therefore, another embodiment includes additional or alternative systems, devices and other network entities, and different arrangements of modules and other components. For example, in at least one embodiment, one or more of the test case management systems 105 can be on the same processing platform and / or on a portion of the same processing platform.

[0051] References such as Figure 2 The flowchart of FIG. 1 describes an exemplary process of the test case management system 105 in the computer network 100 in more detail.

[0052] Figure 2 is a flow chart of a process performed in an illustrative embodiment for test case management system 105. It should be understood that this particular method is only one example, and that additional or alternative methods may be performed in other embodiments.

[0053] In an example embodiment, the test case management system 105 receives a request to identify a subset of test cases that provide test coverage in the software testing lifecycle system 101. For example, a new feature is added to a product, and test coverage for the new feature needs to be added to multiple test cases. In an example embodiment, the test case management system 105 identifies multiple necessary test features required to provide test coverage in the software testing lifecycle system 101. The test case management system 105 identifies test cases that provide the required test coverage, or identifies test cases that are similar enough that those test cases can be modified to provide the required test coverage, or used as a basis for creating new test cases that provide the required test coverage. In an example embodiment, the test case management system 105 provides an efficient way to quickly analyze the dependencies of test cases with respect to other test cases to avoid executing redundant test cases that provide overlapping test feature coverage, and thus reduce duplication of work.

[0054] In an example embodiment, the test case management system 105 includes a feature pool of test cases. In an example embodiment, the test case management system 105 identifies a feature pool that includes multiple test features tested by multiple test cases. In an example embodiment, the test case management system 105 identifies the test features required to provide test coverage (i.e., multiple necessary test features required to provide test coverage in the software testing lifecycle system 101) from the feature pool, and adds the test features required to provide test coverage from the feature pool to the namespace. In an example embodiment, test features can be added to and / or removed from the feature pool.

[0055] At 200, the test case management system 105 identifies a namespace for including test features required to provide test coverage in the software testing lifecycle system 101. In an example embodiment, the namespace is a list of test features. In an example embodiment, the features in the namespace are a subset of the features in the feature pool and are used to generate a test case feature matrix data set. In an example embodiment, the test case management system 105 identifies a new namespace or leverages an existing namespace.

[0056] In an example embodiment, a user may identify a namespace to be created. A namespace contains features used in a similarity analysis. In essence, a namespace contains features that a user cares about, for example, what a new test case should cover. Therefore, when the test case management system 105 outputs test cases that provide desired feature coverage, a user may utilize those test cases for testing, or use those test cases as a basis for creating new test cases that provide desired feature coverage. In an example embodiment, there may be a feature pool and / or a namespace and a visual representation of drag-and-drop features, wherein a user may select features from a feature pool and drag them to a namespace. In an example embodiment, a user may select features from a feature pool and add them to a namespace for similarity analysis. In an example embodiment, a namespace may also include adjustable weights associated with at least some of the features in the namespace.

[0057] At 202, the test case management system creates a test case property matrix data set for test cases in a test case group. A test case group is a subset of a plurality of test cases. In an example embodiment, the test cases in the test case group may be from a plurality of product components. In an example embodiment, a random number of test cases from the plurality of test cases may be assigned to the test case group.

[0058] At 204, the test case management system 105 performs a similarity analysis between the test characteristics in the namespace and the test cases in the test case groups using the test case characteristic matrix data set to identify a subset of test cases that provide at least a portion of the test coverage. The similarity analysis module determines which test cases from one or more test groups are most useful and / or most similar to the new test case.

[0059] In an example embodiment, the test case management system 105 creates a feature mapping library for the test cases in the test case group, wherein the feature mapping library identifies the feature coverage provided by each test case in the test case group. In an example embodiment, the test case management system 105 assigns a binary representation for each feature to each test case, wherein the binary representation indicates whether each test case provides feature coverage for each feature in the namespace. For example, if the test case provides feature coverage for a particular feature, the associated value in the library is "true" or "1", and if the test case does not provide feature coverage, the associated value in the library is "false" or "0". In an example embodiment, the default value for all features associated with all test cases is "false" or "0". In an example embodiment, the test case management system 105 generates a feature map for each test case and records it in the feature mapping library together with at least one test case ID and the corresponding feature. For example:

[0060]

[0061]

[0062] In an example embodiment, the feature map also records the index associated with each corresponding test case ID. For example:

[0063] Index of test case group 1 Test case ID 1 Test case 5 2 Test case 8 …… <![CDATA[I1]]> Test Case X

[0064] In an example embodiment, when a new test case is added to the feature mapping library, the user is required to set feature coverage values ​​for all features associated with the newly added test case. In an example embodiment, the user can also modify at least some of the feature coverage values ​​in the feature mapping library. In an example embodiment, the feature coverage status for each test case is recorded in the feature mapping library.

[0065] In an exemplary embodiment, the test case management system 105 generates a test case characteristic matrix data set using a characteristic mapping library based on the test cases in the test group. In an exemplary embodiment, a user specifies which test case (or test cases) to use for similarity analysis.

[0066] In the example embodiment, "N" is the number of features in the namespace to be analyzed, "I" is the number of test cases to be analyzed in the test group, and "f" is a list of feature IDs to be analyzed for the namespace, where f = [f1, f2, ... f N ], and f n is the identity of feature n. Filtering features in the namespace and in the scope of the test group reduces interference from other factors and promotes desired feature coverage among many features tested in a large number of test cases in the test case pool.

[0067] The test case feature matrix data set “D” is D=[d1, d2, …d i ……d I ], where d i is the i-th column in the matrix D, i is the test case ID in the test group, “I” is the number of test cases in the test group, and d i =[d1, d2, ..., d N ] T represents the feature mapping for test case i, where if test case i has test points or test coverage for feature n, then d n is equal to 1. In this example scenario, "T" represents the vector or matrix transpose:

[0068]

[0069]

[0070] In an example embodiment, weights are used to aggregate similarities based on a list of feature IDs for a namespace to be analyzed (as described above, represented by "f"). In an example embodiment, the test case management system 105 initializes a weight for at least one of the test features in the namespace. For example, for a given namespace, the weight may be W = [w1, w2, ... w N ] indicates that w i Corresponding to f i The weights of which f = [f1, f2, ... f N ], and f n is the identity of feature n. Feature = f and weight = W and both of them ∈ R N×1. In an example embodiment, the weights for one or more of the test characteristics in the namespace may be set to a default weight, such as "1". In another example embodiment, if the user wishes to filter out results without affecting a particular characteristic (i.e., one or more characteristics that have been set to "0"), the weights may be set to "0" for one or more characteristics. In an example embodiment, the user may modify, or adjust, one or more of the initialized weights in the namespace. For example, the user may increase the weights of one or more of the initialized weights for test characteristics that are more relevant to providing test coverage, and / or decrease the weights of one or more of the initialized weights for test characteristics that are less relevant to providing test coverage. For example, if the user wishes to identify test cases that provide feature coverage for the characteristics of f2-"late dedupe", f6-"NDU", f8-"VLB defrag", and f9 "node reboot fault", the user increases the weights of W2, W6, W8, and W9. If the user wishes to filter out results without affecting those characteristics (characteristics that have been reset to zero), the user may also adjust the weight values ​​of other characteristics to, for example, zero.

[0071] In an example embodiment, the normalized weights are given by Indicates that and

[0072] where W∈R N×1

[0073] In this example scenario, W∈R N×1 Represent the array W as an N-dimensional real space transverse vector. The length of W is N, and R represents a set of real numbers. For example, [w1, w2, w3] ∈ R N×1 .

[0074] In an example embodiment, S represents the aggregated similarity values ​​for the test cases in the test group, where S=[s1, s2, ..., s I ], and

[0075]

[0076] In this example scenario, the larger the value of S1 is, the more similarities are present in the test cases.

[0077] In an example embodiment, similarity analysis (i.e., the feature coverage results of the test group using the features in the namespace) can be represented by the sum of the elements in each row of the test case feature matrix data set "D". In other words, the test case management system 105 calculates the number of elements in each row of the test case feature matrix data set, wherein the number of elements indicates the feature representation of the corresponding test case. The feature coverage of the test case in the test group can be determined by the sum of the elements in each row of the test case feature matrix data set "D". The nth row of D represents the coverage for feature n. Therefore, if the sum is equal to zero, this means that the coverage of feature n is lost. Similarly, if the sum is a lower number, this may indicate that although there are some feature coverages, it may not be enough to thoroughly test feature n, and there may be other test cases that need to be added to the test group to increase the result of the similarity analysis.

[0078] In an exemplary embodiment, the test case management system 105 determines that the number of elements in each row of the test case property matrix data does not meet a threshold value. This means that the test cases in the test group do not provide sufficient property coverage. In this example scenario, the user may have to add, remove, and / or replace test cases in the test group so that more test cases in the test group provide property coverage for the necessary properties in the namespace.

[0079] In an example embodiment, the test case management system 105 determines that the number of elements in each row of the test case property matrix data is equal to zero. This means that there is no property coverage in the test cases in the test case group. In this example scenario, the user can add and / or replace additional test cases in the test case group.

[0080] When the test case management system 105 outputs test cases that provide feature coverage, the user can use those test cases to test the necessary features, or can enhance those existing tests to provide the required feature coverage. If the test case management system 105 determines that there are no test cases that provide the necessary feature coverage, the user may have to create at least one new test case from one or more of the existing test cases. In summary, the test case management system 105 reduces duplication of work by identifying these baseline test cases that can be used as a basis for new test cases.

[0081] In an example embodiment, a user can adjust the test cases in a test group by adding or replacing test cases in the test group. For example, if there are not enough test cases to provide feature coverage for feature "n", the user can add d n Test cases with values ​​equal to 1 (meaning those test cases provide feature coverage for feature "n") are used to enhance / improve the results of the similarity analysis.

[0082] In an exemplary embodiment, the test case management system 105 identifies the number of test cases that should be in the subset of test cases that provide at least a portion of test coverage. For example, as a result of the similarity analysis, a user can specify how many test cases with maximum similarity should be output. In an exemplary embodiment, "M" is a variable integer representing the number of test cases that should be output. The user can also trigger the similarity analysis to be performed on a test group using features in a namespace. For example, if the user selects "M" to be equal to 3, the output of the similarity analysis is three test cases with the greatest similarity to the features in the namespace. By examining the feature mapping (d) of these three test cases, the user can determine whether it is necessary to create a new test case, or whether these three test cases can be enhanced to provide the necessary feature coverage instead of creating a new test case.

[0083] In an example embodiment, the test case management system 105 sorts the results of the similarity analysis to identify the most relevant test cases in the test case group, and selects the number of the most relevant test cases ranked first as the subset of test cases. For example, if "M" is determined to be "3", the test case management system 105 sorts the results of the similarity analysis to determine the top 3 test cases of the similarity analysis, and outputs the subset of test cases.

[0084] In an exemplary embodiment, the test case management system 105 identifies an index associated with each of the top most relevant test cases, and maps the index in the test case group to the corresponding test case to identify each test case in the subset of test cases. In other words, the test case management system 105 sorts the results of the similarity analysis to find the top "M" elements, and records their indexes. The test case management system 105 then outputs the corresponding test case according to the index. In an exemplary embodiment, the objective function ObjectFunc is used to find the index of the test case with the top "M" similarity value:

[0085]

[0086] In an example embodiment, the test case management system 105 provides the subset of test cases to the software testing lifecycle system 101 for executing the subset of test cases. At 206, the software testing lifecycle system 101 executes the subset of test cases on the test system 102-N to provide at least a portion of test coverage.

[0087] Therefore, combined with Figure 2The specific processing operations and other functions described in the flowcharts are presented only by way of illustrative examples and should not be interpreted as limiting the scope of the present disclosure in any way. For example, the order of the processing steps can be changed in other embodiments, or certain steps can be performed simultaneously with each other rather than in series.

[0088] The above illustrative embodiments provide significant advantages over traditional methods. For example, some embodiments are configured to significantly improve the selection of test cases to provide desired feature coverage. These and other embodiments can effectively and efficiently improve test coverage and test time relative to traditional methods. For example, the embodiments disclosed herein provide an automated process for outputting test cases that provide feature coverage from a very large number of test cases. The embodiments disclosed herein provide a scalable and customizable process. The embodiments disclosed herein provide a system that allows users to adjust the weight of test features to increase and / or reduce the importance of specific features. The embodiments disclosed herein allow users to specify how many similar test cases should be output as a result. The embodiments disclosed herein provide a system that provides analysis of the relevance of test cases relative to other test cases to reduce overlapping labor and reduce duplication of test labor. The embodiments disclosed herein provide a reference to the global feature coverage of a group of test cases. The embodiments disclosed herein provide a system in which the scalability of the number of features in the namespace and the number of test cases in the test group reduces the interference of other factors and promotes the guidance of specific test features in contrast to filtering the entire pool of test cases.

[0089] It should be understood that the specific advantages described above and elsewhere herein are associated with specific illustrative embodiments and need not be present in other embodiments. Moreover, the specific types of information processing system features and functions as illustrated in the accompanying drawings and described above are merely exemplary, and in other embodiments, many alternative arrangements may be used.

[0090] As previously mentioned, at least a portion of the information handling system 100 can be implemented using one or more processing platforms. Given such a processing platform includes at least one processing device, which includes a processor coupled to a memory. In some embodiments, the processor and memory include corresponding processors and memory elements of a virtual machine or container provided using one or more underlying physical machines. The term "processing device" as used herein is intended to be broadly interpreted to cover a wide variety of different arrangements of virtual instances of physical processors, memory and other device components and such components. For example, in some embodiments, a "processing device" may include one or more virtual processors or execute across one or more virtual processors. Therefore, a processing device may be physical or virtual and may execute across one or more physical processors or virtual processors. It should also be noted that a given virtual device may be mapped to a portion of a physical device.

[0091] Some illustrative embodiments of a processing platform for implementing at least a portion of an information handling system include a cloud infrastructure that includes virtual machines implemented using a hypervisor running on a physical infrastructure. The cloud infrastructure further includes a set of applications running on corresponding ones of the virtual machines under the control of the hypervisor. It is possible to use multiple hypervisors, each hypervisor using at least one underlying physical machine to provide a set of virtual machines. Different sets of virtual machines provided by one or more hypervisors can be utilized to configure multiple instances of various components of the system.

[0092] These and other types of cloud infrastructures may be used to provide what is also referred to herein as a multi-tenant environment. One or more system components, or portions thereof, are illustratively implemented for use by tenants of such a multi-tenant environment.

[0093] As mentioned previously, the cloud infrastructure as disclosed herein may include a cloud-based system. A virtual machine provided in such a system may be used to implement at least a portion of the computer system in the illustrative embodiment.

[0094] In some embodiments, the cloud infrastructure additionally or alternatively includes a plurality of containers implemented using a container host device. For example, as described in detail herein, a given container of the cloud infrastructure illustratively includes a Docker container or other type of Linux container (LXC). The container runs on a virtual machine in a multi-tenant environment, but other arrangements are also possible. Containers are utilized to implement various different types of functions within the information processing system 100. For example, containers can be used to implement corresponding processing devices that provide computing and / or storage services for a cloud-based system. Similarly, containers can be used in conjunction with other virtualization infrastructures (e.g., virtual machines implemented using hypervisors, etc.).

[0095] Now refer to Figure 3 and 4 Illustrative embodiments of the processing platform are described in more detail. Although described in the context of information handling system 100, in other embodiments, these platforms may also be used to implement at least a portion of other information handling systems.

[0096] Figure 3 An example processing platform is shown that includes a cloud infrastructure 300. The cloud infrastructure 300 includes a combination of physical and virtual processing resources for implementing at least a portion of the information processing system 100. The cloud infrastructure 300 includes a plurality of virtual machines (VMs) and / or container sets 302-1, 302-2, ... 302-L implemented using a virtualization infrastructure 304. The virtualization infrastructure 304 runs on a physical infrastructure 305 and illustratively includes one or more hypervisors and / or operating system-level virtualization infrastructure. The operating system-level virtualization infrastructure illustratively includes a kernel control group of a Linux operating system or other type of operating system.

[0097] Cloud infrastructure 300 also includes a group 310-1, 310-2, ... 310-L of applications running on a corresponding set of VMs / containers 302-1, 302-2, ... 302-L under the control of virtualization infrastructure 304. VM / container set 302 includes a corresponding VM, a corresponding set of one or more containers, or a corresponding set of one or more containers running in a VM. Figure 3 In some implementations of the embodiments, the VM / container set 302 includes respective VMs implemented using a virtualization infrastructure 304 including at least one hypervisor.

[0098] A hypervisor platform may be used to implement a hypervisor within the virtualization infrastructure 304, wherein the hypervisor platform has an associated virtual infrastructure management system. The underlying physical machines include one or more distributed processing platforms, which include one or more storage systems.

[0099] exist Figure 3 In other implementations of the embodiment, VM / container set 302 includes corresponding containers implemented using virtualization infrastructure 304, which provides operating system level virtualization functions, such as support for Docker containers running on bare metal hosts, or Docker containers running on VMs. Containers are illustratively implemented using various kernel control groups of the operating system.

[0100] As can be seen from the above, one or more of the processing modules or other components of the information processing system 100 can each run on a computer, server, storage device or other processing platform element. Given such elements are considered examples of what is more generally referred to as a "processing device" herein. Figure 3 The cloud infrastructure 300 shown may represent at least a portion of a processing platform. Another example of such a processing platform is Figure 4 A processing platform 400 is shown.

[0101] The processing platform 400 in this embodiment comprises a portion of the information processing system 100 and includes a plurality of processing devices, denoted as 402 - 1 , 402 - 2 , 402 - 3 . . . 402 -K, which communicate with each other via a network 404 .

[0102] Network 404 includes any type of network, including by way of example a global computer network such as the Internet, a WAN, a LAN, a satellite network, a telephone or wired network, a cellular network, a wireless network such as a Wi-Fi or WiMAX network, or various portions or combinations of these and other types of networks.

[0103] The processing device 402 - 1 in the processing platform 400 includes a processor 410 coupled to a memory 412 .

[0104] Processor 410 includes a microprocessor, a microcontroller, an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other type of processing circuit, as well as portions or combinations of such circuit elements.

[0105] The memory 412 includes any combination of random access memory (RAM), read-only memory (ROM), or other types of memory. The memory 412 and other memories disclosed herein should be viewed as illustrative examples of what is more generally referred to as a "processor-readable storage medium" that stores executable program code of one or more software programs.

[0106] Articles of manufacture including such processor-readable storage media are considered to be illustrative embodiments. Given such articles of manufacture include any computer program product such as a storage array, a storage disk, or an integrated circuit containing RAM, ROM or other electronic memory, as well as a wide variety of other types of computer program products. The term "article of manufacture" as used herein should be understood to exclude transient propagation signals. Many other types of computer program products including processor-readable storage media may be used.

[0107] Also included in the processing device 402-1 is a network interface circuit 414 for interfacing the processing device with the network 404 and other system components, and may include a conventional transceiver.

[0108] It is assumed that the other processing devices 402 of the processing platform 400 are configured in a similar manner as the processing device 402 - 1 shown in the figure.

[0109] Furthermore, the specific processing platform 400 shown in the figure is presented by way of example only, and the information processing system 100 may include any combination of additional or alternative processing platforms, as well as multiple different processing platforms, each such platform including one or more computers, servers, storage devices or other processing devices.

[0110] For example, instead of or in addition to a virtualization infrastructure including virtual machines, other processing platforms for implementing the illustrative embodiments may include a different type of virtualization infrastructure. Such a virtualization infrastructure illustratively includes a container-based virtualization infrastructure configured to provide a Docker container or other type of LXC.

[0111] As another example, in some embodiments, portions of a given processing platform may include a converged infrastructure.

[0112] It should therefore be understood that in other embodiments, different arrangements of additional or alternative elements may be used.At least a subset of these elements may be implemented together on a common processing platform, or each such element may be implemented on a separate processing platform.

[0113] Furthermore, many other arrangements of computers, servers, storage products or devices, or other components are possible in information handling system 100. Such components may communicate with other elements of information handling system 100 via any type of network or other communications medium.

[0114] For example, specific types of storage products that may be used to implement a given storage system of the distributed processing system in the illustrative embodiments include all-flash and hybrid flash storage arrays, scaled-down all-flash storage arrays, scaled-down NAS clusters, or other types of storage arrays. Combinations of multiples of these and other storage products may also be used to implement a given storage system in the illustrative embodiments.

[0115] It should be emphasized again that the above-described embodiments are presented for illustrative purposes only. Many variations and other alternative embodiments may be used. Furthermore, in other embodiments, the specific configurations of the system and device elements illustratively shown in the accompanying drawings and the associated processing operations may be changed. Thus, for example, the specific types of processing devices, modules, systems and resources deployed in a given embodiment and their corresponding configurations may be changed. Furthermore, the various assumptions made above in describing the illustrative embodiments should also be considered exemplary rather than requirements or limitations. Many other alternative embodiments within the scope of the appended claims will be apparent to those skilled in the art.

Claims

1. A method, include: A namespace is identified by a test case management system, the namespace being used to include test features required to provide test coverage in a software testing lifecycle system; The test case management system creates a test case characteristic matrix data set for the test cases in the test case group; performing, by the test case management system, a similarity analysis between the test characteristics in the namespace and the test cases in the test case group using the test case characteristic matrix data set to identify a subset of the test cases that provide at least a portion of the test coverage; as well as The subset of the test cases is executed on a test system by the software testing lifecycle system to provide the at least a portion of the test coverage, wherein the method is implemented by at least one processing device including a processor coupled to a memory.

2. The method according to claim 1, wherein the namespace is identified include: A plurality of necessary test characteristics required to provide the test coverage in the software testing lifecycle system are identified.

3. The method according to claim 1, wherein the namespace is identified include: Identify at least one of the following: a new namespace and an existing namespace.

4. The method according to claim 1, wherein the namespace is identified include: identifying a feature pool, the feature pool comprising a plurality of test features tested by a plurality of test cases; Identifying the test features required to provide the test coverage from the feature pool; as well as The test features required to provide test coverage are added from the feature pool to the namespace.

5. The method according to claim 1, wherein the namespace is identified include: A weight is initialized for at least one of the test properties in the namespace.

6. The method according to claim 5, further comprising: include: At least one of the initialized weights in the namespace is adjusted.

7. The method of claim 6, wherein adjusting the at least one of the initialized weights in the namespace comprises at least one of the following: increasing the weight of the at least one of the initialized weights for a test characteristic that is more relevant to providing the test coverage; The at least one of the initialized weights for a test characteristic that is less relevant to providing the test coverage is weighted down.

8. The method according to claim 5, wherein the weight for the at least one of the test characteristics in the namespace is initialized include: The weight for the at least one of the test properties in the namespace is set to a default weight.

9. The method according to claim 1, wherein the test case characteristic matrix data set is created for the test cases in the test case group. include: A feature mapping library is created for the test cases in the test case group, wherein the feature mapping library identifies feature coverage provided by each of the test cases.

10. The method according to claim 9, further comprising: include: The test case feature matrix data set is generated using the feature mapping library.

11. The method according to claim 9, wherein the feature mapping library for the test cases in the test case group is created include: A binary representation for each feature is assigned to each test case, wherein the binary representation indicates whether the each test case provides the feature coverage for the each feature in the namespace.

12. The method according to claim 9, wherein the feature mapping library for the test cases in the test case group is created include: A random number of the test cases from a plurality of test cases are assigned to the test case group.

13. The method according to claim 1, wherein the similarity analysis between the test characteristics in the namespace and the test cases in the test case group is performed include: The number of elements in each row of the test case characteristic matrix data set is calculated, wherein the number of elements indicates the characteristic representation of the corresponding test case.

14. The method according to claim 13, further comprising: include: Determining that the number of elements in each row of the test case characteristic matrix data does not meet a threshold; as well as It is determined that insufficient feature coverage exists in the test cases in the test case group.

15. The method according to claim 13, further comprising: include: Determining that the number of the elements in each row of the test case characteristic matrix data is equal to zero; Determining that there is no feature coverage in the test cases in the test case group; as well as At least one of: adding an additional test case to the test case group, and replacing at least one of the test cases in the test case group with at least one other test case is performed.

16. The method according to claim 1, wherein the similarity analysis between the test characteristics in the namespace and the test cases in the test case group is performed include: A number of test cases that should be in the subset of the test cases that provide the at least a portion of the test coverage is identified.

17. The method according to claim 16, further comprising: include: sorting the results of the similarity analysis to identify the most relevant test cases in the test case group, and selecting the number of the most relevant test cases ranked top as the subset of the test cases; as well as The subset of the test cases is output.

18. The method of claim 17, wherein the subset of the test cases is output include: identifying an index associated with each of the most relevant test cases in the top ranked most relevant test cases; as well as The indexes in the group of test cases are mapped to corresponding test cases to identify each test case in the subset of the test cases.

19. A system, include: at least one processing device comprising a processor coupled to a memory; The at least one processing device is configured to: A namespace is identified by a test case management system, the namespace being used to include test features required to provide test coverage in a software testing lifecycle system; The test case management system creates a test case characteristic matrix data set for the test cases in the test case group; performing, by the test case management system, a similarity analysis between the test characteristics in the namespace and the test cases in the test case group using the test case characteristic matrix data set to identify a subset of the test cases that provide at least a portion of the test coverage; as well as The subset of the test cases is executed on a test system by the software testing lifecycle system to provide the at least a portion of the test coverage.

20. A computer program product comprising a non-transitory processor-readable storage medium having program code of one or more software programs stored therein, wherein the program code, when executed by at least one processing device, causes the at least one processing device to: A namespace is identified by a test case management system, the namespace being used to include test features required to provide test coverage in a software testing lifecycle system; The test case management system creates a test case characteristic matrix data set for the test cases in the test case group; performing, by the test case management system, a similarity analysis between the test characteristics in the namespace and the test cases in the test case group using the test case characteristic matrix data set to identify a subset of the test cases that provide at least a portion of the test coverage; as well as The subset of the test cases is executed on a test system by the software testing lifecycle system to provide the at least a portion of the test coverage.

Images