Apparatus, method, and computer program product for hierarchical clustering of test cases

Abstract

An apparatus includes a processing device configured to obtain a test log generated by executing a test case on an information technology asset of an information technology infrastructure, parse the test log to generate a set of log event templates for test actions performed during execution of the test case on the information technology asset of the information technology infrastructure, generate a vector representation of the test case utilizing the set of generated log event templates, and perform hierarchical clustering of a plurality of test cases utilizing one or more machine learning-based hierarchical clustering algorithms that take as input the generated vector representations of the test cases. The processing device is further configured to generate a test plan for a given information technology asset of the information technology infrastructure based at least in part on hierarchical clustering results.

Description

Technical Field

[0001] This field generally relates to information processing, and more specifically to the management of information processing systems. Background Technology

[0002] The software development process typically involves multiple environments, such as one or more development environments, integration testing environments, staging environments, and production environments. New software code can be created by individual developers or development teams within the corresponding development environment. Integration environments provide a common environment where software code from multiple developers is combined and tested before being provided to staging environments. Staging environments are designed to simulate production environments and can be used for final review and approval of new software code before deployment in production applications. Summary of the Invention

[0003] The illustrative implementation of this disclosure provides a technique for hierarchical clustering of test cases for generating test plans for information technology assets.

[0004] In one embodiment, an apparatus includes at least one processing means, the at least one processing means including a processor coupled to a memory. The at least one processing means is configured to perform the following steps: obtaining a set of test logs generated by executing multiple test cases on one or more of a plurality of information technology assets of an information technology infrastructure; parsing the set of test logs to generate a set of log event templates for test actions performed during the execution of the multiple test cases on the one or more of the plurality of information technology assets of the information technology infrastructure; and generating vector representations of the plurality of test cases using the generated log event templates. The at least one processing means is further configured to perform the following step: performing hierarchical clustering on the plurality of test cases using one or more machine learning-based hierarchical clustering algorithms, the machine learning-based hierarchical clustering algorithms taking the generated vector representations of the plurality of test cases as input. The at least one processing means is further configured to perform the following step: generating one or more test plans for a given information technology asset among the plurality of information technology assets of the information technology infrastructure, based at least in part on the hierarchical clustering results of the plurality of test cases.

[0005] These and other illustrative embodiments include, but are not limited to, methods, devices, networks, systems, and processor-readable storage media. Attached Figure Description

[0006] Figure 1 This is a block diagram of an information processing system configured to generate test cases for test plans of information technology assets, as shown in the illustrative implementation.

[0007] Figure 2 This is a flowchart of an exemplary process for hierarchical clustering of test cases for generating test plans for information technology assets, as described in the illustrative implementation.

[0008] Figure 3 The text vectorization using bag-of-words natural language processing is illustrated in an illustrative implementation.

[0009] Figure 4 Examples of different product types with different associated test log formats are shown in the illustrative implementation.

[0010] Figure 5 A log template dictionary for converting log events into log event templates is shown in an illustrative implementation.

[0011] Figures 6A to 6E An example of processing raw test logs to generate test log vectors in an illustrative implementation is shown.

[0012] Figure 7A and Figure 7B An example of test case hierarchical clustering in an illustrative implementation is shown.

[0013] Figure 8 The process flow for test case analysis and selection in the illustrative implementation is shown.

[0014] Figure 9 and Figure 10 An example of a processing platform, which can be used to implement at least a portion of an information processing system, is shown in an illustrative embodiment. Detailed Implementation

[0015] This document describes illustrative embodiments with reference to exemplary information processing systems and associated computers, servers, storage devices, and other processing apparatuses. However, it should be understood that the embodiments are not limited to use with the specific illustrative system and apparatus configurations shown. Therefore, the term "information processing system" as used herein is intended to be broadly interpreted to encompass, for example, processing systems including cloud computing and storage systems, as well as other types of processing systems including various combinations of physical and virtual processing resources. Thus, an information processing system may include, for example, at least one data center or other type of cloud-based system that includes one or more clouds hosting tenants accessing cloud resources.

[0016] Figure 1An information processing system 100 configured according to an illustrative embodiment is shown. The information processing system 100 is assumed to be built on at least one processing platform and provides functionality for hierarchical clustering of test cases for generating test plans for information technology (IT) assets. The information processing system 100 includes a set of client devices 102-1, 102-2, ..., ... 102-M (collectively referred to as client devices 102) coupled to a network 104. An IT infrastructure 105 is also coupled to the network 104, comprising one or more IT assets 106, a test database 108, and a test plan design system 110. IT assets 106 may include physical and / or virtual computing resources within the IT infrastructure 105. Physical computing resources may include physical hardware such as servers, storage systems, networking equipment, Internet of Things (IoT) devices, and other types of processing and computing devices, including desktop computers, laptop computers, tablet computers, smartphones, etc. Virtual computing resources may include virtual machines (VMs), containers, etc.

[0017] IT assets 106 of IT infrastructure 105 may host applications utilized by corresponding client devices in client device 102, according to a client-server computer program architecture. In some embodiments, the applications include web applications designed to be delivered from assets in IT infrastructure 105 to users (e.g., users of client device 102) via network 104. Various other examples are possible, such as one or more applications being used internally within IT infrastructure 105 and not exposed to client device 102. It should be understood that in some embodiments, some IT assets 106 of IT infrastructure 105 may themselves be considered as applications or (more generally) software or hardware to be tested. For example, IT assets in IT asset 106 as virtual computing resources implemented as software containers may represent software to be tested. As another example, IT assets in IT asset 106 as physical computing resources may represent hardware devices to be tested.

[0018] Test plan design system 110 utilizes various information stored in test database 108 to design test plans for testing IT asset 106, applications, or other software running on IT asset 106. In some embodiments, test plan design system 110 is used for enterprise systems. For example, an enterprise may subscribe to or otherwise utilize test plan design system 110 to generate and run test plans (e.g., on IT asset 106 of IT infrastructure 105, on client devices 102 operated by users of the enterprise, etc.). As used herein, the term "enterprise system" is intended to be interpreted broadly as including any group of systems or other computing devices. For example, IT asset 106 of IT infrastructure 105 may provide a portion of one or more enterprise systems. A given enterprise system may also, or alternatively, include one or more of the client devices 102. In some embodiments, enterprise systems include one or more data centers, cloud infrastructure including one or more clouds, etc. A given enterprise system, such as cloud infrastructure, may host assets associated with multiple enterprises (e.g., two or more different merchants, organizations, or other entities).

[0019] Client device 102 may include, in any combination, physical computing devices such as IoT devices, mobile phones, laptops, tablets, desktop computers, or other types of devices used by enterprise members. Such devices are examples of what is more generally referred to herein as “processing devices.” Some of these processing devices are also generally referred to herein as “computers.” Client device 102 may also, or alternatively may, include virtual computing resources such as VMs, containers, etc.

[0020] In some implementations, client device 102 includes a corresponding computer associated with a specific company, organization, or other enterprise. Therefore, client device 102 can be considered an example of an asset of an enterprise system. Additionally, at least portions of information processing system 100 may also be referred to herein as collectively constituting one or more “enterprises.” Many other operational scenarios involving a wide variety of different types and arrangements of processing nodes are possible, as those skilled in the art will understand.

[0021] Network 104 is assumed to include global computer networks such as the Internet, but other types of networks may be part of network 104, including wide area networks (WAN), local area networks (LAN), satellite networks, telephone or wired networks, cellular networks, wireless networks (such as WiFi or WiMAX networks), or various parts or combinations of these and other types of networks.

[0022] As discussed above, test database 108 is configured to store and record various information used by test planning system 110 when designing test plans for testing IT asset 106, applications, or other software running on IT asset 106. Such information may include, but is not limited to, information about test cases, test objectives, test points, test coverage, test plans, etc. In some embodiments, test database 108 is implemented using one or more storage systems or devices associated with test planning system 110. In some embodiments, one or more of the storage systems used to implement test database 108 include scale-out all-flash content-addressable memory arrays or other types of storage arrays.

[0023] Therefore, the term "storage system" as used herein is intended to be interpreted broadly and should not be considered limited to content-addressable storage systems or flash-based storage systems. A given storage system as a term used extensively herein may include, for example, network-attached storage (NAS), storage area networks (SAN), direct-attached storage (DAS), and distributed DAS, as well as combinations of these and other storage types (including software-defined storage).

[0024] Other specific types of storage products that can be used to implement a storage system in the illustrative embodiments include all-flash and hybrid flash storage arrays, software-defined storage products, cloud storage products, object-based storage products, and scale-out NAS clusters. In the illustrative embodiments, combinations of these and other storage products can also be used to implement a given storage system.

[0025] Despite Figure 1 Not explicitly shown, but one or more input / output devices such as a keyboard, display or other types of input / output devices may be used to support one or more user interfaces to the test planning system 110, as well as to support communication between the test planning system 110 and other related systems and devices not explicitly shown.

[0026] Client device 102 is configured to access or otherwise utilize IT infrastructure 105. In some embodiments, client device 102 is assumed to be associated with a system administrator, IT manager, or other authorized person responsible for managing IT assets 106 of IT infrastructure 105 (e.g., where such management includes performing tests on IT assets 106 or on applications or other software running on IT assets 106). For example, a given client device of client device 102 may be operated by a user to access a graphical user interface (GUI) provided by test plan design system 110 to manage test plans (e.g., create, review, execute, etc.). Test plan design system 110 may be provided as a cloud service accessible by a given client device 102 to allow its users to manage test plans. In some embodiments, IT assets 106 of IT infrastructure 105 are owned or operated by the same enterprise operating test plan design system 110 (e.g., where an enterprise such as a merchant provides support for the assets it operates). In other implementations, the IT assets 106 of IT infrastructure 105 may be owned or operated by one or more entities different from the entity operating the test plan design system 110 (e.g., the first entity supports assets owned by multiple different customers, merchants, etc.). Various other examples are possible.

[0027] In other implementations, the test plan design system 110 may provide support for testing of client device 102 instead of providing support for or supplementing IT assets 106 of IT infrastructure 105. For example, the test plan design system 110 may be operated by a hardware vendor that manufactures and sells computing devices (e.g., desktop computers, laptop computers, tablet computers, smartphones, etc.), and client device 102 represents a computing device sold by that hardware vendor. The test plan design system 110 may also be operated, or alternatively, by a software vendor that manufactures and sells software (e.g., applications) that runs on client device 102. However, the test plan design system 110 does not need to be operated by any single hardware or software vendor. Instead, the test plan design system 110 may be provided as a service to support computing devices or software sold by any number of hardware or software vendors. Client device 102 may subscribe to the test plan design system 110 to receive support for testing of client device 102 or the software running thereon. Various other examples are possible.

[0028] In some implementations, client device 102 may implement a host agent configured to automatically transmit information about the status of client device 102 (e.g., in the form of test logs periodically provided to test database 108 and / or test planning system 110). Such a host agent may also, or alternatively, be configured to automatically receive commands from test planning system 110 to perform remote actions (e.g., running various test cases on client device 102 and / or IT asset 106 of IT infrastructure 105). The host agent may be similarly deployed on IT asset 106 of IT infrastructure 105.

[0029] It should be noted that the term "host agent" as used herein can include automated entities, such as software entities running on a processing device. Therefore, a host agent does not necessarily have to be a human entity.

[0030] Figure 1 The test plan design system 110 in the proposed implementation is assumed to be implemented using at least one processing device. Each such processing device typically includes at least one processor and associated memory, and implements one or more functional modules or logic for controlling certain features of the test plan design system 110. Figure 1 In the implementation scheme, the test plan design system 110 includes test log vectorization logic 112, test case clustering logic 114, and test case selection logic 116. The test log vectorization logic 112 is configured to obtain a set of test logs generated by executing multiple test cases on one or more IT assets 106 of the IT infrastructure 105. The test log vectorization logic 112 is also configured to parse the set of test logs to generate a set of log event templates for the test actions performed during the execution of multiple test cases on one or more IT assets 106 of the IT infrastructure 105. The test log vectorization logic 112 is further configured to generate vector representations of multiple test cases using the generated log event templates. The test case clustering logic 114 is configured to perform hierarchical clustering on the multiple test cases using one or more machine learning-based hierarchical clustering algorithms, which take the generated vector representations of the multiple test cases as input. The test case selection logic 116 is configured to generate one or more test plans for a given IT asset among the IT assets 106 of the IT infrastructure 105, based at least in part on the hierarchical clustering results of the multiple test cases. The test plan design system 110 is also configured to execute one or more test plans on a given IT asset of IT asset 106 of IT infrastructure 105, either directly or via one or more client devices 102.

[0031] It should be understood that Figure 1The specific arrangement of the client device 102, IT infrastructure 105, and test plan design system 110 shown in the implementation scheme is presented by way of example only, and alternative arrangements may be used in other implementation schemes. As discussed above, for example, one or more of the test plan design system 110 (or parts thereof, such as test log vectorization logic 112, test case clustering logic 114, and test case selection logic 116) may be implemented within one or more of the client device 102 and / or IT infrastructure 105 in some implementation schemes.

[0032] At least each of the test log vectorization logic 112, test case clustering logic 114, and test case selection logic 116 can be implemented, at least in part, as software stored in memory and executed by a processor.

[0033] As will be described in further detail below, the test planning design system 110 and other parts of the information processing system 100 may be part of a cloud infrastructure.

[0034] Assuming implementation is carried out using at least one processing platform comprising one or more processing devices (each processing device having a processor coupled to memory). Figure 1 The implementation scheme includes a test planning design system 110 and other components of the information processing system 100. Such processing apparatus may illustratively include a specific arrangement of computing, storage, and network resources.

[0035] While client device 102, IT infrastructure 105, test database 108, and test plan design system 110 or components thereof (e.g., test log vectorization logic 112, test case clustering logic 114, and test case selection logic 116) can be implemented on corresponding different processing platforms, many other arrangements are possible. For example, in some embodiments, at least a portion of test plan design system 110 and one or more of client device 102, IT infrastructure 105, and / or test database 108 are implemented on the same processing platform. Thus, a given client device (e.g., 102-1) can be implemented at least partially within at least one processing platform that implements at least a portion of test plan design system 110.

[0036] As used herein, the term "processing platform" is intended to be interpreted broadly to include, for example, but not limited to, multiple sets of processing devices and associated storage systems configured to communicate over one or more networks. For example, a distributed implementation of information processing system 100 is possible, wherein some components of the system reside in a data center located in a first geographical location, while other components reside in one or more other data centers located in one or more other geographical locations that may be far from the first geographical location. Therefore, in some implementations of information processing system 100, client device 102, IT infrastructure 105, IT assets 106, test database 108, and test planning system 110, or portions or components thereof, may reside in different data centers. Many other distributed implementations are possible. Test planning system 110 may also be implemented in a distributed manner across multiple data centers.

[0037] The following will combine Figure 9 and Figure 10 Additional examples of the processing platform used to implement the test planning design system 110 and other components of the information processing system 100 in the illustrative implementation are described in more detail.

[0038] It should be understood that these and other features of the illustrative implementation are presented by way of example only and should not be construed as restrictive in any way.

[0039] It should be understood that Figure 1 The specific set of elements in the hierarchical clustering of test cases for generating test plans for IT assets shown in the illustration is presented only as an illustrative example, and in other embodiments, additional or alternative elements may be used. Therefore, another embodiment may include additional or alternative systems, devices, and other network entities, as well as different arrangements of modules and other components.

[0040] It should be understood that these and other features of the illustrative implementation are presented by way of example only and should not be construed as restrictive in any way.

[0041] Now refer to Figure 2 The flowchart describes in more detail an exemplary process for hierarchical clustering of test cases used to generate test plans for IT assets. It should be understood that this particular process is merely an example, and additional or alternative processes for hierarchical clustering of test cases for generating test plans for IT assets may be used in other implementations.

[0042] In this implementation, the process includes steps 200 to 208. These steps are assumed to be performed by the test plan design system 110 using test log vectorization logic 112, test case clustering logic 114, and test case selection logic 116. The process begins at step 200: obtaining a set of test logs generated by executing multiple test cases on one or more of a plurality of IT assets (e.g., 106) of an IT infrastructure (e.g., 105). The plurality of IT assets of the IT infrastructure may include at least one of: one or more physical computing resources and one or more virtual computing resources; and software that runs on at least one of the one or more physical computing resources and one or more virtual computing resources. The set of test logs may include at least one of: one or more test case execution logs generated by the plurality of test cases; and one or more system logs generated by combining the execution of multiple test cases on one or more of the plurality of IT assets of the IT infrastructure. A given test log in this set of test logs may specify: one or more test actions taken during the execution of at least one of multiple test cases; the reaction of one or more of multiple IT assets of the IT infrastructure to one or more test plans; and at least one of the hardware and software configurations of one or more of the multiple IT assets of the IT infrastructure.

[0043] Figure 2 The process continues to step 202: parsing the set of test logs to generate a set of log event templates for test actions performed during the execution of multiple test cases on one or more IT assets of the IT infrastructure. For a given test log in the set of test logs, step 202 may include: identifying multiple log events in the given test log; for each of the multiple log events in the given test log, extracting a constant portion and removing one or more variables; and converting the extracted constant portion of each of the multiple log events in the given test log into a given log event template among the multiple log event templates. In step 204, vector representations of multiple test cases are generated using the set of log event templates. Step 204 may include: creating a log event template dictionary, which includes a set of deduplicated log event templates used in the multiple test cases; and translating each of the test logs into a log vector using the log event template dictionary. The log event template dictionary can include specified padding elements, and translating each item in the test log into a log vector using the log event template dictionary can include translating each item in the test log into a log vector of equal dimension by padding the log vector with additional entries of the specified padding elements to a length less than the longest log vector.

[0044] In step 206, one or more machine learning-based hierarchical clustering algorithms are used to perform hierarchical clustering on multiple test cases, wherein the machine learning-based hierarchical clustering algorithm takes the generated vector representations of the multiple test cases as input. Step 206 may include: initializing each of the multiple test cases as a corresponding cluster in a first level of a cluster hierarchy; calculating the connections between each of the multiple clusters in the first level of the cluster hierarchy; and clustering a subset of the multiple test cases in a second level of the cluster hierarchy, at least in part based on the calculated connections. Calculating connections may include calculating a distance metric between the generated vector representations of the multiple test cases. The calculated connections may include at least one of single connections, complete connections, and average connections.

[0045] In step 208, one or more test plans are generated for a given IT asset among multiple IT assets of the IT infrastructure, based at least in part on the hierarchical clustering results of multiple test cases. For a given test plan among the one or more test plans, step 208 may include selecting a deduplicated subset of multiple test cases that provides coverage for a specified set of test points to include in the given test plan. For a given test plan among the one or more test plans, step 208 may also, or alternatively, include selecting a subset of multiple test cases that provides coverage for a specified set of features of the product to be tested by the given test plan to include in the given test plan. For a given test plan among the one or more test plans, step 208 may also, or alternatively, include selecting a sequence of two or more subsets of multiple test cases to include in the given test plan. The selected sequence of two or more subsets of multiple test cases is determined at least in part based on the hierarchical clustering results of multiple test cases. A sequence of two or more subsets of multiple test cases may include: a first subset of test cases to be executed first, the first subset of test cases testing a first subset of features of the product to be tested by a given test plan; and at least a second subset of test cases to be executed after the first subset of test cases has passed, the second subset of test cases testing a second subset of features of the product to be tested by a given test plan.

[0046] The illustrative implementation provides intelligent identification techniques for test cases used to test IT infrastructure and its assets. In some implementations, intelligent identification of test cases leverages hierarchical clustering, at least in part based on analysis of system test logs (also referred to as "test logs"), to optimize or improve test case design. The techniques described herein further provide methods for deduplicating test cases in test plans, identifying test gaps in test plans, and providing intelligent or smart suggestions for dynamic test case selection in test plans. Log-based vectorization can be used to reflect test case design and automated execution of sequential test cases. The test case analysis mechanisms described herein offer various improvements and efficiencies, thereby reducing manpower and supporting larger-scale test case analysis that is impractical for human analysts.

[0047] When testing large, complex products (such as IT assets in IT infrastructure), system testing may involve scheduling testers, hardware, software, and many other resources. Before product development and release, there may be many (e.g., more than a thousand, more than ten thousand, etc.) test cases to be tested. The number of test cases will continue to increase, and the test cases used may be modified based on changes in the functionality of the product under test. Due to reasons such as unclear design concepts and test case descriptions, different test cases may at least partially reproduce each other (e.g., two or more test cases may overlap in terms of test coverage). Furthermore, test cases may only provide partial coverage. Therefore, selecting an appropriate set of test cases is very difficult (e.g., affecting a range of outcomes such as testing manpower and material costs, testing time, and product issue efficiency).

[0048] Conventional methods for test case selection rely on testers' (e.g., human users') understanding of test cases and experience in the test plan to choose from existing test cases. Various factors influence the selection of test case sets, including the complexity of the product to be tested, the accuracy of the information used by the testers to select test cases, and the experience or knowledge of the testers choosing the test cases. These problems become more pronounced when the test object is complex and as the number of test cases increases. Expecting or requiring testers to deeply understand, analyze, and manually select all test cases correctly is unrealistic. When selecting test cases, testers may choose inappropriate test case sets (e.g., those leading to test duplication or repetition, missing test coverage, etc.), resulting in wasted resources and inefficient product testing. Therefore, effective methods are needed to extract the actual test steps and characteristic paths of test cases to help select more effective test case sets and significantly improve testing efficiency.

[0049] Test cases can use natural language to describe the test objectives, test procedures, and expected results. Typically, system testers select test cases based on the content described in such natural language test case descriptions. Test case descriptions can be highly dependent on language usage habits, the detail and accuracy of the test case descriptions written by the designers. If a test case description is ambiguous or inaccurate, testers may be misled in selecting appropriate test cases. Furthermore, different test cases and associated test case descriptions may have overlapping test points, and testers may not be able to distinguish or record all overlaps in detail. As a result, when testers must manually select large sets of test cases (e.g., thousands or more), there may be many duplicate tests, or the selected tests may miss coverage of target points.

[0050] Because test case descriptions have many problems in accurately identifying test points, this can affect the accuracy of methods that rely solely or primarily on test case descriptions for test case selection. Therefore, it is necessary to find an accurate and consistent description of test points and test case operations. In some implementations, test logs are used as a supplement to or replacement of test case descriptions. Test logs include product system logs generated during test case execution, serving as a supplement to or replacement of test case execution logs. Such product system logs advantageously include test actions and the system's response to these actions (e.g., information such as hardware configuration, functionality, execution sequence, I / O patterns, etc.). The content and timing of test events in the test logs can fully reflect the test points and operational steps of the test cases, and this correspondence will not change with execution time or the description of the test cases. In other words, the essential correspondence between test logs and test cases is one-to-one and unique.

[0051] With the widespread adoption of automated testing, test logs can provide a wealth of raw and effective data for in-depth analysis of test cases. Some internal relationships within test logs (e.g., the potential for a large number of log events) are not easily perceptible to the human eye, making manual comparison and analysis impractical and leading to various inaccuracies. Machine learning techniques can help handle these intricate relationships in large datasets. However, test logs are free-form text (e.g., unstructured text data) and therefore cannot be directly used as input for any machine learning approach. In some implementations, log vectorization and Natural Language Processing (NLP) are used to translate test logs into a format suitable for use as input to one or more machine learning algorithms. NLP can be used to extract log features that can vectorize the test log vocabulary. Log event vectors can then be obtained using custom log event templates. After such NLP and vectorization processing, the test logs can be fed into one or more machine learning algorithms for clustering test cases. Test case clusters generated based on log event vectorization can be used in various optimization fields, including but not limited to: deduplication of highly similar test cases; analysis of test coverage and hit rate of key test features in existing test case sets; and recommendation of suitable test cases for test case sets based on the characteristics and development maturity of the product under test during iterative product development.

[0052] Some implementations provide an intelligent test case identification system that is at least in part based on hierarchical clustering results generated from analyzing test logs, in order to optimize test case design and test case selection for test plans. The intelligent test case identification system may include functionality for parsing and vectorizing test logs, test case clustering, and test case set optimization.

[0053] The vectorization of test logs will now be described. Various NLP methods (including bag-of-words, word2vec, etc.) can be used for text vectorization. A text vectorization model can create an index for each word and then use such an index to represent the sentence. For example, Figure 3The process of bag-of-words text vectorization is illustrated, where dictionary 301 is generated from a set of statements 303, which is then transformed into multiple sets of text vectors 305-1, 305-2, and 305-3. Text vector set 305-1 suffers from the problem of different vector dimensions for different statements, making it unsuitable for input into machine learning algorithms. Text vector set 305-1 can therefore be transformed into text vector set 305-2, which includes a 6-dimensional vector for each word in a statement (e.g., using the frequency of each word in a statement instead of the word index). This is used for bag-of-words text vectorization. Word2vec can be used because bag-of-words text vectorization cannot reflect word relationships, and the human vocabulary corpus is enormous, resulting in vector dimensions that are too large for feasible computation in any algorithm. In system testing environments, test logs are semi-structured and have limited textual descriptions. Time series data is also important for test logs. In some implementations, a modified bag-of-words text vectorization method is used, which eliminates its shortcomings, making it suitable for applications in log-based test case identification scenarios.

[0054] Individual words in test logs are not always meaningful (e.g., they are not in human-readable form). A log statement can be viewed or considered a log event. The overall characteristics of a test log depend not only on individual log events, but also on the sequence of log events, frequency, arrival intervals (e.g., average arrival interval), time interval distribution, etc. Conventional log vectorization models cannot avoid coordinate transformations (e.g., from words to logs, and from logs to sequences) and also have high computational costs (e.g., for training), which can be prohibitive for large-scale testing environments. Conventional log vectorization models designed for log anomaly detection may abstract features and are not intended for test case comparison.

[0055] Different test logs (e.g., for different products) may have their own formats and templates. For example, Figure 4 Two product types, 401-A and 401-B, are illustrated, each with different associated test log formats 403-A and 403-B. For example, product type 401-A could be an internet server with an associated test log format 403-A, which focuses on information transformation. Product type 401-B could be a storage-related product with an associated test log format 403-B, which focuses on logical events. Various other examples are possible. Identifying the log format and associated log template for a specific test log based on its log structure is important for preprocessing.

[0056] In some implementations, log vectorization extracts the constant portion from log entries. For example, consider the following log entry:

[0057] "A_QA_ACTION [KILL:Appliance-WX-D0579-node-A-PM] Finished- Killing Appliance-WX-D0579-node-A-PM: pid=16752 uptime=1 / 11 / 2021 13:33:52 killSig=6

[0058] The log events will be structured into the following template:

[0059] "*_QA_ACTION [KILL:*-PM] Finished- Killing Appliance-*-PM: pid=* uptime=* killSig=6".

[0060] Test logs can be transformed into a combination of multiple log event templates. The general principle for log event templates is to ignore variables (e.g., numbers, object names, etc.) while retaining the logic and other parts of the log event. The process of parsing test logs to generate log event templates can be represented as follows:

[0061]

[0062] in This represents a line from the original log message. This indicates the total number of lines in the original log message. This represents the first... Okay, among them , This represents the function used to pass each line to the log event template as described above. This represents the log event template, and This represents a set of log event templates, which are called log templates.

[0063] The generation of log vectors will now be described. Assume... This indicates the total number of test cases. After parsing all test logs as described above, the resulting test case log template is represented as:

[0064]

[0065] each or multiple This will be reproduced because system log event templates are limited, and different test cases can invoke similar test steps. The length of the deduplicated test case set. This will be the capacity of the log event dictionary:

[0066]

[0067] In some cases, the product under test may have many unique log events that are too large (e.g., causing the dictionary to be too long to be processed using available computing resources). In such cases, the most sensitive events can be filtered out, allowing log messages irrelevant to test case steps to be ignored.

[0068] The process for generating log vectors may include creating a dictionary of log event templates. ), and through this dictionary, using the representation as The function is used to interpret the log template, such as Figure 5 As shown in Table 500. Figure 5 It also demonstrates the use of dictionaries and functions. An example of a transformed log event template 505. Different lengths of log event template 505 reflect different lengths of the original log message. The maximum length of the original log message can be determined based on the following:

[0069]

[0070] Log event templates shorter than the maximum length can be filled with 0 values, therefore the dictionary can include 0 elements, such as... Figure 5 As shown in Table 510. Each log template can be vectorized based on the following:

[0071]

[0072] in This indicates a log template. Indicates the first One test case log template, among which and This indicates the total number of test cases. This represents a function used to translate log event templates into vectors using a created dictionary. This represents the log event template, and The first in the log event template set for deduplication There are 10 log event templates, among which and Indicates the length of the deduplicated log event template set. Indicates the first Total number of lines in the test case log template This represents the total number of lines in the longest test case log template. Log vectors naturally represent the sequence of test events, and the dimensions of a log vector are... It should not contain a large number of uncounted numbers. The dictionary capacity can also be customized to make it acceptable for different product areas. Figures 6A to 6E The image shows an example of log vectorization of a set of original test logs. Figure 6A and Figure 6B The images show portions 600-1 and 600-2 of the original test log (collectively referred to as original test log 600). Original test log 600 consists of 38 lines. Figure 6C and Figure 6DParts 605-1 and 605-2 are shown, with the variables in parts 600-1 and 600-2 highlighted. Figure 6E The deduplicated log event template set 610 is shown, which includes 14 log event templates. Figure 6E The original log vector 615 of the log event template set 610 is also shown.

[0073] After generating the test log vectors, one or more clustering algorithms can be applied. Clustering algorithms can include artificial intelligence (AI) or machine learning (ML) algorithms. In some implementations, hierarchical clustering is utilized because the number of clusters is unknown before log computation, and the goal is to collect similar test cases (rather than delete test cases). Hierarchical clustering (also known as hierarchical cluster analysis) is an algorithm that groups similar objects into clusters. The endpoints of hierarchical clustering are a set of clusters, where each cluster is distinct from every other cluster, and objects within each cluster are generally similar to each other. When log vectors are provided, the distances between log vectors can be calculated (e.g., using Euclidean distance). The output of hierarchical clustering can be in the form of a dendrogram, which shows the hierarchical relationships between clusters. Hierarchical clustering may include the following steps:

[0074] 1. Initialization: Treat each sample (e.g., each log vector) as a cluster.

[0075] 2. Calculate Links: Use distance metrics (e.g., Euclidean distance) to determine links between clusters. Various methods can be used to calculate links, including but not limited to single links, complete links, average links, etc.

[0076] 3. Clustering: Determine the nearest clusters based at least in part on computational connections and merge the nearest clusters into the cluster hierarchy.

[0077] 4. Repeat steps 2 and 3 until a single cluster is reached at the highest level of the cluster hierarchy. The resulting cluster hierarchy can be used as input for test case selection and test plan improvement.

[0078] continue Figures 6A to 6EThe example will use fourteen test cases as source data to describe hierarchical clustering of test cases. After parsing the log event templates from the raw test logs, only those log events that significantly affect the test output are retained. Log events that significantly affect the test output can include, for example, log events corresponding to test step actions and processes, hardware information, I / O patterns, etc. Duplicate temporary information (such as network failures) may be filtered out or discarded. Triggering actions and system responses can be combined. The application of these filtering actions can significantly increase the test case vector dimension and log event dictionary width while preserving test case characteristics. For clarity, seven log event templates within a time period are selected to describe a test case, and it is assumed that there are twenty log event types within that time period.

[0079] Figure 7A Table 700 shows a dictionary of important log events for test cases, and Table 705 shows fourteen test case vectors. Figure 7B Figure 710 shows the hierarchical test case clustering results using the test case vectors in Table 705. The hierarchical test case clustering results shown in Figure 710 are generated by using average connectivity calculations on cluster connections. Accurate results are needed for the test case dataset to follow test design inputs and control overall computational costs. In Figure 710, the vertical axis represents samples (e.g., the individual test case names T1, T2, ... T7 in Table 705), and the horizontal axis represents the distances (e.g., Euclidean distances) between test case clusters (test case clusters TC1, TC2, ... TC14 in Table 705).

[0080] The test case hierarchical clustering results shown in Figure 710 can provide various useful information for designing test plans. For example, Figure 710 shows that the distance between test case clusters TC4 and TC7 is less than 1, which means that test case clusters TC4 and TC7 are highly likely to have at least some overlap or recurrence in test points. During test case refinement, test case clusters TC4 and TC7 can therefore be evaluated to reconsider their corresponding test objectives and test steps (e.g., to avoid overlap or recurrence of test points).

[0081] As another example, test plan design can choose test case clusters that are geographically dispersed as a quick and efficient way to select test cases to include in a test plan that maximizes test coverage (e.g., selecting test case clusters TC4, TC2, and TC10 to maximize test coverage is more efficient than selecting test case clusters T11, T13, and T14). Consider the constraint or requirement that a test plan can only include three test cases. Given this constraint, selecting one test case from each of test case clusters TC4, TC2, and TC10 to maximize test coverage is more efficient than selecting one test case from each of test case clusters T11, T13, and T14.

[0082] As another example, different test case clusters can represent or indicate different product module test points. In the early stages of testing, it is sufficient to run one (or a few) test cases from each of the test case clusters TC6, TC1, and TC10. If those selected tests pass, additional or future test cases can be selected from the test case clusters TC6, TC1, and TC10 (or other test case clusters within the test case cluster) to verify more details of the test points in different product modules.

[0083] Figure 8 The overall process flow 800 for test case analysis and selection is illustrated. Process flow 800 begins with a set of raw test logs 801, which are then processed in the log vectorization phase 803. In the log vectorization phase 803, a log event template dictionary 830 is used to parse the raw test logs 801 to identify a set of log event templates 832 (e.g., as described above). Figure 5 (As shown and described). The process flow 800 then proceeds to the test case clustering phase 805. In the test case clustering phase 805, the test case significance log event dictionary 850 is used to generate a set of test case vectors 852, which are analyzed to produce test case hierarchical clustering analysis results 854 (e.g., as described above in conjunction with...). Figure 7A and Figure 7B (As shown and described).

[0084] The process flow 800 then proceeds to the AI ​​test case optimizer phase 807. In the AI ​​test case optimizer phase 807, various optimizers are used to design test plans based on the test case hierarchical clustering analysis results 854 generated in the test case clustering phase 805. Such optimizers may include, but are not limited to, a test case deduplication optimizer 870, a feature coverage optimizer 872, and a production-stage test case selection optimizer 874. The test case deduplication optimizer 870 can be used to determine a set of deduplicated test cases (e.g., a set of test cases with reduced overlap in test coverage). The feature coverage optimizer 872 can be used to determine a set of test cases that provide optimal or improved feature coverage across desired test points. The production-stage test case selection optimizer 874 can be used to determine a set of test cases based on the product development maturity of the product to be tested. As described above, for example, a limited set of test cases can be selected early in product development, and if such tests pass, product development can continue, where additional test cases can be selected and run, and so on.

[0085] It should be understood that the specific advantages described above and elsewhere herein are associated with specific illustrative embodiments and are not required to exist in other embodiments. Furthermore, the specific types of information processing system features and functionalities shown in the accompanying drawings and as described above are merely exemplary, and numerous other arrangements may be used in other embodiments.

[0086] Now refer to Figure 9 and Figure 10 This section describes in more detail an illustrative implementation of a processing platform for implementing hierarchical clustering functionality used to generate test plans for IT assets. Although described in the context of information processing system 100, these platforms may also be used to implement at least parts of other information processing systems in other embodiments.

[0087] Figure 9 An exemplary processing platform including cloud infrastructure 900 is shown. Cloud infrastructure 900 includes components that can be used to implement… Figure 1 The information processing system 100 comprises at least a portion of its physical and virtual processing resources. The cloud infrastructure 900 includes multiple virtual machines (VMs) and / or container sets 902-1, 902-2, ... 902-L implemented using virtualization infrastructure 904. Virtualization infrastructure 904 runs on physical infrastructure 905 and illustratively includes one or more hypervisors and / or operating system-level virtualization infrastructures. Operating system-level virtualization infrastructure illustratively includes the kernel control group of a Linux operating system or other types of operating systems.

[0088] The cloud infrastructure 900 further includes a set of applications 910-1, 910-2, ... 910-L running on corresponding sets of VMs / containers in VM / container sets 902-1, 902-2, ... 902-L, under the control of the virtualization infrastructure 904. The VM / container set 902 may include a corresponding VM, a corresponding set of one or more containers, or a corresponding set of one or more containers running in a VM.

[0089] exist Figure 9 In some implementations of the scheme, the VM / container set 902 includes corresponding VMs implemented using a virtualization infrastructure 904 that includes at least one hypervisor. A hypervisor platform can be used to implement the hypervisor within the virtualization infrastructure 904, wherein the hypervisor platform has an associated virtual infrastructure management system. The underlying physical machines may include one or more distributed processing platforms, which include one or more storage systems.

[0090] exist Figure 9 In other implementations of the scheme, the VM / container set 902 includes corresponding containers implemented using virtualization infrastructure 904 that provides operating system-level virtualization functionality, such as support for Docker containers running on bare metal hosts or running on VMs. The containers are implemented illustratively using the corresponding kernel control group of the operating system.

[0091] As is evident from the above, one or more of the processing modules or other components of the information processing system 100 may each run on a computer, server, storage device, or other processing platform element. Such a given element can be considered as an example of an element more generally referred to herein as a "processing device". Figure 9 The cloud infrastructure 900 shown can represent at least a portion of a processing platform. Another example of such a processing platform is... Figure 10 The processing platform 1000 shown in the figure.

[0092] In this embodiment, the processing platform 1000 includes a portion of the information processing system 100 and includes a plurality of processing devices denoted as 1002-1, 1002-2, 1002-3, ... 1002-K, which communicate with each other via a network 1004.

[0093] Network 1004 may include any type of network, such as global computer networks (such as the Internet), WANs, LANs, satellite networks, telephone or wired networks, cellular networks, wireless networks (such as WiFi or WiMAX networks), or portions or combinations of these and other types of networks.

[0094] The processing device 1002-1 in the processing platform 1000 includes a processor 1010 coupled to a memory 1012.

[0095] The processor 1010 may include a microprocessor, a microcontroller, an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), a central processing unit (CPU), a graphics processing unit (GPU), a tensor processing unit (TPU), a video processing unit (VPU), or other types of processing circuitry, as well as portions or combinations of such circuitry elements.

[0096] Memory 1012 may include random access memory (RAM), read-only memory (ROM), flash memory, or other types of memory in any combination. Memory 1012 and other memories disclosed herein should be considered as illustrative examples of memories more generally referred to as “processor-readable storage media” storing executable program code of one or more software programs.

[0097] Articles of manufacture including such processor-readable storage media are considered illustrative embodiments. A given such article of manufacture may include, for example, a storage array, a storage disk, or an integrated circuit comprising RAM, ROM, flash memory, or other electronic memory, or any of a variety of other types of computer program products. As used herein, the term “article of manufacture” should be understood to exclude transient propagated signals. Numerous other types of computer program products including processor-readable storage media may be used.

[0098] The processing device 1002-1 also includes a network interface circuit 1014, which is used to interface the processing device with the network 1004 and other system components, and may include a conventional transceiver.

[0099] The other processing devices 1002 of the processing platform 1000 are assumed to be configured in a manner similar to that shown for the processing device 1002-1 in the figure.

[0100] Furthermore, the specific processing platform 1000 shown in the figure is presented only by way of example, and the information processing system 100 may include additional or alternative processing platforms, as well as a number of different processing platforms in any combination, wherein each such platform includes one or more computers, servers, storage devices or other processing devices.

[0101] For example, other processing platforms used to implement illustrative implementation schemes may include converged infrastructure.

[0102] Therefore, it should be understood that in other embodiments, different arrangements of additional or alternative elements may be used. At least a subset of these elements may be implemented together on a common processing platform, or each such element may be implemented on a separate processing platform.

[0103] As previously indicated, components of the information processing system disclosed herein can be implemented, at least in part, in the form of one or more software programs stored in memory and executed by a processor of a processing device. For example, at least part of the functionality of hierarchical clustering for generating test cases for test plans for IT assets, as disclosed herein, is illustratively implemented in the form of software running on one or more processing devices.

[0104] It should be emphasized again that the above embodiments are presented for illustrative purposes only. Many variations and other alternative embodiments can be used. For example, the disclosed technology is applicable to a variety of other types of information processing systems, test logs, test cases, etc. Moreover, the specific configurations of the system and apparatus elements illustratively shown in the accompanying drawings, and the associated processing operations, may change in other embodiments. Furthermore, the various assumptions made above in describing the illustrative embodiments should be considered exemplary and not as requirements or limitations of this disclosure. Numerous other alternative embodiments within the scope of the appended claims will be apparent to those skilled in the art.

Claims

1. A device for hierarchical clustering of test cases, comprising: At least one processing device, the at least one processing device including a processor coupled to a memory; The at least one processing device is configured to perform the following steps: Obtain a set of test logs generated by executing multiple test cases on one or more of the information technology assets of the information technology infrastructure; Parsing this set of test logs generates a set of log event templates for the test actions performed during the execution of the multiple test cases on one or more of the multiple information technology assets of the information technology infrastructure; The generated log event templates are used to generate vector representations of the multiple test cases; Hierarchical clustering is performed on the plurality of test cases using one or more machine learning-based hierarchical clustering algorithms, wherein the generated vector representations of the plurality of test cases are taken as input by the machine learning-based hierarchical clustering algorithms. Based at least on the hierarchical clustering results of the plurality of test cases, generate one or more test plans for a given information technology asset among the plurality of information technology assets of the information technology infrastructure; as well as Execute the one or more test plans on a given information technology asset among the plurality of information technology assets of the information technology infrastructure.

2. The device of claim 1, wherein the plurality of information technology assets of the information technology infrastructure includes at least one of the following: One or more physical computing resources and one or more virtual computing resources; and Software that runs on at least one of one or more physical computing resources and one or more virtual computing resources.

3. The device of claim 1, wherein the set of test logs includes at least one of the following: One or more test case execution logs generated from the plurality of test cases; and One or more system logs, which are generated by executing the multiple test cases on one or more of the multiple information technology assets of the information technology infrastructure.

4. The device of claim 1, wherein a given test log in the set of test logs specifies: One or more of the test actions taken during the execution of at least one of the plurality of test cases; The response of one or more of the plurality of information technology assets of the information technology infrastructure to the one or more test actions; and At least one of the hardware and software configurations of one or more of the plurality of information technology assets of the information technology infrastructure.

5. The device according to claim 1, wherein for a given test log in the set of test logs, parsing the set of test logs to generate the set of log event templates includes: Identify multiple log events in the given test log; For each of the plurality of log events in the given test log, extract the constant portion and remove one or more variables; as well as The constant portion of each of the plurality of log events in the given test log is extracted and converted into a given log event template from the plurality of log event templates.

6. The device of claim 1, wherein generating the vector representation of the plurality of test cases using the log event templates generated by the group comprises: Create a log event template dictionary, which includes a set of deduplicated log event templates used in the plurality of test cases; as well as Each item in the test log is translated into a log vector using the log event template dictionary.

7. The device of claim 6, wherein the log event template dictionary includes specified padding elements, and wherein translating each of the test logs into a log vector using the log event template dictionary includes translating each of the test logs into a log vector of equal dimension by padding the log vector with additional entries of the specified padding elements to a length less than that of the longest log vector.

8. The device of claim 1, wherein performing the hierarchical clustering on the plurality of test cases comprises: Each of the multiple test cases is initialized as a corresponding cluster in one of the multiple clusters in the first level of the cluster hierarchy; Calculate the connections between each of the plurality of clusters in the first level of the cluster hierarchy; as well as Based at least on computational connections, subsets of the multiple test cases are clustered in the second level of the cluster hierarchy.

9. The device of claim 8, wherein calculating the connection includes calculating a distance metric between the generated vector representations of the plurality of test cases.

10. The device of claim 8, wherein the calculated connection includes at least one of a single connection, a complete connection, and an average connection.

11. The device of claim 1, wherein generating a given test plan in one of the one or more test plans includes selecting a deduplicated subset of the plurality of test cases that provides coverage for a specified set of test points to include in the given test plan, the deduplicated subset of the plurality of test cases being determined at least based on the hierarchical clustering results of the plurality of test cases.

12. The device of claim 1, wherein generating a given test plan in one of the one or more test plans includes selecting a subset of the plurality of test cases to provide feature coverage for a set of specified features of the product to be tested by the given test plan, the selected subset of the plurality of test cases being determined at least based on the hierarchical clustering results of the plurality of test cases.

13. The device of claim 1, wherein generating a given test plan in one of the one or more test plans includes selecting a sequence of two or more subsets of the plurality of test cases to include in the given test plan, the selected sequence of the two or more subsets of the plurality of test cases being determined at least based on the hierarchical clustering results of the plurality of test cases, wherein the sequence of the two or more subsets of the plurality of test cases includes: The first subset of test cases to be executed first, wherein the first subset of test cases is to be tested by the first subset of features of the product to be tested by the given test plan; as well as The test cases are to be executed after the first subset of the test cases has passed, wherein the second subset of the test cases tests a second subset of the features of the product to be tested by the given test plan.

14. The device of claim 13, wherein performing the given test plan on the one given information technology asset of the plurality of information technology assets of the information technology infrastructure comprises: Based on the selected sequence, execute two or more subsets of the plurality of test cases.

15. A computer program product comprising a non-transitory processor-readable storage medium storing program code of one or more software programs, wherein the program code, when executed by at least one processing device, causes the at least one processing device to perform the following steps: Obtain a set of test logs generated by executing multiple test cases on one or more of the information technology assets of the information technology infrastructure; Parsing this set of test logs generates a set of log event templates for the test actions performed during the execution of the multiple test cases on one or more of the multiple information technology assets of the information technology infrastructure; The generated log event templates are used to generate vector representations of the multiple test cases; Hierarchical clustering is performed on the plurality of test cases using one or more machine learning-based hierarchical clustering algorithms, wherein the generated vector representations of the plurality of test cases are taken as input by the machine learning-based hierarchical clustering algorithms. as well as Based at least on the hierarchical clustering results of the plurality of test cases, one or more test plans are generated for a given information technology asset among the plurality of information technology assets of the information technology infrastructure.

16. The computer program product of claim 15, wherein generating a given test plan in one of the one or more test plans includes selecting a deduplicated subset of the plurality of test cases that provides coverage for a specified set of test points to include in the given test plan, the deduplicated subset of the plurality of test cases being determined at least based on the hierarchical clustering results of the plurality of test cases.

17. The computer program product of claim 15, wherein generating a given test plan in one of the one or more test plans includes selecting a subset of the plurality of test cases to provide feature coverage for a set of specified features of the product to be tested by the given test plan, the selected subset of the plurality of test cases being determined at least based on the hierarchical clustering results of the plurality of test cases.

18. A method for hierarchical clustering of test cases, comprising: Obtain a set of test logs generated by executing multiple test cases on one or more of the information technology assets of the information technology infrastructure; Parsing this set of test logs generates a set of log event templates for the test actions performed during the execution of the multiple test cases on one or more of the multiple information technology assets of the information technology infrastructure; The generated log event templates are used to generate vector representations of the multiple test cases; Hierarchical clustering is performed on the plurality of test cases using one or more machine learning-based hierarchical clustering algorithms, wherein the generated vector representations of the plurality of test cases are taken as input by the machine learning-based hierarchical clustering algorithms. as well as Based at least on the hierarchical clustering results of the plurality of test cases, generate one or more test plans for a given information technology asset among the plurality of information technology assets of the information technology infrastructure; The method is performed by at least one processing device, the processing device including a processor coupled to a memory.

19. The method of claim 18, wherein generating a given test plan in one of the one or more test plans includes selecting a deduplicated subset of the plurality of test cases that provides coverage for a specified set of test points to include in the given test plan, the deduplicated subset of the plurality of test cases being determined at least based on the hierarchical clustering results of the plurality of test cases.

20. The method of claim 18, wherein generating a given test plan in one of the one or more test plans comprises selecting a subset of the plurality of test cases that provides feature coverage for a specified set of features of the product to be tested by the given test plan to include in the given test plan, the selected subset of the plurality of test cases being determined at least based on the hierarchical clustering results of the plurality of test cases.

Images