Continuous production log monitoring drives automated test case generation and bug triage

Abstract

Embodiments of this disclosure relate to automated test case generation and error classification driven by continuous production log monitoring. This document provides methods, systems, and non-transitory processor-readable storage media for test generation systems. An example method includes the test generation system retrieving a sequence of log entries associated with a transaction identifier from a centralized log storage system in response to an alert notification. The test generation system uses a log parsing engine to process the sequence of log entries to generate structured log data. The test generation system uses a test case generation engine to generate test cases based on the structured log data, wherein the test cases include a test objective, steps to reproduce the error condition, expected results, environment details, and test data.

Description

Technical Field

[0001] This field generally involves software test cases within information processing systems. Background Technology

[0002] Software applications in production environments often encounter errors and anomalies that were not anticipated during testing. Summary of the Invention

[0003] Illustrative embodiments provide techniques for implementing a test generation system in a storage system. For example, an illustrative embodiment includes a test generation system that retrieves a sequence of log entries associated with a transaction identifier from a centralized log storage system in response to an alarm notification. The test generation system uses a log parsing engine to process the sequence of log entries to generate structured log data. The test generation system uses a test case generation engine to generate test cases based on the structured log data, wherein the test cases include a test objective, steps to reproduce the error condition, expected results, environment details, and test data. Other types of processing devices may be used in other embodiments. These and other illustrative embodiments include, but are not limited to, apparatuses, systems, methods, and processor-readable storage media. Attached Figure Description

[0004] Figure 1 An information processing system including a test generation system is shown in an illustrative embodiment.

[0005] Figure 2 A flowchart illustrating the process for testing the generation system in an illustrative embodiment is shown.

[0006] Figure 3 A system architecture for testing the generation system is shown in an illustrative embodiment.

[0007] Figure 4 The illustrated embodiment shows parsed log information mapped to a predefined data format.

[0008] Figure 5 Sample output of the generated test cases in the illustrative embodiment is shown.

[0009] Figure 6 An example of misclassified data generated by the misclassification engine is shown in an illustrative embodiment.

[0010] Figure 7 and Figure 8 An example of a processing platform that can be used to implement at least a portion of a test generation system embodiment is shown. Detailed Implementation

[0011] This document will describe illustrative embodiments with reference to exemplary computer networks and associated computers, servers, network devices, or other types of processing devices. However, it should be understood that these and other embodiments are not limited to use with the specific illustrative network and device configurations shown. Therefore, the term "computer network" as used herein is intended to be interpreted broadly to cover, for example, any system comprising multiple networked processing devices.

[0012] The following describes a technique used in implementing a test generation system to generate test cases based on monitoring logs created during the execution of a software application. The test generation system retrieves a sequence of log entries associated with transaction identifiers from a centralized log storage system in response to alert notifications. The test generation system uses a log parsing engine to process the log entry sequence to generate structured log data. The test generation system then uses a test case generation engine to generate test cases based on the structured log data, where each test case includes a test objective, steps to reproduce the error condition, expected results, environment details, and test data.

[0013] Production systems running software applications may encounter errors and issues. These errors and issues are identified in application logs that capture details during the execution of the software application. Ideally, regression test cases can capture these errors and issues, allowing them to be identified and resolved during regression testing, preventing them from occurring in production.

[0014] Traditional techniques rely on manually creating test cases, which fails to account for actual errors found in application logs. They cannot effectively parse raw application logs, process them into meaningful insights, and transform them into structured data. Traditional techniques cannot analyze application logs to generate missing test scenarios and test cases. Manually categorizing and integrating test cases into regression test suites is time-consuming and prone to human error.

[0015] In contrast, improvements to test case generation are achieved through a test generation system, as described herein, in at least some implementations of the current art. This test generation system retrieves a sequence of log entries associated with a transaction identifier from a centralized log storage system in response to an alert notification. The test generation system processes the sequence of log entries using a log parsing engine to generate structured log data. The test generation system then uses a test case generation engine to generate test cases based on the structured log data, where each test case includes a test objective, steps to reproduce the error condition, expected results, environment details, and test data.

[0016] Therefore, the current technology aims to provide a method and system for a test generation system that parses application logs to obtain error information and then automatically generates test cases using that information. Another objective is to generate test cases, including identifying workflows, errors, and points of failure. Yet another objective is to leverage a natural language processing engine to parse application logs to generate test scenarios in a human-readable format. A further objective is to categorize errors derived from application logs and associate them with missing or insufficient test cases to provide targeted test improvements. Another objective is to ensure that issues discovered in the production test environment are correctly addressed in future test cycles. Finally, a suitable technique can be applied to a variety of domain-specific use cases.

[0017] In at least some implementations of the current technology described herein, using a test generation system can provide one or more of the following advantages: automatically generating, classifying, and integrating test cases exported from production logs using natural language processing, automatically adding missing test cases to a regression test case suite, and automatically refining the regression test case suite each time the software application is executed.

[0018] Compared to traditional techniques, improvements in test case generation are achieved through a test generation system, as described herein, in at least some implementations of the current technology. This test generation system retrieves a sequence of log entries associated with a transaction identifier from a centralized log storage system in response to an alert notification. The test generation system uses a log parsing engine to process the sequence of log entries to generate structured log data. The test generation system then uses a test case generation engine to generate test cases based on the structured log data, where each test case includes a test objective, steps to reproduce the error condition, expected results, environment details, and test data.

[0019] In an example embodiment of the current technology, the test generation system uses vector embedding to compare the generated test cases with existing test cases in a test case repository to identify similar test cases, and stores the generated test cases in the test case repository when no similar test cases are identified.

[0020] In an example embodiment of the current technology, the test generation system uses a pre-trained language model to convert existing test cases into vector embeddings, stores the embeddings in a vector database, and calculates a cosine similarity score between the generated test cases and existing test cases.

[0021] In example embodiments of the current technology, the vector database includes at least one of the following: Pinecone and Milvus.

[0022] In an example embodiment of the current technology, the test generation system categorizes cosine similarity scores into percentage ranges to identify relevant matches.

[0023] In an example embodiment of the current technology, an alarm notification is received from an alarm system that indicates erroneous conditions in the production environment.

[0024] In example embodiments of the current technology, alarm notifications include predefined error patterns and thresholds for triggering alarms.

[0025] In example embodiments of the present technology, a centralized log storage system includes at least one of the following: ELKStack, Splunk, or Fluentd.

[0026] In example embodiments of the current technology, the test generation system identifies log entries based on the transaction ID or timestamp associated with the alert.

[0027] In an example embodiment of the current technology, the test generation system extracts metadata including date, time, component information, and environment details, and extracts log information including error messages, stack traces, and test data.

[0028] In example embodiments of the current technology, the metadata includes a log type selected from at least one of the following: entry point log, user action log, and failure point log.

[0029] In an example embodiment of the current technology, the test generation system uses a transformer-based language model with Wordpiece segmentation.

[0030] In example embodiments of the current technology, the test generation system uses a small number of sample hints with domain-specific examples.

[0031] In example embodiments of the present technology, environment details include at least one of the following: operating system name, operating system version, database, database version, and hardware specifications.

[0032] In an example embodiment of the current technology, the test generation system uses an error classification engine to classify error conditions to determine at least one of the following: error type, severity, and impact.

[0033] In example embodiments of the current technology, error types include at least one of the following: related infrastructure, data problems, and corrupted functionality.

[0034] In example embodiments of the current technology, the impact includes at least one of the following: service impact and transaction failure.

[0035] In example embodiments of the current technology, test cases are generated in a human-readable format.

[0036] Figure 1 A computer network (also referred to herein as an information processing system) 100 configured according to an illustrative embodiment is shown. Computer network 100 includes a test generation system 101, a log storage system 105, a test case repository 103, and production systems 102-N. Test generation system 101, log storage system 105, test case repository 103, and production systems 102-N are coupled to network 104, wherein in this embodiment, network 104 is assumed to represent a subnetwork or other relevant portion of the larger computer network 100. Therefore, elements 100 and 104 are both examples referred to herein as "networks," but... Figure 1 In the context of this embodiment, the latter is assumed to be a component of the former. The test generation system 101 may reside on a storage system. Such a storage system may include any of a variety of different types of storage, including network-attached storage (NAS), storage area network (SAN), direct-attached storage (DAS), and distributed DAS, as well as combinations of these and other storage types, including software-defined storage.

[0037] Each of the test generation system 101, log storage system 105, and production systems 102-N may include, for example, portions of servers and / or one or more server systems, as well as devices such as mobile phones, laptop computers, tablet computers, desktop computers, or other types of computing devices. Such devices are examples of what is more generally referred to herein as “processing devices.” Some of these processing devices are also generally referred to herein as “computers.”

[0038] In some embodiments, the test generation system 101, the log storage system 105, and the production system 102-N include corresponding computers associated with a specific company, organization, or other enterprise. Furthermore, at least a portion of the computer network 100 may also be referred to herein as collectively constituting an "enterprise network." As those skilled in the art will recognize, many other operational scenarios involving various types and arrangements of processing equipment and networks are possible.

[0039] Furthermore, it should be recognized that the term “user” in this context and elsewhere in this document is intended to be interpreted broadly to include entities such as humans, hardware, software, or firmware, and various combinations of such entities.

[0040] Assuming network 104 includes part of a global computer network such as the Internet, although other types of networks may also be part of computer network 100, including wide area networks (WANs), local area networks (LANs), satellite networks, telephone or wired networks, cellular networks, wireless networks (such as Wi-Fi or WiMAX networks), or portions or combinations of these and other types of networks. Therefore, in some embodiments, computer network 100 includes a combination of multiple different types of networks, each including processing devices configured to communicate using Internet Protocol (IP) or other relevant communication protocols.

[0041] Associated with the test generation system 101 are one or more input / output devices, illustratively including a keyboard, a display, or other types of input / output devices in any combination. For example, such input / output devices may be used to support one or more user interfaces of the test generation system 101, as well as to support communication between the test generation system 101 and other related systems and devices not explicitly shown. For example, a dashboard may be provided to the user to view the results generated by the test generation system 101. The one or more input / output devices may also be associated with any of the log storage system 105, the test case repository 1035, and the production systems 102-N.

[0042] Additionally, assuming Figure 1 The test generation system 101 in the embodiments is implemented using at least one processing device. Each such processing device typically includes at least one processor and associated memory, and implements one or more functional modules for controlling certain features of the test generation system 101.

[0043] More specifically, the test generation system 101 in this embodiment may include a processor coupled to a memory and a network interface.

[0044] Processors illustratively include microprocessors, microcontrollers, application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs) or other types of processing circuitry, as well as portions or combinations of these circuitry elements.

[0045] Memory illustratively includes any combination of random access memory (RAM), read-only memory (ROM), or other types of memory. The memory and other memories disclosed herein can be considered as examples of what is more generally referred to as a “processor-readable storage medium” for storing executable computer program code or other types of software programs.

[0046] One or more embodiments include articles of manufacture, such as computer-readable storage media. Examples of articles of manufacture include, but are not limited to, storage devices such as storage disks, storage arrays, or integrated circuits containing memory, as well as various other types of computer program products. As used herein, the term "article of manufacture" should be understood to exclude transient propagating signals. These and other references to "disk" herein are intended to generally refer to storage devices, including solid-state drives (SSDs), and should therefore not be considered as being limited in any way to rotating magnetic media.

[0047] The network interface allows the test generation system 101 to communicate with the log storage system 105, the test case repository 103, and the production system 102-N via network 104, and illustratively includes one or more conventional transceivers.

[0048] The test generation system 101 can be implemented at least in part as software stored in memory and executed by a processor, and can reside in any processing device. The test generation system 101 can be a standalone plug-in that can be included within the processing device.

[0049] It should be understood that this is presented only as an illustrative example. Figure 1 The test generation system 101 shown relates to computer network 100 and includes a specific set of elements such as the test generation system 101, log storage system 105, test case repository 103, and production systems 102-N. Additional or alternative elements may be used in other embodiments. Therefore, another embodiment includes additional or alternative systems, devices, and other network entities, as well as different arrangements of modules and other components. For example, in at least one embodiment, one or more test generation systems in test generation system 101 may reside on the same processing platform and / or a portion of the same processing platform.

[0050] References, for example Figure 2 The flowchart describes in more detail the exemplary process of the test generation system 101 in the computer network 100.

[0051] Figure 2 This is a flowchart illustrating the process for performing the test generation system 101 in an illustrative embodiment. It should be understood that this particular process is merely an example, and additional or alternative processes may be performed in other embodiments.

[0052] At point 200, test generation system 101 retrieves a sequence of log entries associated with a transaction identifier from centralized log storage system 105 in response to an alarm notification. In an example embodiment, centralized log storage system 105 includes at least one of the following: ELK Stack, Splunk, or Fluentd. In an example embodiment, test generation system 101 includes log storage system 105 to act as a centralized log aggregation solution to collect logs from various sources in a production environment, including production systems 102-N. In an example embodiment, test generation system 101 stores the aggregated logs in a searchable and scalable storage system to enable rapid retrieval and analysis.

[0053] In the example embodiment, alarm notifications are received from an alarm system that indicates fault conditions in the production environment. In the example embodiment, when production system 102-N executes applications, logs associated with these applications (i.e., application logs, production logs, or production application logs) are stored in a centralized, searchable log data store, such as... Figure 3 As shown in the example, an alerting system is integrated to issue alerts whenever an application error or anomaly occurs. In an example embodiment, the alerting system integration includes alert configuration and alert triggers. Alert configuration includes alert rules set in the alerting system (e.g., Prometheus Alert Manager, Splunk Alerts, etc.) based on predefined error patterns, thresholds, or specific conditions in the logs. Alert triggers include notifications sent by the alerting system to the log analyzer when an alert is triggered. In an example embodiment, log storage system 105 includes an alerting system and a centralized, searchable log data store.

[0054] In an example embodiment, when an alert is triggered, transaction-specific application logs are received and the transaction flow is traced sequentially. In an example embodiment, the alert notification includes a predefined error pattern and a threshold for triggering the alert. In an example embodiment, the test generation system 101 identifies log entries based on the transaction ID or timestamp associated with the alert. In an example embodiment, the test generation system 101 includes log retrieval and log tracing. In an example embodiment, when the test generation system 101 receives an alert, it uses the TraceID from the alert to query logs from the log data store. This includes all logs from the entry point as well as logs related to user actions up to the point of failure.

[0055] In an example embodiment, when the test generation system 101 receives an alert, a log analyzer performs log retrieval. The log analyzer identifies relevant log entries by parsing logs associated with a specific transaction or time range indicated by the alert. In an example embodiment, the test generation system 101 performs log tracing by linking log entries (i.e., using transaction IDs, correlation IDs, or timestamps) to trace the entire transaction flow to reconstruct the sequence of events that led to the error or anomaly.

[0056] At point 202, the test generation system 101 uses a log parsing engine to process the sequence of log entries to generate structured log data. In an example embodiment, the test generation system 101 adds additional context to the logs, such as environment details and system configuration (such as hardware specifications). In an example embodiment, this information is present in received alerts. In an example embodiment, the environment details include at least one of the following: operating system name, operating system version, database, and database version.

[0057] In an example embodiment, test generation system 101 uses a large language model (LLM) based on natural language processing (NLP) to analyze logs to extract key information. In another example embodiment, test generation system 101 uses a transformer-based LLM (such as GPT or BERT) to synthesize logs using efficient word segmentation techniques (such as Wordpiece) and builds structured data for each log entry. In yet another example embodiment, test generation system 101 includes a log retrieval module that retrieves the logs for any given transaction. Test generation system 101 generates a structured dataset for the log sequence retrieved by the log retrieval module. Test generation system 101 feeds the structured dataset into the test generation module, which includes a test case generation engine to generate test steps and test data.

[0058] In an example embodiment, the test generation system 101 feeds the retrieved log traces into a log parsing engine (i.e., a natural language processing (NLP) engine), which processes the unstructured log data into meaningful information such as error messages, stack traces, and user actions. Figure 4 The illustration shows parsed log information mapped to a predefined data format. In the example embodiment, the log parsing engine uses a finely tuned LLM, which efficiently parses logs, extracts key information, and transforms the data into a standard template. In another example embodiment, the log parsing engine is deployed with a small sample hinting technique that includes more domain-specific examples to carefully examine the logs and extract data.

[0059] In an example embodiment, the test generation system 101 extracts metadata including date, time, component information, and environment details, and extracts log information including error messages, stack traces, and test data. In the example embodiment, the metadata includes log types selected from at least one of the following: entry point logs, user action logs, and failure point logs.

[0060] In an example embodiment, the test generation system 101 includes an error classification engine that consumes a structured dataset derived from log traces and classifies it based on the nature of the problem.

[0061] At position 204, the test generation system 101 uses a test case generation engine to generate test cases based on structured log data. The test cases include test objectives, steps to reproduce the error condition, expected results, environment details, and test data.

[0062] In an example embodiment, the test case generation engine includes a pre-trained LLM that is capable of understanding structured data templates extracted from application logs (such as...). Figure 4 (As shown in the image) and automatically generate test cases. The extracted data sequence enables the test case generation engine to understand key elements, such as entry points, user workflows, failure points, and test data, to effectively transform these key elements into test elements designed to reproduce the identified problem in test cases.

[0063] In the example embodiment, the structured dataset is linked to a test case generation engine that generates corresponding test cases. The intent of generating test cases is to help users replicate conditions that lead to errors or exceptions. In the example embodiment, the generated test cases include defined inputs, expected outputs, and associated test conditions. In the example embodiment, the test generation system 101 generates test cases in a human-readable format.

[0064] In the example implementation, structured data (i.e., structured log data) is passed as input to the test case generation engine to generate test cases with test objectives, reproducible steps, expected results, environment details, and test data. The test objective is the high-level purpose of the test case, such as "verify that the database connection was not lost during user login." The steps to be reproduced are the step-by-step sequence of actions that lead to the error. The expected result is the expected behavior when the test case passes, such as "the user should log in successfully without a database connection error." Environment details are information about the system environment in which the error occurred. Test data includes data inputs related to the test case. Figure 5 The sample output of the generated test cases is shown.

[0065] In an example embodiment, the test generation system 101 compares the generated test cases with existing test case suites in the test case repository 103 to check for overlaps or gaps. A vector database is used to index all test cases in the test case repository 103 so that the test generation system 101 can search for newly generated test cases. In an example embodiment, the test generation system 101 uses vector embeddings for comparison to identify similar test cases. In an example embodiment, the test generation system 101 uses a test comparison engine (i.e., a pre-trained language model, such as the Sentence Transformers library) to convert existing test cases into vector embeddings and stores the embeddings in the vector database. In an example embodiment, the vector database includes at least one of the following: Pinecone and Milvus, because these vector databases are optimized for high-dimensional data retrieval.

[0066] In an example embodiment, the test generation system 101 converts test cases from the test case repository 103 into vector embeddings and stores them in a vector database. In the example embodiment, once the test case generation engine generates newly generated test cases, the test comparison engine transforms the newly generated test cases into vector embeddings using the same embedding model. The test comparison engine compares the newly generated test cases (now transformed into vectors) with the vector embeddings in the vector database. If no match is found, the test generation system 101 stores the newly generated test cases in the test case repository 103.

[0067] Test generation system 101 queries a vector database to find similar test cases based on cosine similarity. Test generation system 101 calculates a cosine similarity score between the generated test cases and existing test cases. The vector database returns a sorted list of matches, which can be converted into percentage similarity scores. In an example embodiment, test generation system 101 categorizes the cosine similarity scores into percentage ranges to identify relevant matches, such as 90-100%, 80-89%, etc., to highlight the most relevant matches. Test generation system 101 utilizes efficient vector indexing and search capabilities to streamline test case validation and comparison, ensuring rapid identification of duplicates or near-duplicates within the repository. If any match is found, test generation system 101 will not mark the newly generated test case as a new test case. When no similar test cases are identified, test generation system 101 stores the generated test cases in test case repository 103. Figure 3 The system architecture used for testing the generation system 101 is shown.

[0068] In an example embodiment, the test generation system 101 includes an error classification engine to identify error labels from structured log data templates extracted by the test generation system 101 from application logs. In the example embodiment, the error classification engine is a pre-trained LLM that understands the structured log data templates. The error classification engine classifies errors by analyzing key information such as error type, error message, detailed stack trace, environment, and system configuration. In the example embodiment, various training datasets are used to train the error classification engine to improve the performance of error labeling.

[0069] In an example embodiment, test generation system 101 passes a structured dataset through an error classification engine to categorize the problem type into at least one of the following: error type, severity, and impact. In an example embodiment, error type includes at least one of the following: related infrastructure, data issues, and corrupted functionality. For example, the error type details what kind of problem the logs describe. In an example embodiment, severity defines the degree of criticality of the problem, such as minor, significant, or critical. In an example embodiment, impact includes at least one of the following: service impact and transaction failure. For example, which part of the system or workflow is affected. Figure 6 This shows a sample of misclassified data generated by the misclassification engine.

[0070] In the example embodiment, the test generation system 101 stores test cases and error classifications in a test case repository 103 so that users can view and add them to their existing test case repositories. In the example embodiment, newly generated test cases can be viewed by a viewer and added to the regression test repository.

[0071] The illustrative embodiments described above offer significant advantages over conventional methods. For example, some embodiments are configured to significantly optimize the generation of regression test cases. These and other embodiments can effectively improve anomaly detection before executing software applications on production systems. The embodiments disclosed herein use natural language processing to automatically generate, classify, and integrate test cases derived from production logs. The embodiments disclosed herein automatically add missing test cases to the regression test case suite. The embodiments disclosed herein automatically refine the regression test case suite each time the software application is executed.

[0072] It should be recognized that the specific advantages described above and elsewhere herein are associated with specific illustrative embodiments and are not necessarily present in other embodiments. Furthermore, the specific types of information processing system features and functions shown in the accompanying drawings and described above are merely exemplary, and many other arrangements may be used in other embodiments.

[0073] As mentioned above, at least a portion of the information processing system 100 can be implemented using one or more processing platforms. A given such processing platform includes at least one processing device, which includes a processor coupled to memory. In some embodiments, the processor and memory include corresponding processor and memory elements using virtual machines or containers provided by one or more underlying physical machines. The term "processing device" as used herein is intended to be interpreted broadly to encompass various arrangements of physical processors, memory, and other device components, as well as virtual instances of these components. For example, in some embodiments, a "processing device" may include one or more virtual processors or execute across one or more virtual processors. Thus, a processing device can be physical or virtual and can execute across one or more physical or virtual processors. It should also be noted that a given virtual device may be mapped to a portion of a physical device.

[0074] Some illustrative embodiments of a processing platform for implementing at least a portion of an information processing system include a cloud infrastructure comprising virtual machines implemented on a hypervisor running on a physical infrastructure. The cloud infrastructure also includes a collection of applications running on respective virtual machines within the virtual machines under the control of the hypervisor. Multiple hypervisors may also be used, each providing a collection of virtual machines using at least one underlying physical machine. Different collections of virtual machines provided by one or more hypervisors can be utilized across multiple instances of various components of the configuration system.

[0075] These and other types of cloud infrastructure can be used to provide an environment also referred to in this article as a multi-tenant environment. One or more system components, or portions thereof, are illustratively implemented for use by tenants in such a multi-tenant environment.

[0076] As mentioned above, cloud infrastructure as disclosed herein can include cloud-based systems. In illustrative embodiments, virtual machines provided in such systems can be used to implement at least a portion of a computer system.

[0077] In some embodiments, the cloud infrastructure additionally or alternatively includes multiple containers implemented using container host devices. For example, as detailed herein, a given container in the cloud infrastructure illustratively includes Docker containers or other types of Linux containers (LXC). Containers run on virtual machines in a multi-tenant environment, although other arrangements are possible. Containers are utilized to implement various different types of functionality within the information processing system 100. For example, containers can be used to implement corresponding processing devices that provide computing and / or storage services for cloud-based systems. Similarly, containers can be used in conjunction with other virtualization infrastructure, such as virtual machines implemented using hypervisors.

[0078] Now refer to Figure 7 and Figure 8 An illustrative embodiment of the processing platform is described in more detail. Although described in the context of information processing system 100, these platforms can also be used to implement at least a portion of other information processing systems in other embodiments.

[0079] Figure 7 An example processing platform including cloud infrastructure 700 is illustrated. Cloud infrastructure 700 includes a combination of physical and virtual processing resources for implementing at least a portion of information processing system 100. Cloud infrastructure 700 includes multiple virtual machines (VMs) and / or container sets 702-1, 702-2, ... 702-L implemented using virtualization infrastructure 704. Virtualization infrastructure 804 runs on physical infrastructure 705 and illustratively includes one or more hypervisors and / or operating system-level virtualization infrastructures. Operating system-level virtualization infrastructure illustratively includes a kernel control group of a Linux operating system or other types of operating system.

[0080] Cloud infrastructure 800 also includes collections of applications 710-1, 710-2, ... 710-L running on a corresponding VM / container set 702-1, 702-2, ... 702-L under the control of virtualization infrastructure 704. VM / container set 702 includes a corresponding VM, a corresponding collection of one or more containers, or a corresponding collection of one or more containers running within a VM. Figure 7 In some implementations of the embodiments, the VM / container set 702 includes a corresponding VM implemented using a virtualization infrastructure 704 including at least one hypervisor.

[0081] A hypervisor platform can be used to implement hypervisors within a virtualized infrastructure 704, wherein the hypervisor platform has an associated virtual infrastructure management system. The underlying physical machines include one or more distributed processing platforms, which include one or more storage systems.

[0082] exist Figure 7 In other implementations of the embodiments, the VM / container set 702 includes corresponding containers implemented using virtualization infrastructure 704, which provides operating system-level virtualization capabilities, such as support for Docker containers running on a bare-metal host or on a VM. The containers are illustratively implemented using the corresponding kernel control group of the operating system.

[0083] As is evident from the foregoing, one or more processing modules or other components of the information processing system 100 may each run on a computer, server, storage device, or other processing platform element. The given examples of such elements are considered as examples of what is more generally referred to herein as "processing devices". Figure 7 The cloud infrastructure 700 shown can represent at least a portion of a processing platform. Another example of such a processing platform is... Figure 8 The processing platform 800 shown is shown.

[0084] The processing platform 800 in this embodiment includes a part of the information processing system 100 and includes multiple processing devices, labeled 802-1, 802-2, 802-3, ... 802-K, which communicate with each other via network 804.

[0085] Network 804 includes any type of network, such as global computer networks, such as the Internet, WAN, LAN, satellite networks, telephone or cable networks, cellular networks, wireless networks such as Wi-Fi or WiMAX networks, or parts or combinations of these and other types of networks.

[0086] The processing device 802-1 in the processing platform 800 includes a processor 810 coupled to a memory 812.

[0087] Processor 810 includes a microprocessor, microcontroller, application-specific integrated circuit (ASIC), field-programmable gate array (FPGA) or other types of processing circuitry, and portions or combinations of such circuitry elements.

[0088] Memory 812 includes any combination of random access memory (RAM), read-only memory (ROM), or other types of memory. Memory 812 and other memories disclosed herein should be considered as illustrative examples of what is more generally referred to as a “processor-readable storage medium” storing executable program code of one or more software programs.

[0089] Articles of manufacture including such processor-readable storage media are considered illustrative embodiments. A given article of manufacture includes, for example, any of a memory array, a memory disk, or an integrated circuit containing RAM, ROM, or other electronic memory, or various other types of computer program products. The term "article of manufacture" as used herein should be understood to exclude transient propagation signals. Many other types of computer program products including processor-readable storage media can be used.

[0090] The processing device 802-1 also includes a network interface circuit 814 for interfacing the processing device with the network 804 and other system components, and may include a conventional transceiver.

[0091] Assume that the other processing devices 802 of the processing platform 800 are configured in a similar manner to the processing device 802-1 shown in the figure.

[0092] Similarly, the specific processing platform 800 shown in the figure is presented only as an example, and the information processing system 100 may include additional or alternative processing platforms, as well as a number of different processing platforms in any combination, each of which includes one or more computers, servers, storage devices or other processing devices.

[0093] For example, other processing platforms used to implement the illustrative embodiments may include different types of virtualization infrastructure to replace or supplement virtualization infrastructure that includes virtual machines. Such virtualization infrastructure illustratively includes container-based virtualization infrastructure configured to provide Docker containers or other types of LXC.

[0094] As another example, in some embodiments, a portion of a given processing platform may include converged infrastructure.

[0095] Therefore, it should be understood that in other embodiments, different arrangements of additional or alternative elements may be used. At least a subset of these elements may be implemented together on a common processing platform, or each such element may be implemented on a separate processing platform.

[0096] Furthermore, various other arrangements of computers, servers, storage products or devices, or other components are possible within the information processing system 100. These components can communicate with other elements of the information processing system 100 via any type of network or other communication medium.

[0097] For example, in illustrative embodiments, specific types of storage products that can be implemented in a given storage system implementing a distributed processing system include all-flash and hybrid flash storage arrays, scale-out all-flash storage arrays, scale-out NAS clusters, or other types of storage arrays. In illustrative embodiments, combinations of these and other storage products can also be used to implement a given storage system.

[0098] It should be emphasized again that the above embodiments are presented for illustrative purposes only. Many variations and other alternative embodiments can be used. Furthermore, the specific configurations of the system and device elements illustrated in the figures, and the associated processing operations, can be changed in other embodiments. Thus, for example, the specific types of processing devices, modules, systems, and resources deployed in a given embodiment, and their corresponding configurations, can be changed. Moreover, the various assumptions made above in describing the illustrative embodiments should also be considered illustrative and not as requirements or limitations of this disclosure. Many other alternative embodiments within the scope of the appended claims will become apparent to those skilled in the art.

Claims

1. A method comprising: The test generation system retrieves a sequence of log entries associated with a transaction identifier from the centralized log storage system in response to an alarm notification; The test generation system uses a log parsing engine to process the sequence of log entries to generate structured log data; as well as The test generation system uses a test case generation engine to generate test cases based on the structured log data. The test cases include test objectives, steps to reproduce the error condition, expected results, environment details, and test data. The method is implemented by at least one processing device, which includes a processor coupled to a memory.

2. The method according to claim 1, further comprising: Vector embedding is used to compare the generated test cases with existing test cases in the test case repository to identify similar test cases; as well as When no similar test cases are identified, the generated test cases are stored in the test case repository.

3. The method of claim 2, wherein comparing the generated test cases with existing test cases comprises: The existing test cases are converted into vector embeddings using a pre-trained language model; The embedding is stored in a vector database; as well as Calculate the cosine similarity score between the generated test cases and the existing test cases.

4. The method of claim 3, wherein the vector database comprises at least one of the following: Pinecone and Milvus.

5. The method according to claim 3, further comprising: The cosine similarity scores are categorized into percentage ranges to identify relevant matches.

6. The method of claim 1, wherein the alarm notification is received from an alarm system indicating erroneous conditions in the production environment.

7. The method of claim 1, wherein the alarm notification includes a predefined error pattern and a threshold for triggering the alarm.

8. The method of claim 1, wherein the centralized log storage system comprises at least one of the following: ELK Stack, Splunk, or Fluentd.

9. The method of claim 1, wherein retrieving the sequence of log entries in response to the alarm notification comprises: Log entries are identified based on the transaction ID or timestamp associated with the alert.

10. The method of claim 1, wherein using the log parsing engine to process the sequence of log entries comprises: Extract metadata, which includes date, time, component information, and environment details; as well as Extract log information, including error messages, stack traces, and test data.

11. The method of claim 10, wherein the metadata includes a log type selected from at least one of the following: entry point log, user action log, and failure point log.

12. The method of claim 1, wherein using the log parsing engine to process the sequence of log entries comprises: Use a transformer-based language model with Wordpiece segmentation.

13. The method of claim 1, wherein generating the test cases based on the structured log data using the test case generation engine comprises: Use a few-sample hint with domain-specific examples.

14. The method of claim 1, wherein the environment details include at least one of the following: operating system name, operating system version, database, database version, and hardware specifications.

15. The method of claim 1, further comprising using an error classification engine to classify the error condition to determine at least one of the following: error type, severity, and impact.

16. The method of claim 15, wherein the error type includes at least one of the following: related infrastructure, data problems, and corrupted functionality.

17. The method of claim 15, wherein the impact includes at least one of the following: service impact and transaction failure.

18. The method of claim 1, wherein the test cases are generated in a human-readable format.

19. A system comprising: At least one processing device, the at least one processing device including a processor coupled to a memory; The at least one processing device is configured to: The test generation system retrieves a sequence of log entries associated with a transaction identifier from the centralized log storage system in response to an alarm notification; The test generation system uses a log parsing engine to process the sequence of log entries to generate structured log data; as well as The test generation system uses a test case generation engine to generate test cases based on the structured log data. The test cases include test objectives, steps to reproduce the error condition, expected results, environment details, and test data.

20. A computer program product comprising a non-transitory processor-readable storage medium storing program code for one or more software programs, wherein the program code, when executed by at least one processing device, causes the at least one processing device to: The test generation system retrieves a sequence of log entries associated with a transaction identifier from the centralized log storage system in response to an alarm notification; The test generation system uses a log parsing engine to process the sequence of log entries to generate structured log data; as well as The test generation system uses a test case generation engine to generate test cases based on the structured log data. The test cases include test objectives, steps to reproduce the error condition, expected results, environment details, and test data.

Images