Dynamic testing using captured event sequences

Abstract

Methods and apparatus for regression testing. A first plurality of event sequences is generated by executing a plurality of test paths on a first computing environment. A test workload is constructed based on the first plurality of event sequences. A second plurality of event sequences is generated by executing the test workload on a second computing environment. One or more changes are identified by comparing the first plurality of event sequences and the second plurality of event sequences. And the second computing environment is reconfigured based on the one or more changes.

Description

Technical Field

[0001] This disclosure relates to regression testing, and more specifically, to simplifying the testing process by capturing a sequence of events and dynamically replaying the captured sequence of events in a new computing system. Background Technology

[0002] Regression testing is an important practice in the development and maintenance of production computing systems. The primary purpose of regression testing is to evaluate the system's functionality and ensure coordinated operation between all components, especially after modifications or updates have been made to the system (or any of its components). With the development of multi-service and distributed computing architectures, the complexity of regression testing has increased.

[0003] Distributed systems, spread across various machines or nodes, rely on the coordination of numerous components, applications, and services to ensure that each part functions as expected within this complex network. This environment demands well-designed test suites capable of detecting unexpected behavior, identifying bottlenecks, and evaluating resource utilization under varying workloads. While full-system testing is the industry standard today, significant challenges remain, including a lack of machine-driven automated testing mechanisms that can effectively simulate real-world user workloads.

[0004] Current end-to-end testing primarily relies on one of two approaches: heavy reliance on human testers to simulate workloads, or automation using random interactions. While the former provides a human feel, it can be time-consuming, prone to human error, and lacks the scalability and repeatability inherent in automated testing. On the other hand, while automated random interactions can elicit numerous interactions without human intervention, the random nature of these interactions may not accurately simulate real-world usage patterns. Therefore, the second approach leads to potentially underrepresented or incomplete test results, making the tests less effective at capturing real-world problems. Summary of the Invention

[0005] One embodiment described in this disclosure provides a method comprising: generating a first plurality of event sequences by executing multiple test paths on a first computing environment; constructing a test workload based on the first plurality of event sequences; generating a second plurality of event sequences by executing the test workload on a second computing environment; identifying one or more variations by comparing the first plurality of event sequences with the second plurality of event sequences; and reconfiguring the second computing environment based on the one or more variations.

[0006] Other embodiments of this disclosure provide a non-transient computer-readable medium containing computer program code and a system that, when executed by the operation of one or more computer processors, performs operations according to one or more of the methods described above, and the system includes one or more computer processors and one or more memories, the one or more memories containing one or more programs that, when executed by the one or more computer processors, perform operations according to one or more of the methods described above. Attached Figure Description

[0007] To gain a more detailed understanding of the features described above in this disclosure, a more specific description of the disclosure than briefly outlined above can be obtained by referring to the embodiments, some of which are illustrated in the accompanying drawings. However, it should be noted that the drawings illustrate only typical embodiments and should therefore not be considered limiting; other equally effective embodiments are envisioned.

[0008] Figure 1 An example computing environment is described for executing at least some of the computer code involved in performing creative methods.

[0009] Figure 2 An example architecture of a distributed system utilizing one embodiment of the present disclosure is described.

[0010] Figure 3 According to some embodiments of this disclosure, a timeline of events between three nodes having Lamport timestamps is depicted.

[0011] Figure 4 A Lamport tree of event sequences among three nodes is depicted according to some embodiments of this disclosure.

[0012] Figure 5 A timing diagram for replaying a test workload on a new computing system is depicted according to some embodiments of this disclosure.

[0013] Figure 6 Example methods for test event sequencing, re-execution, and analysis are described according to some embodiments of this disclosure.

[0014] Figure 7 This is a flowchart depicting an example method for dynamic testing according to some embodiments of this disclosure.

[0015] Figure 8 An example computing device configured to perform various aspects of this disclosure is depicted according to one embodiment.

[0016] For ease of understanding, the same reference numerals have been used where possible to designate the same elements that are common to the drawings. It should be understood that elements disclosed in one embodiment may be advantageously used in other embodiments without specific mention. Detailed Implementation

[0017] The description of various embodiments of this disclosure herein is for illustrative purposes and is not intended to be exhaustive or limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. The terminology used herein has been chosen to best explain the principles of the embodiments, practical applications of techniques found in the market or improvements to techniques, or to enable those skilled in the art to understand the embodiments disclosed herein.

[0018] This disclosure provides techniques for implementing machine-driven mechanisms to automate regression testing in multi-service distributed computing systems. Instead of relying on human-driven testing, the machine-driven mechanisms are specifically designed to simulate the inherent complexities associated with testing on these multi-service distributed infrastructures. This disclosure aims to enhance the accuracy, efficiency, and scalability of regression testing to ensure that these complex systems are thoroughly tested to meet the demands of real-world scenarios.

[0019] In one embodiment, a human tester can initiate various test cases within a first computing system (also referred to in some embodiments as a reference system or benchmark system). Each test case (referring to a condition (or set of conditions) used to determine whether the system functions correctly) may result in one or more unique test paths. Each test path may represent an actual sequence of code or instructions executed during runtime due to specific inputs (or a set of inputs) from the test case. Each test path may include a sequence of events or interactions within the system, and each interaction is captured and logged along with its corresponding logical / Lamport timestamp. In the above configuration, based on these logical / Lamport timestamps, a binary tree (also referred to in some embodiments as a Lamport tree) can be created for each test path, capturing the causal relationships and sequences of events or interactions associated with each test path.

[0020] In the above configuration, the test system can use these Lamport trees to create detailed test workloads and introduce them into a second computing system (also referred to in some embodiments as a new computing system) for re-execution. In the second computing system, sequences of events or interactions can be captured again, along with a new set of logical / Lamport timestamps. In some embodiments, when testing of the second computing system is complete, Lamport timestamps from the two computing systems can be compared to generate one or more Lamport time variations (e.g., micro-Lamport variations and / or macro-Lamport variations, as discussed in more detail below). Using Lamport time variations, in some embodiments, the test system can evaluate potential overfitting and / or underfitting between the two computing systems and perform further analysis to understand the performance and functionality of the new computing system.

[0021] Figure 1 An example computing environment is described for executing at least some of the computer code involved in performing creative methods.

[0022] Computing environment 100 includes examples of environments for executing at least some computer code involved in performing creative methods, such as test capture and redeployment code 180. In addition to test capture and redeployment code 180, computing environment 100 also includes, for example, a computer 101, a wide area network (WAN) 102, an end-user device (EUD) 103, a remote server 104, a public cloud 105, and a private cloud 106. In this embodiment, computer 101 includes a processor set 110 (including processing circuitry 120 and cache 121), communication infrastructure 111, volatile memory 112, persistent storage device 113 (including an operating system 122 and test capture and redeployment code 180, as described above), a peripheral device set 114 (including a user interface (UI) device set 123, storage device 124, and an Internet of Things (IoT) sensor set 125), and a network module 115. Remote server 104 includes a remote database 130. Public cloud 105 includes gateway 140, cloud orchestration module 141, host physical machine set 142, virtual machine set 143, and container set 144.

[0023] Computer 101 may take the form of a desktop computer, laptop computer, tablet computer, smartphone, smartwatch or other wearable computer, mainframe computer, quantum computer, or any other form of computer or mobile device now known or to be developed in the future, capable of running programs, accessing networks, or querying databases, such as remote database 130. As is well known in the computer technology of the art, and depending on the technology, the performance of a computer-implemented method may be distributed among multiple computers and / or across multiple locations. On the other hand, in the presentation of the computing environment 100 of this disclosure, the focus is placed on a single computer, specifically computer 101, to keep the presentation as simple as possible. Computer 101 may be located in the cloud, even if it is... Figure 1 It is not shown in the cloud. On the other hand, unless explicitly stated otherwise, computer 101 does not need to be in the cloud.

[0024] Processor set 110 includes one or more computer processors of any type now known or to be developed in the future. Processing circuitry 120 may be distributed across multiple packages, for example, multiple coordinated integrated circuit chips. Processing circuitry 120 may implement multiple processor threads and / or multiple processor cores. Cache 121 is memory located within the processor chip package(s) and is typically used for data or code that should be quickly accessed by the threads or cores running on processor set 110. Cache memory is typically organized into multiple levels depending on its relative proximity to the processing circuitry. Alternatively, some or all of the cache of the processor set may be located “off-chip.” In some computing environments, processor set 110 may be designed to work with qubits and perform quantum computing.

[0025] Computer-readable program instructions are typically loaded onto computer 101 to cause a series of operational steps to be performed by processor set 110 of computer 101, thereby implementing a computer-implemented method such that the instructions thus executed instantiate the method specified in the flowcharts and / or narrative descriptions of the computer-implemented method included in this document (collectively, the “inventive method”). These computer-readable program instructions are stored in various types of computer-readable storage media, such as cache 121 and other storage media discussed below. Processor set 110 accesses the program instructions and associated data to control and direct the execution of the inventive method. In computing environment 100, at least some of the instructions for performing the inventive method may be stored in test capture and redeployment code 180 in persistent storage 113.

[0026] Communication structure 111 is a signal transmission path that allows various components of computer 101 to communicate with each other. Typically, this structure consists of switches and electrically conductive paths, such as those forming buses, bridges, physical input / output ports, etc. Other types of signal communication paths, such as fiber optic communication paths and / or wireless communication paths, can be used.

[0027] Volatile memory 112 is any type of volatile memory known now or to be developed in the future. Examples include dynamic random access memory (RAM) or static RAM. Typically, volatile memory 112 is characterized by random access, but this is not necessary unless explicitly stated otherwise. In computer 101, volatile memory 112 is located in a single package and inside computer 101, but alternatively or additionally, volatile memory may be distributed across multiple packages and / or located externally to computer 101.

[0028] The persistent storage device 113 is any form of non-volatile storage device for a computer, now known or to be developed in the future. The non-volatility of this storage device means that the stored data is retained regardless of whether the computer 101 is powered or directly to the persistent storage device 113. The persistent storage device 113 may be a read-only memory (ROM), but typically at least a portion of the persistent storage device allows data to be written, deleted, and rewritten. Some familiar forms of persistent storage devices include hard disks and solid-state storage devices. The operating system 122 may take many forms, such as various known proprietary operating systems or operating systems using an open-source portable operating system interface type with a kernel. The code included in the test capture and redeployment code 180 typically includes at least some computer code related to performing the inventive method.

[0029] Peripheral device set 114 includes the peripheral device set of computer 101. Data communication connections between peripheral devices and other components can be implemented in various ways, such as Bluetooth connections, near field communication (NFC) connections, connections made via cables (such as Universal Serial Bus (USB) type cables), plug-in connections (e.g., Secure Digital (SD) cards), connections made via local area communication networks, and even connections made via wide area networks such as the Internet. In various embodiments, UI device set 123 may include components such as displays, speakers, microphones, wearable devices (such as goggles and smartwatches), keyboards, mice, printers, touchpads, game controllers, and haptic devices. Storage device 124 is an external storage device, such as an external hard drive, or a pluggable storage device, such as an SD card. Storage device 124 may be persistent and / or volatile. In some embodiments, storage device 124 may take the form of a quantum computing storage device for storing data in the form of qubits. In embodiments where computer 101 requires a large amount of storage (e.g., where computer 101 locally stores and manages a large database), such storage can be provided by peripheral storage devices designed to store very large amounts of data, such as a storage area network (SAN) shared by multiple geographically distributed computers. The IoT sensor set 125 consists of sensors that can be used in IoT applications. For example, one sensor could be a thermometer, and another sensor could be a motion detector.

[0030] Network module 115 is a collection of computer software, hardware, and firmware that allows computer 101 to communicate with other computers via WAN 102. Network module 115 may include hardware (such as a modem or Wi-Fi transceiver), software for packetizing and / or depacketizing data for transmission over a communication network, and / or web browser software for communicating data across the Internet. In some embodiments, the network control and network forwarding functions of network module 115 are performed on the same physical hardware device. In other embodiments (e.g., embodiments utilizing software-defined networking (SDN), the control and forwarding functions of network module 115 are performed on physically separate devices, such that the control functions manage multiple different network hardware devices. Computer-readable program instructions for performing the methods of the invention can typically be downloaded to computer 101 from an external computer or external storage device via a network adapter card or network interface included in network module 115.

[0031] WAN 102 is any wide area network (e.g., the Internet) that can communicate computer data over non-local distances using any technology known now or to be developed in the future for communicating computer data. In some embodiments, WAN 102 can be replaced and / or supplemented by a local area network (LAN) designed to communicate data between devices located in a local area, such as a Wi-Fi network. WANs and / or LANs typically include computer hardware such as copper transmission cables, optical fiber transmissions, wireless transmissions, routers, firewalls, switches, gateway computers, and edge servers.

[0032] End User Equipment (EUD) 103 is any computing system that can be used and controlled by an end user (e.g., an enterprise customer operating computer 101) and can take any of the forms discussed above regarding computer 101. EUD 103 typically receives helpful and useful data from the operation of computer 101. For example, assuming computer 101 is designed to provide advice to the end user, this advice would typically be sent from computer 101's network module 115 via WAN 102 to EUD 103. In this way, EUD 103 may display or otherwise present the advice to the end user. In some embodiments, EUD 103 may be a client device, such as a thin client, a thick client, a mainframe computer, a desktop computer, etc.

[0033] Remote server 104 is any computing system capable of providing at least some data and / or functionality to computer 101. Remote server 104 can be controlled and used by the same entity operating computer 101. Remote server 104 represents one or more machines that collect and store helpful and useful data for use by other computers, such as computer 101. For example, assuming computer 101 is designed and programmed to provide recommendations based on historical data, that historical data could be provided to computer 101 by a remote database 130 of remote server 104.

[0034] Public cloud 105 is any computing system that can be used by multiple entities, providing on-demand availability of computing system resources and / or other computing capabilities (especially data storage (cloud storage) and computing power) without direct active management by users. Cloud computing typically leverages resource sharing to achieve consistency and economies of scale. Direct and active management of the computing resources of public cloud 105 is performed by the computer hardware and / or software of cloud orchestration module 141. The computing resources provided by public cloud 105 are typically implemented by virtual computing environments running on individual computers in host physical machine set 142, which is all physical computers in and / or available to public cloud 105. Virtual computing environments (VCEs) typically take the form of virtual machines in virtual machine set 143 and / or containers in container set 144. It should be understood that these VCEs can be stored as images and can be transferred between individual physical machine hosts, either as images or after VCE instantiation. Cloud orchestration module 141 manages the transfer and storage of images, deploys new VCE instantiations, and manages active VCE deployment instantiations. Gateway 140 is a collection of computer software, hardware, and firmware that allow public cloud 105 to communicate via WAN 102.

[0035] Now, we will provide some further explanation about Virtualized Computing Environments (VCEs). A VCE can be stored as an "image." New active instances of a VCE can be instantiated from an image. Two familiar types of VCEs are virtual machines and containers. A container is a VCE that uses operating system-level virtualization. This refers to an operating system feature where the kernel allows the existence of multiple isolated user-space instances (called containers). From the perspective of the programs running within them, these isolated user-space instances typically behave like a real computer. Computer programs running on a regular operating system can utilize all the resources of that computer, such as connected devices, files and folders, network shares, CPU power, and quantifiable hardware capabilities. However, programs running within a container can only use the contents of the container and the devices allocated to the container, a characteristic known as containerization.

[0036] Private cloud 106 is similar to public cloud 105, except that its computing resources are available only to a single enterprise. Although private cloud 106 is depicted as communicating with WAN 102, in other embodiments, private cloud can be completely disconnected from the internet and accessed only via a local / private network. A hybrid cloud is a combination of multiple clouds of different types (e.g., private, community, or public cloud types), typically implemented by different vendors. Each of the multiple clouds remains a separate and independent entity, but the larger hybrid cloud architecture is combined through standardized or proprietary technologies that enable orchestration, management, and / or data / application portability. In this embodiment, both public cloud 105 and private cloud 106 are part of a larger hybrid cloud.

[0037] Figure 2 An example architecture of a distributed system 200 utilizing one embodiment of the present disclosure is depicted. In the illustrated example, system 200 includes one or more client devices 205 that are communicatively coupled to a cluster of nodes 210-1, 210-2, and 210-3, a shared database 270, and middleware 255 for coordinating global Lamport timestamps and recording sequences of events or interactions.

[0038] In the illustrated example, client device 205 represents the entry point into distributed system 200 and can be any computing device capable of communicating with one or more nodes 210-1, 210-2, and 210-3 in the system to initiate tests. In some embodiments, client device 205 may correspond to conventional computing devices such as laptops, desktops, tablets, and smartphones. In some embodiments, client device 205 may correspond to dedicated devices such as Internet of Things (IoT) sensors, embedded systems, and web applications, provided they have the necessary software and networking capabilities to interface with the distributed system. In some embodiments, client device 205 may each include one or more CPUs, one or more memories, one or more storage devices, one or more network interfaces, and / or one or more I / O interfaces, wherein the CPU can retrieve and execute programming instructions stored in the memory, and store and retrieve application data residing in the storage device. As shown, client device 205 is connected to node 210 via network connection 245, which enables seamless communication and interaction. Network connection 245 allows client device 205 to initiate tests and interact with the distributed system to collect information needed for further analysis (e.g., Lamport trees, interaction sequences with Lamport timestamps).

[0039] In the illustrated example, nodes 210-1, 210-2, and 210-3 in distributed system 200 can be any type of computing device, ranging from traditional servers in a data center to cloud-based virtual machines, edge devices, and / or workstations. These nodes 210-1, 210-2, and 210-3 collaborate with each other to provide seamless service to end users. In the illustrated example, nodes 210-1, 210-2, and 210-3 are connected to a shared database 270 and middleware 230 via network connection 250. In some embodiments, network connections 245 and 250 may include or correspond to any combination of a wide area network (WAN), local area network (LAN), Internet, intranet, or any suitable communication medium that may be available, and may include wired, wireless, or a combination of wired and wireless links. Network connections 245 and 250 can provide connectivity for various systems, components, or resources within distributed system 200 and can be implemented using protocols such as Transmission Control Protocol (TCP) and / or Internet Protocol (IP). In some embodiments, client device 205, node 210, shared database 270, and middleware 255 may be local to each other (e.g., within the same local network and / or the same hardware system) and communicate with each other using any suitable local communication medium, such as a local area network (LAN) (including wireless LAN (WLAN)), hardwired, wireless link, or intranet, etc. In some embodiments, one or more of client device 205, node 210, shared database 270, and middleware 255 may be remote to each other (e.g., located in different geographical locations) and communicate with each other using any suitable communication medium, such as a wide area network (WAN) or the Internet.

[0040] In some embodiments, one or more of the client devices 205 can initiate one or more test cases and provide relevant inputs or parameters for one or more tests. Upon receiving a test command and its associated parameters, nodes 210-1, 210-2, and 210-3 can collaboratively execute test operations. In the illustrated example, each node (e.g., 210-1) contains three components: a processor (e.g., 215-1), a memory (e.g., 220-1), and a storage device (e.g., 225-1). In some embodiments, the memory can be any type of volatile memory, such as dynamic random access memory (RAM) or static RAM. As shown, the memory (e.g., 220-1) acts as a temporary storage medium for the active execution of various components within the node. Specifically, in the illustrated example, the memory contains program code for application components (e.g., 230-1), test components (e.g., 235-1), and a Lamport clock (e.g., 240-1). During operation, the processor (e.g., 215-1) can access and execute programming instructions stored in memory (e.g., 220-1), as well as store and retrieve application data residing in storage device (e.g., 225-1).

[0041] In some embodiments, the application component (e.g., 230-1) may contain software (e.g., program code) that provides the primary functions or services that the node is designed to provide. The application component (e.g., 230-1) may process data, handle primary tasks, and manage interactions with other components within the distributed system (e.g., other nodes or client devices). The application component (e.g., 230-1) may ensure that the node provides the services it is intended to provide to users or other components.

[0042] In some embodiments, the testing component (e.g., 235-1) may include program instructions specifically designed to evaluate the performance, reliability, and functionality of the application component (e.g., 230-1) when executing test cases. The testing component (e.g., 235-1) may execute predefined test cases, simulate user behavior, and generate corresponding test results. In some embodiments, the test results may be further analyzed to identify potential problems, bottlenecks, or defects in the application component.

[0043] In some embodiments, a local Lamport clock (e.g., 240-1) can use the Lamport clock algorithm to sort events in the distributed system. The local Lamport clock (e.g., 240-1) can generate a Lamport timestamp for each event or interaction occurring within a node (e.g., 210-1) and maintain its own local timestamp (e.g., stored in storage device 225-1). When a node (e.g., 210-1) communicates with other nodes (e.g., 210-2) within the distributed system, it can send its current local timestamp along with the message. Upon receiving a timestamped message from another node, the receiving node (e.g., 210-2) can update its local timestamp to the maximum of its current timestamp and the received timestamp (e.g., from 210-1) plus one.

[0044] In the example shown, middleware 255 includes a global Lamport clock 260 and an event sequencer 265. In some embodiments, the global Lamport clock 260 may communicate with each node (e.g., 210-1, 210-2, and 210-3) via network connection 250. During test case execution, the global Lamport clock 260 may synchronize with the local Lamport clock within each node to generate a uniform and system-wide Lamport timestamp map (e.g., ...). Figure 3 (As shown in the diagram). This diagram visually represents the entire sequence of events in the system, with each event labeled with a corresponding Lamport timestamp. For example, in this diagram, when node 210-1 sends a message to node 210-2, node 210-1's local Lamport timestamp is marked as "1", and after receiving the message, node 210-2's local Lamport timestamp is updated to "2". See below for reference. Figure 3 and Figure 5 Let's discuss more details.

[0045] In some embodiments, the event sequencer 265 can be configured to capture sequences of events or interactions that occur during the execution of a particular test case (e.g., leading to a unique test path). In some embodiments, by monitoring these events and updated Lamport timestamps within each node, the event sequencer 265 can construct a binary tree (also referred to in some embodiments as a Lamport tree) for each test path, providing a structured representation of these events or interactions and their causal order. In some embodiments, for different test paths, their respective binary trees can be edited, pruned, and / or combined to create test workloads that include a variety of test scenarios and interaction patterns. In some embodiments, a large number of such path records are generated by representing the test path for each interaction as a single Lamport tree. When constructing test workloads, in some embodiments, these unique or dense paths can be identified and pruned to improve test efficiency. In some embodiments, similar paths can be compressed to eliminate redundancy. Various techniques can be used to edit and combine binary trees for different test paths to create custom and reproducible test workloads. In some embodiments, custom and reproducible workloads are designed to more accurately and effectively stress test new computing systems, reflecting the complexity of real-world user behavior. After defining the test workload, in some embodiments, the test workload can be replayed in a new computing environment. Replaying the test workload can simulate real-world user behavior to evaluate the performance of the new computing system under various conditions, all without requiring (or reducing) human intervention or manual testing.

[0046] In some embodiments, the Lamport tree constructed for each test path may capture only sequences of active events (e.g., operations such as "send," "read," and "update"), omitting passive events (e.g., "receive"). In distributed systems, active events can originate from internal system operations or explicit actions initiated by users. Therefore, active events are often reproducible because they are based on application logic or direct user behavior. On the other hand, passive events such as "receive" can be affected by various external factors (e.g., network latency, congestion, system settings, the operational state of other processes), and reproducing an accurate sequence of passive events when replayed in a new environment can be difficult. By focusing on sequences of active events, the Lamport tree ensures that it captures reproducible sequences of events that can be re-executed in a new computing system. In some embodiments, re-execution of sequences of events or interactions can simulate real-world user behavior without (or with reduced) human intervention and result in a new Lamport timestamp graph (e.g., ...). Figure 5(As shown). During re-execution, the event sequence remains consistent, but nodes within the new computational system may derive different Lamport timestamps than those in the reference computational system. The difference in Lamport timestamps between the two graphs can provide further insights into behavioral and performance differences between the two computational systems. In some embodiments, this difference can be used as a diagnostic tool to help identify potential overfitting / underfitting between the two systems. References will follow below. Figure 6 Let's discuss this in more detail.

[0047] Figure 3 A timeline 300 depicting an event sequence among three nodes with Lamport timestamps is described according to some embodiments of this disclosure.

[0048] The illustrated timeline 300 represents three nodes within a first distributed system, designated N1 (e.g., node 301), N2 (e.g., node 302), and N3 (e.g., node 303). When test cases are initiated in the first distributed system, the three nodes N1, N2, and N3 interact with each other in a sequence of events to develop a unique test path, where each interaction uses a local Lamport clock (e.g., ...). Figure 2 The timestamps for events (240-1, 240-2, and 240-3) are performed. The sequence of events or interactions and their corresponding Lamport timestamps are captured and recorded in the illustrated time graph 300.

[0049] In this illustrated timeline, N1 first reads a value (e.g., reads x = 10 from memory or storage device), causing its Lamport clock to increment to 1 (as shown in Lamport timestamp 305). In step two, N1 updates the value based on various application logic (e.g., updates x = 15), and its Lamport clock changes to 2 (as shown in Lamport timestamp 310). In step three, N1 communicates the update to N2 (e.g., sends x = 15) (as shown by arrow 320), and N1's Lamport clock further increments to 3 (as shown in Lamport timestamp 315). After receiving the update from N1 (e.g., receiving x = 15), N2 adjusts its Lamport clock to 4 (as shown in Lamport timestamp 330), which is derived from the maximum of its own clock value (0) and the timestamp (3) received from N1 plus one. Simultaneously with step three, N3 makes its own update based on various logics (e.g., updating y = 10), setting its Lamport clock to 1 (as shown in Lamport timestamp 325). In step four, N3 sends a message to N2 about the update (e.g., sending y = 10) (as shown by arrow 340), and N3's Lamport clock is further incremented to 2 (as shown in Lamport timestamp 335). After receiving the update from N3 (e.g., receiving y = 10), N2 adjusts its Lamport clock to 5 (as shown in Lamport timestamp 345), which is derived from the maximum of its own clock value (4) and the timestamp (2) received from N3 plus one. In step five, N2 reads the updated value of x (e.g., reading x = 15), and its Lamport clock changes to 6 (as shown in Lamport timestamp 350). In step six, N2 reads the updated value of y from N3 (e.g., reads y = 10), and its Lamport clock changes to 7 (as shown in Lamport timestamp 355). In step seven, N2 performs the calculation of the updated values ​​x and y (e.g., calculates y = x + y = 25), incrementing its Lamport clock to 8 (as shown in Lamport timestamp 360). In step eight, N2 sends a message to N3 (as shown by arrow 370), updating N3 with the value (e.g., sending y = 25), and incrementing its Lamport clock to 9 (as shown in Lamport timestamp 365). After receiving the message about the update (e.g., y = 25), N3 adjusts its Lamport clock to 10 (as shown in Lamport timestamp 375), which is derived from the maximum of its own clock value (2) and the timestamp (9) received from N1 plus one.

[0050] The illustrated timeline captures a sequence of events or interactions between three nodes within a distributed system. The illustrated system is discussed for clarity of concept. In some embodiments, the distributed system may include any number of nodes that interact within the scope of a triggered test case (including one).

[0051] Figure 4 Lamport tree 400, representing a sequence of events between three nodes, is depicted according to some embodiments of this disclosure.

[0052] The Lamport tree 400 illustrated is captured in the first computing system (e.g., Figure 3 The causal relationships and sequence of events or interactions executed in the test path (as shown). In some embodiments, the Lamport tree constructed for the test path can capture the causal relationships and sequence of all types of events or interactions, including both active and passive events.

[0053] In the illustrated example, only active events or interactions (e.g., operations such as "read", "update", "send", and "compute") are recorded. Passive events or interactions between the three nodes (e.g., operations such as "receive") are omitted. By focusing on the sequence of active actions, the Lamport tree ensures that it captures reproducible sequences of events that can be re-executed in a new computing system. When constructing the Lamport tree, the position of actions indicates their temporal relationship. Actions that occur concurrently, without an explicit order, are placed at the same level in the tree. For example, node 420, which depicts N3 updating its value (e.g., updating y = 10) and incrementing its Lamport clock to 1 (Lamport clock = 1), is placed at the same level as leaf node 415, which depicts N1 sending a message to N2 (e.g., sending x = 15) and incrementing its Lamport clock to 3 (Lamport clock = 3). Conversely, actions that occur sequentially are placed vertically in the tree. For example, the action of step one involves N1 reading a value (e.g., reading x = 10) and incrementing its Lamport clock to 1 (Lamport clock = 1). The action of step one is represented by root node 405. Step one is followed by the action of step two, where N1 updates the value (e.g., updates x = 15) and increments its Lamport clock to 2 (Lamport clock = 2). The action of step two is represented by node 410 and is placed below root node 405. Similarly, the action of step four follows the action of step three (represented by node 420), which involves N3 sending a message to N2 about the update (e.g., updating y = 10) and further incrementing its Lamport clock to 2 (Lamport clock = 2). The action of step four is represented by node 425 and is placed at a lower level below node 420. After the action of step four, the action of step five is represented by node 430 (located below node 425), which includes N2 reading the updated value from N1 (e.g., reading x = 15) and changing its Lamport clock to 6 (Lamport clock = 6). Step six is ​​represented by node 435 (located below node 430), which involves N2 reading the updated value from N3 (e.g., y = 10) and setting its Lamport clock to 7 (Lamport clock = 7). Node 440 represents the action of step seven, where N2 adds the updated x and y together (e.g., y = x + y = 25), incrementing its Lamport clock to 8 (Lamport clock = 8). Since the action of step seven occurs after the action of step six, node 440 is placed below node 435. Leaf node 445 represents the action of step eight, where N2 sends a message to N3, updating the value (e.g., y = 25) to N3 and incrementing its Lamport clock to 9 (Lamport clock = 9).Since step eight occurs after step seven, leaf node 445 is located below node 440.

[0054] The illustrated example depicts the construction of a Lamport tree for a test path involving interactions between three different nodes. The illustrated Lamport tree is discussed for conceptual clarity. In some embodiments, the distributed system may include any number of nodes interacting within the scope of the test path (including one).

[0055] In some embodiments, multiple test cases can be initiated in the first distributed computing system. For each unique test path that appears, a corresponding Lamport tree is generated to capture the sequence of events or interactions occurring within the first distributed computing system. In the illustrated example, a test workload for the new computing system is generated based on a single Lamport tree 400. The illustrated test workload is discussed for conceptual clarity. In some embodiments, test workloads for the new computing system can be generated based on more than one Lamport tree 400. In some embodiments, Lamport trees developed from different test paths can be edited, pruned, or aggregated to create test workloads. When the test workload is executed in the new computing system, it can simulate real-world usage behavior and generate corresponding test results for the new computing system.

[0056] Figure 5 A timing diagram 500 for replaying a test workload in a new computing system is depicted according to some embodiments of this disclosure.

[0057] In the illustrated example, based on a single Lamport tree (e.g., Figure 4The test workload for the new computing system is generated in time map 500. In some embodiments, the test workload for the new computing system may be generated based on more than one Lamport tree. In some embodiments, the Lamport tree captures a sequence of events or interactions captured within the Lamport tree for each test path during execution in the first computing system. In the illustrated example, the test workload is re-executed in the second computing system. The second computing system includes three nodes, depicted in the illustrated time map 500 as N1 (e.g., node 401), N2 (e.g., node 402), and N3 (e.g., node 403). However, this re-execution results in the nodes being marked with different Lamport timestamps compared to their counterparts in the first computing system. For example, node N1 in the second computing system is marked with a Lamport value of 4, node N2 with a Lamport value of 10, and node N3 with a Lamport value of 8. The differences in timestamps may be caused by various factors such as network latency, different system configurations and processing speeds, interference from other concurrent operations in the new system, or interactions with external systems or services. More details are discussed below.

[0058] In the illustrated example, the re-execution follows a sequence of events captured by the Lamport tree 400. This sequence begins with N1, which reads a value (e.g., reads x = 10) and increments its Lamport clock to 1 (as shown by Lamport timestamp 505). In step two, N1 performs an update operation (e.g., updates x = 15) and increments its Lamport clock to 2 (as shown by Lamport timestamp 510). Subsequently, in step three, N1 sends a message to N2 (as shown by arrow 520) indicating its most recent update (e.g., sending x = 15), setting its Lamport clock to 3 (as shown by Lamport timestamp 515). However, due to factors such as network latency or system processing delays, N2 does not immediately receive this message. Simultaneously with step three, N3 begins its operation by updating a different value y (e.g., updating y = 10) and incrementing its Lamport clock to 1 (as shown by Lamport timestamp 530). Following the actions in step three, N3 then sends a message to N2 in step four, providing details about the update (e.g., sending y = 10) and setting its Lamport clock to 2 (as shown by Lamport timestamp 535). The message from N3, unlike the message from N1, is not delayed. Therefore, at the receiving end, N2 receives the message from N3 first (e.g., receiving y = 10). Given the Lamport clock rules and the timestamp of the incoming message, N2 sets its clock to 3 (as shown by Lamport timestamp 545). In step five, N2 reads the value x. However, since the message from N1 regarding the update (e.g., updating x = 15) is delayed, N1 reads the old value of x (e.g., reading x = 10) and increments its clock to 4 (as shown by Lamport timestamp 550). In step six, N2 reads the updated value of y (e.g., reading y = 10) and increments its clock to 5 (as shown by Lamport timestamp 550). In step seven, N2 calculates the value y as the sum of the old value of x and the updated value of y (e.g., calculates y = x + y = 20), and sets its clock to 6 (as shown by Lamport timestamp 560). Here, due to the delayed message from N1 to N2, the calculation is performed before the delayed message is received, which thus disrupts the overall event sequence. In step eight, N2 sends a message to N3 (as shown by arrow 570), updates N3 with the value (e.g., sends y = 20) and increments its Lamport clock to 7 (as shown by Lamport timestamp 565). N3 increments its clock to 8 (as shown by Lamport timestamp 575) upon receiving the message.After N2 completes its computation and even communicates with N3, a delayed message from N1 to update the value of x (e.g., sending x = 15) (as indicated by arrow 520) is received, and N2's Lamport clock is further incremented to 9 (as indicated by Lamport timestamp 525). This sequence deviation caused by external factors such as network conditions and / or system configuration results in changes to the Lamport timestamp. For example, nodes N1, N2, and N3 in the first computing system are marked with Lamport values ​​of 3, 9, and 10, respectively. However, when the same sequence of events is replayed in the second computing system, the external factors disrupt the timing and causality of the events. As a result, nodes N1, N2, and N3 in the second computing system have changed Lamport values ​​of 3, 9, and 8, respectively.

[0059] When a specific error correction mechanism is triggered, the changes in the Lamport timestamps can increase further. In the illustrated example, N1 has an error correction mechanism for monitoring delays in message transmission. When N1 determines that the delay exceeds a predetermined threshold, N1 takes corrective action by resending a message with an updated value for x to N2 (e.g., sending x = 15) (as shown by arrow 585). This resending process causes an additional increment to N1's clock, changing it to 4 (as shown by Lamport timestamp 580). When N2 receives the resent message, it adjusts its clock to 10 (as shown by Lamport timestamp 590), which is derived from the maximum of its own clock value (9) and the timestamp (4) received from N1, plus one. As a result, nodes N1, N2, and N3 in the second computing system are labeled with Lamport values ​​of 4, 10, and 8, respectively. Error correction mechanisms are crucial for maintaining data integrity and ensuring secure message delivery, but can introduce additional complexity and variability into system testing.

[0060] In the illustrated example, a test workload for the second computing system is generated based on a Lamport tree 400 that captures sequences of events or interactions in the first computing system. The generation of the test workload from a single Lamport tree, as shown, is discussed for conceptual clarity. In some embodiments, multiple Lamport trees can be generated (and stored in memory), each representing a test path developed within the first computing system. In some embodiments, the test workload can be a combination of various trees. In some embodiments, when constructing the test workload, certain unique paths (or leaf nodes) can be pruned. These paths (or leaf nodes) correspond to unique test paths that are not typically shared between different systems designed for specific functions, to improve efficiency during testing. In some embodiments, similar or redundant paths (or leaf nodes) can be compressed or merged to ensure that the test workload is both tailored to test requirements and reproducible across different computing environments.

[0061] Following re-execution within the second computing system, in some embodiments, the newly generated Lamport timestamps across the three nodes can serve as valuable system analysis metrics. These timestamps can be further analyzed when compared to timestamps from the first computing system to identify potential problems, bottlenecks, or defects within the second computing system. In some embodiments, changes in Lamport timestamps between the two systems can help identify whether one system is overfitting or underfitting the other. In some embodiments, when multiple Lamport trees are replayed in the second computing system, changes in Lamport timestamps can be analyzed at both macro and micro levels. In some embodiments, micro Lamport changes can refer to the timestamp deviation of a particular test path when its corresponding Lamport tree is replayed. In some embodiments, macro Lamport changes can refer to the cumulative Lamport timestamp deviation of all test paths replayed in the second computing system, providing a system-wide view of how the second system performs relative to the collective test workload.

[0062] Figure 6 Example methods 600 for test event sequencing, re-execution, and analysis are described according to some embodiments of this disclosure. In some embodiments, method 600 may be executed by one or more computing devices, such as... Figure 1 The computer 101 shown Figure 2 Client device 205, node device 210, middleware 255 and / or such Figure 8 The computing device 800 shown.

[0063] The method begins at box 605, where one or more designed test paths are executed within a first computing system (also referred to in some embodiments as a first computing environment). In some embodiments, the test paths may be designed by human testers based on various scenarios to reflect real-world operations and workflows. In some embodiments, the first computing system may refer to a system consisting of one or more nodes (e.g., Figure 3 The system is a distributed system consisting of nodes 301, 302, and 303, and each test path can represent a sequence of interactions or events between these nodes to provide the functionality of the design. During execution, each interaction is timestamped according to the Lamport clock algorithm.

[0064] At box 610, for each test path already executed within the first computing system, generate the first Lamport timeline (e.g., ...). Figure 3 As shown above, the first Lamport time map includes timestamps for each interaction, reflecting the causal relationships between interactions or events within the first computing system.

[0065] At box 615, based on the Lamport timeline, the test equipment creates a binary tree (also referred to as a Lamport tree in some embodiments) for each test path (e.g., ...). Figure 4 As shown above, the binary tree reflects the causal relationships and order of events or interactions within each test path, providing a clear roadmap for subsequent re-execution in different computing systems.

[0066] At box 620, the test workload is generated. As described above, in some embodiments, the test workload may include any number of Lamport trees (including one). When generating the test workload, certain unique but non-essential paths (or leaf nodes) may be pruned to improve efficiency during testing. In some embodiments, Lamport trees for different test paths may be compressed (e.g., by removing similar or redundant leaf nodes from different trees). This approach ensures that the final test workload is tailored and reproducible across different computing environments.

[0067] At box 625, the test workload is re-executed within a second computing system (also referred to in some embodiments as a second computing environment). In some embodiments, the second computing system may refer to a system consisting of one or more nodes (e.g., Figure 2A distributed system consisting of 210-1, 210-2, and 210-3. Although the test workload provides a sequence of events or interactions defined and derived from the original Lamport tree, in some embodiments, new executions may cause changes to the Lamport timestamps. Various internal factors (such as system configuration affecting processing speed, interference from other concurrent operations) or external factors (such as network latency) may cause these changes.

[0068] At box 630, generate a second Lamport timeline (e.g.) Figure 5 (As shown). As described above, within the second Lamport timemap, each interaction is timestamped according to the Lamport clock algorithm. The new Lamport timestamps capture the causal relationships between interactions observed during re-execution within the second computing system.

[0069] At box 635, one or more changes in Lamport timestamps are calculated by comparing the Lamport time graphs of the first and second computing systems. In some embodiments, changes can be determined by comparing the timestamps of corresponding events in the two graphs. In some embodiments, changes can be categorized into two types: macro Lamport changes and micro Lamport changes. In some embodiments, as described above, micro Lamport changes can refer to changes in timestamps obtained by summing the individual timestamp differences across all events in a particular test path. In some embodiments, micro Lamport changes can represent collective timestamp deviations within a single test path. In some embodiments, micro Lamport changes can be used to identify whether there are significant changes at any point during the re-execution of a particular test path in the second computing system. In some embodiments, as described above, macro Lamport changes can refer to the overall change in Lamport timestamps between the two systems. In some embodiments, when multiple test paths are re-executed (e.g., multiple test paths are aggregated to generate a test workload), macro Lamport changes can be calculated by summing the individual timestamp differences across all test paths. In some embodiments, macro Lamport changes can provide a system-wide view of the differences in executing test workloads within the two computing systems. In some embodiments, each of the two types of variations can be used to evaluate performance and identify potential overfitting or underfitting between the two systems.

[0070] At box 640, the calculated changes are compared to determine whether one or more overfitting criteria are met. In some embodiments, overfitting criteria may involve setting a specific threshold for either micro Lamport change or macro Lamport change. When these overfitting criteria are met (e.g., micro Lamport change = 0, or macro Lamport change = 0), the method proceeds to box 645, where both computational systems are determined to be overfitted. In some embodiments, overfitting may indicate that the functionality of the two systems is considered equivalent, and that causality between events is maintained even when performed in different systems. In some embodiments, functional equivalence may refer to the two systems producing the same output when receiving the same input. In some embodiments, overfitting may imply that the two computational systems behave almost identically in terms of event ordering and causality. Specifically, in some embodiments, when the overfitting criterion is set such that the macroscopic Lamport change equals zero, overfitting can indicate that interactions or events are processed in the same order across all executed test paths in both systems, regardless of any differences in their configurations or network conditions, and that the causal relationships between interactions or events in the two systems remain consistent, and, or in either system, there are no significant bottlenecks, defects, or problems that would cause Lamport timestamps to diverge. In some embodiments, the overfitting threshold can be set such that the macroscopic Lamport change equals a small, non-zero value. In this configuration, overfitting can imply that the two computing systems are functionally equivalent and behave similarly, even with minor differences in event ordering and causality. Additionally, in some embodiments, the macroscopic Lamport change can be used to determine the average reliability and resilience of the system for a set of test cases. By setting a small, non-zero value as the threshold, the reliability of the second computing system can be evaluated. For example, if the overfitting criterion is met (e.g., the macroscopic Lamport change is below a small, non-zero value), it can imply that the second computing system has a high degree of reliability and resilience because, despite minor variations, the second computing system is still able to maintain similar causal relationships and event ordering as the first computing system.

[0071] If these overfitting criteria are not met, the method proceeds to box 650, where an underfitting analysis is performed. Within the underfitting analysis, one or more underfitting criteria can be established to determine whether the difference in Lamport timestamps between two systems is large enough to conclude that they are functionally dissimilar and therefore underfit. In some embodiments, the underfitting criteria may involve setting a specific threshold (e.g., a non-zero value) for either micro or macro Lamport variation to indicate acceptable Lamport timestamp variations that do not affect the overall functionality or performance of the system. If the underfitting criteria are met (e.g., macro Lamport variation exceeds a non-zero value), it can indicate that the two systems are functionally dissimilar and that the second computational system is unsuitable or unreliable for a given test workload.

[0072] Figure 7 This is a flowchart of an example method for dynamic testing according to some embodiments of the present disclosure.

[0073] Method 700 begins at box 705, where the system (e.g., Figure 1 Computer 101 or Figure 8 The computing device 800 generates a first plurality of event sequences (e.g., by executing multiple test paths on a first computing environment) Figure 3 (300). In some embodiments, each corresponding event sequence in the first plurality of event sequences may include a timestamp order (e.g., 300). Figure 3 (305-375), each timestamp (e.g.) Figure 3 (305-375) corresponds to the corresponding test operation performed during the execution of the corresponding test path on the first computing environment.

[0074] At box 710, the system constructs a test workload based on the first sequence of multiple events (e.g., Figure 4 As shown, the test workload is generated based on a single Lamport tree 400. In some embodiments, the test workload may include one or more binary trees (e.g., Figure 4 The Lamport tree (400) represents a binary tree, with each corresponding binary tree corresponding to one of a plurality of test paths. In some embodiments, the test workload may include one or more binary trees, each corresponding binary tree corresponding to a combination of at least two of the plurality of test paths.

[0075] At box 715, the system generates a second plurality of event sequences (e.g., by executing the test workload on a second computing environment) Figure 5 (500). In some embodiments, each corresponding event sequence in the second plurality of event sequences may include a timestamp (e.g., 500). Figure 5 The order of 505-590, each timestamp (e.g., Figure 5 (505-590) corresponds to the corresponding test operation performed during the execution of the corresponding binary tree on the second computing environment.

[0076] At box 720, the system identifies one or more variations (such as...) by comparing a first plurality of event sequences with a second plurality of event sequences. Figure 6 (as shown in box 635). In some embodiments, one or more variations may include macroscopic variations, wherein the macroscopic variations may include the difference between timestamps associated with a first plurality of event sequences and timestamps associated with a second plurality of event sequences. In some embodiments, one or more variations may include microscopic variations, wherein the microscopic variations may include the difference between timestamps associated with a first event sequence from the first plurality of event sequences and timestamps associated with a second event sequence from the second plurality of event sequences. In some embodiments, when executed on a first computing environment, the first event sequence may correspond to a first test path among a plurality of test paths, and when executed on a second computing environment, the second event sequence may correspond to the first test path. In some embodiments, method 700 may further include comparing one or more variations with one or more overfitting criteria (such as... Figure 6 (as shown in box 640), and when it is determined that one or more overfitting criteria are met, it is determined that the causal relationship between the test events in the first plurality of event sequences is maintained in the second plurality of event sequences (as shown in box 640), and when it is determined that one or more overfitting criteria are met, the causal relationship between the test events in the first plurality of event sequences is maintained in the second plurality of event sequences (as shown in box 640). Figure 6 (as shown in box 645).

[0077] At box 725, the system reconfigures the second computing environment based on one or more changes.

[0078] Figure 8 An example computing device configured to perform various aspects of this disclosure is depicted according to one embodiment. Although depicted as a physical device, in some embodiments, computing device 800 may be implemented using a virtual device and / or across multiple devices (e.g., in a cloud environment). Computing device 800 may be embodied as any computing device, such as Figure 1 The computer 101 shown Figure 2 The client device 205, node device 210, and / or middleware 255 are shown.

[0079] As shown in the figure, computing device 800 includes a CPU 805, memory 810, storage device 815, one or more network interfaces 825, and one or more I / O interfaces 820. In the illustrated embodiment, CPU 805 retrieves and executes programming instructions stored in memory 810, and stores and retrieves application data residing in storage device 815. CPU 805 typically represents a single CPU and / or GPU, multiple CPUs and / or GPUs, a single CPU and / or GPU with multiple processing cores, etc. Memory 810 is typically included to represent random access memory. Storage device 815 can be any combination of disk drives, flash-based storage devices, etc., and can include fixed and / or removable storage devices, such as fixed disk drives, removable memory cards, caches, optical storage devices, network-attached storage (NAS), or storage area networks (SANs).

[0080] In some embodiments, I / O devices 835 (such as a keyboard, monitor, etc.) are connected via I / O interface 820. Furthermore, via network interface 825, computing device 800 can be communicatively coupled to one or more other devices and components (e.g., via a network, which may include the Internet, a local area network, etc.). As shown, CPU 805, memory 810, storage device 815, network interface 825, and I / O interface 820 are communicatively coupled via one or more buses 830.

[0081] In the illustrated embodiment, the memory 810 includes a data collection component 850, a change calculation component 855, an overfit / underfit analysis component 860, and a reporting component 865.

[0082] Although depicted as discrete components for conceptual clarity, in some embodiments, the operation of the depicted components (and other components not shown) may be combined or distributed across any number of components. Furthermore, although depicted as software residing in memory 810, in some embodiments, the operation of the depicted components (and other components not shown) may be implemented using hardware, software, or a combination of hardware and software.

[0083] In one embodiment, the data collection component 850 can collect timestamp data for different events or interactions in the first and second computing systems (e.g., ...). Figure 3 305-375 and Figure 5 (505-590). In some embodiments, the first computing system (also referred to in some embodiments as a reference system or baseline system) may refer to a system that executes an initial test path and generates a corresponding binary tree (also referred to in some embodiments as a Lamport tree). In some embodiments, the first system may refer to a system consisting of one or more nodes (e.g., Figure 3A distributed system consisting of nodes 301, 302, and 303. In some embodiments, the second system may refer to a system that re-executes the test workload generated from the binary tree of the first system. In some embodiments, the second system may refer to a system consisting of one or more nodes (e.g., Figure 5 A distributed system consisting of 501, 502, and 503.

[0084] In one embodiment, the change calculation component 855 can calculate the change in Lamport timestamps between two systems. In some embodiments, two types of change can be calculated: macro Lamport change and micro Lamport change. Micro Lamport change can refer to the change in timestamps obtained by summing the individual timestamp differences of all events in a particular test path. Conversely, macro Lamport change can refer to the overall change in timestamps obtained by summing the individual timestamp differences across all test paths.

[0085] In one embodiment, the overfitting / underfitting analysis component 860 can determine whether overfitting has occurred between two systems by comparing the computational changes with overfitting criteria from a database. When the overfitting criteria are met, it can be determined that the two systems are overfitted. As described above, the overfitting criteria can involve setting a specific threshold for either micro-Lamport changes or macro-Lamport changes. For example, in some embodiments, the overfitting criteria can be set such that the macro-Lamport change is equal to zero. In this configuration, overfitting can indicate that across all executed test paths, despite any differences in their configurations or network conditions, interactions or events are processed in the same order within both systems, causal relationships between interactions or events are consistently maintained in both systems, and / or there are no significant bottlenecks, defects, or problems in either system that would cause a divergence in Lamport timestamps. In some embodiments, the overfitting criteria can be set such that the macro-Lamport change is equal to a small non-zero value. In this configuration, overfitting may indicate that the two computational systems are functionally equivalent and perform similarly, even if there are subtle differences in event ordering and causal relationships.

[0086] In one embodiment, the overfitting / underfitting analysis component 860 can determine whether underfitting has occurred between two systems by comparing the calculated changes with underfitting criteria from a database. As described above, underfitting criteria may involve setting a specific threshold for either micro-Lamport changes or macro-Lamport changes to indicate changes in an acceptable Lamport timestamp that do not affect the overall functionality or performance of the system. In some embodiments, the underfitting criteria may be set such that the macro-Lamport change equals a non-zero value. When the underfitting criteria are met (e.g., the macro-Lamport change exceeds a non-zero value), this may indicate that the first and second systems are functionally dissimilar, and that the second computational system is unsuitable or unreliable for a given test workload.

[0087] In one embodiment, when overfitting and / or underfitting criteria are met, the reporting component 865 can generate a report or alert to notify a human tester of a potential problem. In some embodiments, the report generated by the reporting component 865 may include all relevant information regarding text re-execution, Lamport timemaps, and calculated Lamport changes. In some embodiments, the report may include a Lamport tree, test workload, calculated micro and macro Lamport changes, a comparison of Lamport timestamps for each event or interaction in the first and second systems, and any external or internal factors that may have caused the change. In some embodiments, the reporting component 865 may create a visual representation of the data, such as a chart or line graph, to help the tester better understand the nature and extent of the change.

[0088] In the example shown, storage device 815 may include overfitting criteria 870, underfitting criteria 875, historical timestamp data 880, and historical change data 885. In some embodiments, the above information may be stored in a remote database (e.g., Figure 2 In 270), the remote database is connected to computing device 800 via a network.

[0089] Reference has been made to the embodiments set forth in this disclosure. However, the scope of this disclosure is not limited to the specifically described embodiments. Rather, any combination of features and elements (whether or not associated with different embodiments) is contemplated for implementing and practicing the contemplated embodiments. Furthermore, while the embodiments disclosed herein may be advantageous relative to other possible solutions or to prior art, whether a particular advantage is achieved by a given embodiment does not limit the scope of this disclosure. Therefore, the aspects, features, embodiments, and advantages discussed herein are merely illustrative and should not be considered as elements or limitations of the appended claims unless expressly recited in the claims. Similarly, references to “invention” should not be construed as a generalization of any inventive subject matter disclosed herein and should not be considered as elements or limitations of the appended claims unless expressly recited in the claims.

[0090] The aspects of this disclosure may take the form of a completely hardware embodiment, a completely software embodiment (including firmware, resident software, microcode, etc.), or an embodiment combining software and hardware aspects, all of which may generally be referred to as a "circuit", "module", or "system".

[0091] Various aspects of this disclosure are described through narrative text, flowcharts, block diagrams of computer systems, and / or block diagrams of machine logic included in computer program product (CPP) embodiments. Regarding any flowchart, depending on the technology involved, operations may be performed in a different order than shown in a given flowchart. For example, again depending on the technology involved, two operations shown in consecutive flowchart blocks may be performed in reverse order, as a single integrated step, simultaneously, or with at least partial overlap in timing.

[0092] Computer Program Product Embodiment (“CPP Embodiment” or “CPP”) is a term used herein to describe a group of one or more storage media (also referred to as “media”) collectively included in a group of one or more storage devices that collectively include machine-readable code corresponding to computer operations specified in a given CPP claim. A storage device is any tangible device capable of holding and storing instructions for use by a computer processor. Without limitation, a computer-readable storage medium can be an electronic storage medium, a magnetic storage medium, an optical storage medium, an electromagnetic storage medium, a semiconductor storage medium, a mechanical storage medium, or any suitable combination of the foregoing. Some known types of storage devices that include these media include: magnetic disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), static random access memory (SRAM), optical disc read-only memory (CD-ROM), digital universal disc (DVD), memory sticks, floppy disks, mechanical encoding devices (such as punched cards or pits / bumps formed on the main surface of the disk), or any suitable combination of the foregoing. As used herein, the term "computer-readable storage medium" should not be construed as storing transient signals themselves, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating in waveguides, optical pulses transmitted through fiber optic cables, electrical signals communicated through wires, and / or other transmission media. As those skilled in the art will understand, during normal operation of a storage device, such as during access, defragmentation, or garbage collection, data is typically moved at certain accidental points in time, but this does not make the storage device transient, because data is not transient when it is stored.

[0093] While the foregoing describes embodiments of this disclosure, other and further embodiments may be devised without departing from its basic scope, the scope of which is defined by the following claims.

Claims

1. A method comprising: The first multiple event sequences are generated by executing multiple test paths on the first computing environment; A test workload is constructed based on the first plurality of event sequences; A second plurality of event sequences are generated by executing the test workload on a second computing environment; One or more changes are identified by comparing the first plurality of event sequences with the second plurality of event sequences; as well as The second computing environment is reconfigured based on one or more of the aforementioned changes.

2. The method according to claim 1, wherein, Each of the first plurality of event sequences includes a sequence of timestamps, each timestamp corresponding to a corresponding test operation performed during the execution of a corresponding test path on the first computing environment.

3. The method according to claim 1 or 2, wherein, The test workload includes one or more binary trees, each corresponding binary tree corresponding to a specific test path among the multiple test paths.

4. The method according to claim 1 or 2, wherein, The test workload includes one or more binary trees, each corresponding binary tree corresponding to a combination of at least two of the multiple test paths.

5. The method according to any of the preceding claims, wherein, Each of the second plurality of event sequences includes a sequence of timestamps, each timestamp corresponding to a corresponding test operation performed during the execution of the corresponding binary tree on the second computing environment.

6. The method according to any of the preceding claims, wherein, The one or more changes include macroscopic changes, wherein the macroscopic changes include the difference between timestamps associated with the first plurality of event sequences and timestamps associated with the second plurality of event sequences.

7. The method according to any of the preceding claims, wherein, The one or more changes include micro-changes, wherein the micro-changes include the difference between a timestamp associated with a first event sequence from the first plurality of event sequences and a timestamp associated with a second event sequence from the second plurality of event sequences.

8. The method according to claim 7, wherein, When executed on the first computing environment, the first event sequence corresponds to the first test path among the plurality of test paths, and when executed on the second computing environment, the second event sequence corresponds to the first test path.

9. The method according to any of the preceding claims, further comprising: Compare the one or more changes with one or more overfitting criteria; as well as When it is determined that one or more overfitting criteria are met, it is determined that the causal relationship between test events in the first plurality of event sequences will be maintained in the second plurality of event sequences.

10. A system comprising: One or more memories that together store computer-executable instructions; as well as One or more processors are configured to jointly execute the computer-executable instructions and enable the system: The first multiple event sequences are generated by executing multiple test paths on the first computing environment; A test workload is constructed based on the first plurality of event sequences; A second plurality of event sequences are generated by executing the test workload on a second computing environment; One or more changes are identified by comparing the first plurality of event sequences with the second plurality of event sequences; as well as The second computing environment is reconfigured based on one or more of the aforementioned changes.

11. The system according to claim 10, wherein, Each of the first plurality of event sequences includes a sequence of timestamps, each timestamp corresponding to a corresponding test operation performed during the execution of a corresponding test path on the first computing environment.

12. The system according to claim 10 or 11, wherein, The test workload includes one or more binary trees, each corresponding binary tree corresponding to a specific test path among the multiple test paths.

13. The system according to claim 10 or 11, wherein, The test workload includes one or more binary trees, each corresponding binary tree corresponding to a combination of at least two of the multiple test paths.

14. The system according to any one of claims 10 to 13, wherein, Each of the second plurality of event sequences includes a sequence of timestamps, each timestamp corresponding to a corresponding test operation performed during the execution of the corresponding binary tree on the second computing environment.

15. The system according to any one of claims 10 to 14, wherein, The one or more changes include macroscopic changes, wherein the macroscopic changes include the difference between timestamps associated with the first plurality of event sequences and timestamps associated with the second plurality of event sequences.

16. The system according to any one of claims 10 to 15, wherein, The one or more changes include micro-changes, wherein the micro-changes include the difference between a timestamp associated with a first event sequence from the first plurality of event sequences and a timestamp associated with a second event sequence from the second plurality of event sequences.

17. The system according to any one of claims 10 to 16, wherein, The one or more processors are configured to jointly execute the computer-executable instructions and enable the system: Compare the one or more changes with one or more overfitting criteria; as well as When it is determined that one or more overfitting criteria are met, it is determined that the causal relationship between test events in the first plurality of event sequences will be maintained in the second plurality of event sequences.

18. A computer program product comprising: A computer-readable storage medium having computer-readable program code, the computer-readable program code being executable to cause the computer program product to: The first multiple event sequences are generated by executing multiple test paths on the first computing environment; A test workload is constructed based on the first plurality of event sequences; A second plurality of event sequences are generated by executing the test workload on a second computing environment; as well as One or more changes are identified by comparing the first plurality of event sequences with the second plurality of event sequences; The second computing environment is reconfigured based on one or more of the aforementioned changes; as well as One or more processors, each of the one or more processors being configured to execute at least a corresponding portion of the computer-readable program code.

19. The computer program product according to claim 18, wherein, Each of the first plurality of event sequences includes a sequence of timestamps, each timestamp corresponding to a corresponding test operation performed during the execution of a corresponding test path on the first computing environment.

20. The computer program product according to claim 18 or 19, wherein, The test workload includes one or more binary trees, each corresponding binary tree corresponding to a specific test path among the multiple test paths.

21. A computer program product comprising a computer-readable storage medium having computer-readable program code implemented therewith, the computer-readable program code being configured to implement the method according to any one of claims 1 to 9.

Images