Testing distributed immutable data objects

Abstract

A system, method, and apparatus for managing a distributed environment is disclosed. [Solution] A method for testing nodes in a distributed system includes configuring a number of virtual nodes to emulate a distributed system used to process and manage distributed immutable data objects, initiating testing of the emulated distributed system, the testing including performing one or more transactions on a first distributed immutable data object to obtain a modified first distributed immutable data object, and validating the emulated distributed system by comparing the modified first distributed immutable data object with a second distributed immutable data object being processed in the verified system.

Description

[Technical field]

[0001] Priority claim This application claims priority to and the benefit of U.S. Non-provisional Patent Application No. 17 / 578,432, filed in the U.S. Patent Office on January 18, 2022, U.S. Provisional Patent Application No. 63 / 140,849, filed in the U.S. Patent Office on January 23, 2021, and U.S. Provisional Patent Application No. 63 / 282,042, filed in the U.S. Patent Office on November 22, 2021, the entire contents of which are incorporated by reference herein as if fully set forth below and for all applicable purposes.

[0002] Technical Field The present invention relates to a testing environment for systems, and more particularly to a deployed testing platform configured as a virtual network of virtual nodes that can be used for training, testing, and simulation. [Background technology]

[0003] background In 2009, someone named Satoshi Nakamoto, or a group of people using that name, wrote a paper that would eventually become the driving force behind Bitcoin. Bitcoin was the first successful cryptocurrency based on a decentralized, networked, distributed computing technology. At the heart of this technology is a distributed ledger that is updated using transactions held in a blockchain and verified using proof-of-work.

[0004] Certain well-known technologies continue to have a significant impact on computing today. For example, containers and virtual machines are well known in the art and have had a significant impact on distributed computing. A container can be defined as a package or image that includes an application and its corresponding runtime libraries and dependencies. More recently, turnkey container products and management such as Docker® have become popular, naturally leading to the existence of successful container orchestration products such as Kubernetes®. A virtual machine can be defined as a software computer that emulates the behavior of a physical computer. Containers and virtual machines can enable the deployment and scaling of applications according to the design requirements and capabilities of the physical processing environment. Summary of the Invention [Means for solving the problem]

[0005] overview Certain aspects of the present disclosure provide systems, apparatus, and methods that can be used to safely and reliably test the operation and elements of a distributed computing system. In some aspects, the test system can be implemented using any combination of physical processing units, virtual machines, and / or distributed containers to emulate a physical or real-world distributed computing system.

[0006] In one aspect of the disclosure, a method for testing nodes in a distributed system includes configuring a plurality of virtual nodes to emulate a distributed system used to process and manage distributed immutable data objects (DIDOs); initiating testing of the emulated distributed system, where the testing includes executing one or more transactions against the first DIDO to obtain a modified first DIDO; and validating the emulated distributed system by comparing the modified first DIDO with a second DIDO that has been generated from the first DIDO by a verified system.

[0007] In one aspect of the disclosure, a testing platform for a distributed system includes a plurality of processing systems, each configured to operate one or more of a plurality of virtual nodes, a network coupled to the plurality of processing systems and configured to communicatively couple the plurality of virtual nodes such that the plurality of virtual nodes are configured to emulate a distributed system used to process and manage DIDOs, and a test management processor. The test management processor may be configured to initiate testing of the emulated distributed system, the testing including performing one or more transactions on a first DIDO to obtain a modified first DIDO, and validate the emulated distributed system by comparing the modified first DIDO with a second DIDO that has been generated from the first DIDO by a verified system.

[0008] In one aspect of the disclosure, a method for configuring a virtual environment includes configuring a virtual network of virtual nodes in response to a request for testing, simulation, or training on a distributed system that deploys the distributed system among a plurality of virtual nodes in the virtual network, receiving results of the testing, simulation, or training from the plurality of virtual nodes, and providing a response to the request by combining the results of the testing, simulation, or training received from the plurality of nodes.

[0009] In one aspect of the disclosure, a system includes a virtual network of virtual nodes and a virtual environment manager that can configure the virtual network of virtual nodes in response to a request for testing, simulation, or training on the distributed system, deploy the distributed system among multiple virtual nodes in the virtual network, receive test, simulation, or training results from the multiple virtual nodes, and provide a response to the request by combining the test, simulation, or training results received from the multiple nodes. [Brief description of the drawings]

[0010] BRIEF DESCRIPTION OF THE DRAWINGS [Figure 1] We present certain aspects of an example of developing a blockchain-based distributed ledger. [Diagram 2] 1 illustrates a first example of a distributed testing environment (TE) configured in accordance with certain aspects of the present disclosure. [Diagram 3] 1 illustrates a virtual node that may be configured in accordance with certain aspects of the present disclosure. [Figure 4] 1 illustrates configurable features of a distributed immutable data object (DIDO) TE provided in accordance with certain aspects of the present disclosure. [Diagram 5] 1 is a flow diagram illustrating a process for defining a DIDO TE platform according to certain aspects of the disclosure. [Figure 6] 1 is a flow diagram illustrating a process for defining a DIDO TE node set according to certain aspects of the disclosure. [Figure 7] 1 is a flow diagram illustrating a process for defining a DIDO TE virtual network according to certain aspects of the disclosure. [Figure 8] 1 illustrates certain configuration settings for initializing a DIDO in accordance with certain aspects of the present disclosure. [Figure 9] 1 illustrates an example of code debugging used within a DIDO TE in accordance with certain aspects of the present disclosure. [Figure 10] 1 illustrates an example of a DIDO TE monitor according to certain aspects of the present disclosure. [Figure 11] 1 illustrates an example of testing for malicious agents, according to certain aspects of the present disclosure. [Figure 12] 1 illustrates an example of a DIDO TE manager in accordance with certain aspects of the present disclosure. [Figure 13] 1 illustrates an example of a test and test plan structure provided in accordance with certain aspects of the present disclosure. [Figure 14] 1 is a flow diagram illustrating a process for defining a test to be performed using a DIDO TE, according to certain aspects of the disclosure. [Figure 15] 1 is a flow diagram illustrating a process for performing a test using a DIDO TE, according to certain aspects of the disclosure. [Figure 16] 1 illustrates a second example of a distributed TE configured in accordance with certain aspects of the present disclosure. [Figure 17] 1 is a flow diagram illustrating configuring a test platform in accordance with certain aspects of the present disclosure. [Figure 18] 1 is a flow diagram illustrating using a test platform in accordance with certain aspects of the present disclosure. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0011] Detailed Description The detailed description set forth below with reference to the accompanying drawings is intended to be a description of various configurations and is not intended to represent the only configurations in which the concepts described herein may be practiced. The detailed description includes specific details for the purpose of providing a thorough understanding of various concepts. However, it will be apparent to one skilled in the art that the concepts may be practiced without those specific details. In some instances, well-known structures and components are shown in block diagram form to avoid obscuring such concepts.

[0012] The difficulties associated with testing a system increase significantly when the system to be tested is a distributed system. The components and platforms of a distributed system may be physically located anywhere in the world. Proper testing requires subject matter experts, distributed product experts, hardware experts, operating system experts, test experts, etc. Such testing also requires management experts to coordinate people, resources, and test scenarios within a specified time frame.

[0013] Cryptocurrencies provide an example that relies on distributed computing. The rising excitement about distributed computing brought about by the success of cryptocurrencies such as Bitcoin has led to great enthusiasm and energy on the part of businesses and investors to participate in the opportunities and benefits of distributed computing systems in general and cryptocurrencies specifically. Distributed computing products, and cryptocurrencies in particular, are inherently developed as part of a community-of-interest (COI). If not managed correctly, COIs can exacerbate risks to the success of distributed systems.

[0014] Cryptocurrency is a term applied to a type of asset used in exchange mechanisms that rely on digital media to operate. In some examples, records of coin ownership are recorded and maintained in a distributed ledger. A distributed ledger may be implemented using distributed ledger technology and computerized databases. Strong encryption techniques are used to protect the transaction records kept by the distributed ledger. In some examples, a distributed ledger may be constructed using a blockchain, where ledger entries, records, or transactions are maintained as blocks, with each block containing a cryptographic hash of the previous block and a timestamp.

[0015] FIG. 1 illustrates certain aspects of an example of formulating a blockchain-based distributed ledger 100. The distributed ledger 100 begins with a first block 102 from which subsequent blocks, including blocks 104, 106, 108, 110, 112, 114, 116, are created. Each block holds one or more valid transactions that are hashed and encoded into a tree structure. Each block contains a cryptographic hash of a previous block in the blockchain that links each subsequent block to its immediately preceding block. In one example, block 104 contains a cryptographic hash of block 102, and block 106 contains a cryptographic hash of block 104. Blocks 102, 104, and 106 form a chain. In some examples, subsequent blocks 112 or 114, which are branches or orphans, are created in parallel with or in addition to the corresponding subsequent blocks 106 or 110 that are created from the preceding blocks 104 or 108 contained in the main chain.

[0016] A blockchain may be controlled and managed by a peer-to-peer network that operates according to an inter-node communication protocol and a validation protocol that is used to create new blocks 104, 106, 108, 110, 112, 114, 116. Although control of the cryptocurrency is exercised in a decentralized manner, the underlying technology is considered secure by design. When a transaction is recorded in a block, the block cannot be modified without modifying all blocks subsequently created.

[0017] Certain aspects of the present disclosure use community-based testing, which can be used safely and reliably to test distributed computing systems. In some examples, the community-based testing disclosed herein can be used to test, verify, authorize, or certify distributed computing systems that operate or manage distributed ledgers, such as the blockchain-based distributed ledger 100 shown in FIG. 1. Adopting distributed computing presents several new and unique technical challenges for existing enterprises to overcome, and also requires enterprises to adapt to participate in products that run on networks and computers outside of their control. Using products based on distributed computing may require enterprises to participate as peers in a community to achieve a common goal, while the participating enterprises can retain confidence in the quality of the products being tested.

[0018] Using traditional community-based testing for cryptocurrencies cannot be easily done in an iterative manner. The use of blockchain entails that any testing that involves amending the ledger will change the blockchain in an irreversible manner. As a result, any repeated testing is performed on a modified blockchain. Furthermore, errors or problems caused by a faulty system during testing cannot be removed or undone from the blockchain, and the erroneous or problematic data remains in the blockchain.

[0019] Certain aspects of the present disclosure provide a virtualized environment that can be established on an existing physical infrastructure to test cryptocurrencies and other blockchain-based systems without leaving a persistent footprint in the blockchain. This virtualized environment can be initialized multiple times so that testing can be repeatedly started from a desired starting point with a pre-configured context and blockchain. The results of the test can be evaluated and the test can be re-run with the same virtual system configuration or a modified or updated system configuration. A virtual system can be configured to emulate or simulate a physical system or an aspect of a physical system. In the present disclosure, a virtual system emulates a physical real-world system when it mimics and closely matches the behavior of a real-world system. In the present disclosure, a virtual system simulates a physical real-world system when it mimics a real-world system without matching one or more behaviors of the real-world system. For example, an emulation can operate according to a timeline that runs at real-world speed, whereas a simulation can operate according to an accelerated or decelerated timeline.

[0020] In some examples, a virtual system time can be defined that is not fixed or locked to real time. For example, the start of the timeline can be configured when starting a virtual system on a virtual network. Thus, identical subsequent blocks in the blockchain can be created in different executions of the virtualized system at the same virtual time to ensure that transactions are recorded in a consistent manner. In one example, using virtual system time allows certain time-dependent cryptographic hashing algorithms to produce predictable or repeatable results during multiple test cycles. For simulation purposes, virtual system time may be unlinked from real time and may proceed at a different rate than real time, allowing for expansion or compression of virtual time.

[0021] Certain aspects of the present disclosure may provide a simplified testing environment that keeps the potential risk of adopting a new, reconfigured, or untested system within an acceptable level with tolerances set by enterprise goals. By using a distributed immutable data object (DIDO) or multiple DIDOs, a testing environment is provided that can simplify testing procedures and significantly shorten the process of starting up a distributed system for testing. The testing environment disclosed herein may be referred to herein as a "Distributed Immutable Data Object Testing Environment (DIDO TE)." The term "DIDO" may also be used to refer to the underlying technology that supports distributed data and computation across a distributed network of peers, using a consensus algorithm to maintain integrity and consistency across the network. DIDOs are immutable to the extent that certain data objects contained within or managed by a DIDO can never be changed. New copies of data can be created and can coexist with the original data, but each new copy of a DIDO points back to a previous version of the DIDO or into the DIDO. Different versions of DIDOs may be created when real-world DIDOs are created to be used as baseline DIDOs in DIDO TE. A baseline DIDO version may be created when an identical copy of the baseline DIDO is operated within the DIDO TE according to a test plan. Different versions of the baseline DIDO may be compared when evaluating test results. After execution of part or all of the test plan, the initial content of a version of the baseline DIDO may be compared to the content of the same version of the baseline DIDO.

[0022] After DIDO is configured or instantiated, users can use DIDO TE to rigorously test systems. DIDO TE can be implemented as a virtualized, containerized testbed to enable various types of tests to be performed. DIDO TE can include multiple virtual nodes, each configured to run one or more types of tests, including comparative testing and evaluation between distributed system implementations. DIDO TE can be used for testing, simulation, or training. In some examples, DIDO TE can be used to verify or certify the operation of system components using a defined timeline, one or more pre-configured immutable objects, and a sequence of actions or transactions that include the pre-configured immutable objects. In the latter example, known repeatable contexts can be created and recreated, and deterministic results can be expected from systems, nodes, platforms, and functional elements that function according to the corresponding specifications.

[0023] In some examples, the DIDO TE can be configured to test new or updated platforms. It is generally desirable to test new platforms under real-world operating conditions without affecting or impairing the operational system. The DIDO TE can be configured with various virtual nodes that emulate components of the real-world operational system. An artificial timeline can be defined for the DIDO TE. The DIDO TE can be configured with real-world status information, data, and copies of one or more DIDOs captured from the real-world operational system. The DIDO TE can then begin operating at selected time points configured by the DIDO TE manager. The DIDO TE can be initially configured with verified virtual nodes and used to generate one or more baselines for a test plan. The baselines can define expected results from tests following a corresponding test plan executed on a specified or identified DIDO TE. The DIDO TE can then be used to exercise new or modified virtual nodes that are introduced into the DIDO TE that emulates the new or updated physical nodes using the test plan. A test plan may be executed using a DIDO TE that includes new or modified virtual nodes and results that are compared to a baseline. The results may include status information, data, and one or more DIDOs captured from the DIDO TE after execution of the test results. The results may include information collected from the modified virtual node and / or one or more verified virtual nodes. The results may include, for example, analyzing differences between a version of the DIDO collected using the test plan and a version of the DIDO provided by a baseline. A test plan may include repetition, timing variations, and introduction of events or stressors intended to test the system capabilities, reliability, and functionality of the virtual node or one or more individual nodes.

[0024] In some examples, the DIDO TE may be configured to simulate a physical network of verified nodes. In some examples, the DIDO TE may be configured to simulate a certain physical network or node is emulated by the system, network, or node under test. Simulations may be performed to evaluate or predict the performance and reliability of a physical system under varying conditions or conditions that rarely occur in normal operation. Simulations may be used to evaluate overall system performance or the performance of individual nodes, groups of nodes, types of nodes, or versions of the same type of node. For example, a simulation may examine the effect of lost processing power in nodes, including cellular telephone devices, when processor throttling is used to reduce battery drain rate and reduce temperature rise. The DIDO TE may be composed of various virtual nodes that emulate components of real-world operational systems. An artificial timeline may be defined for the DIDO TE. In some circumstances, the artificial timeline may proceed at the same speed (full rate) as the real-world system. In some simulations, the artificial timeline may be accelerated relative to the real-world system to simulate long-term real-world behavior within a more manageable test period. In some simulations, the artificial timeline can be slowed down relative to a real-world system. Slowing down can improve visibility of complex and rapidly occurring transactions. A DIDO TE can be configured with real-world status information, data, and copies of one or more DIDOs captured from real-world operational systems. The DIDO TE can then begin operating at selected time points configured by the DIDO TE manager. A DIDO TE can be initially configured with verified virtual nodes and used to generate one or more baselines for scripts or test plans. The DIDO TE can then be used to exercise new or modified virtual nodes that are introduced into the DIDO TE, which emulates new or updated physical nodes using the scripts or test plans.

[0025] In some examples, the DIDO TE can be configured for training purposes. For training purposes, the DIDO TE can be configured for test or simulation modes based on the nature of the training to be performed. In some examples, end users or system operators can be trained using a script-driven DIDO TE that provides pre-configured flows that familiarize and guide trainees on the proper operation of the system and are configured to identify and resolve and issue error handling. In some examples, system engineers and developers can be trained using test plans that run on the DIDO TE that familiarize and guide trainees on system functionality, development tools, and troubleshooting.

[0026] FIG. 2 illustrates an example of a DIDO test framework 200 that includes or incorporates a test environment or (DIDO TE 202). The DIDO test framework 200 may operate within any environment that is comprised of concepts inherent to the DIDO solution and a specialized collection of interfaces that can facilitate control and management of a distributed network of nodes within the DIDO TE. The DIDO test framework 200 may orchestrate an on-demand workflow through a set of distributed workflow processes. In one example, the DIDO test framework 200 may provide, implement, or use a mailbox messaging paradigm to communicate status or control requests.

[0027] Throughout the stages of a defined workflow, processes are spawned, messages are acted upon, responses are received, and resources are released after work is completed. Processes can be distributed in the manner of a distributed application, and in some implementations the DIDO test framework 200 may host its own instances, allowing the system to grow indefinitely.

[0028] In certain examples, the DIDO TE202 may be built on a functional programming language such as the Elixir functional programming language, which may be configured for use with a virtual machine such as the BEAM virtual machine, which may be referred to herein as an Erlang virtual machine, which is a core component of the Erlang Open Telecom Platform (OTP). In other examples, other virtual machines and associated programming languages ​​may be used. The illustrated DIDO TE202 has four main applications that manage the messaging layer through custom application programming interfaces (APIs). In some examples, interaction with these APIs may be through public representational state transfer (REST) ​​interfaces or through a simple, easy-to-use graphical user interface (GUI). The term "REST interface" may refer to a distributed system framework that uses web protocols and technologies. The REST architecture may relate to client and server interactions. Applications may be packaged and distributed for any host operating system, thereby enabling portability and use of isolated operating environments.

[0029] In the illustrated example, DIDO TE 202 may include a test environment manager 204. The test environment manager 204 may include one or more communicatively coupled processing systems that operate to configure one or more virtual nodes 212. In some examples, the virtual nodes 212 may be physically co-located with the test environment manager 204 or with a processing system that is a component of the test environment manager 204. In some examples, the virtual nodes 212 may be in a location that is remote and may communicate with the test environment manager 204 via a local area network or a wide area network such as the Internet. DIDO TE 202 may include or configure a virtual network 210 that allows the test environment manager 204 to communicate with the virtual nodes 212 regardless of the physical distance that separates the test environment manager 204 from the virtual nodes 212. In some examples, the virtual network 210 may be configured to represent the distances, latencies, and communication protocols used to communicate between the nodes and the management system in a physical real-world distributed system.

[0030] In some implementations, a secure HTTP(S) web server (DIDO web server 222) may be used to host a user-friendly web GUI for interacting with and configuring the DIDO TE 202. In some implementations, the DIDO web server 222 application may be co-hosted with a REST interface for client and third-party application access. These two services may be tightly coupled for use case flexibility and ease of feature implementation. Third-party authentication gives the application multiple integration possibilities for additional plug-in functionality in the future.

[0031] According to one aspect of the disclosure, a series of transactions and / or events defined in a set of test plans 230 may be used to test the virtual network 210, one or more virtual nodes 212, and / or an emulated or simulated system. Testing may be performed using multiple active test plans in the set of test plans 230. In some examples, the testing may include simulated events caused by or related to a malicious agent 208, which may be simulated by the DIDO TE 202.

[0032] In one example, one or more virtual nodes may be configured to perform acceptance testing according to an acceptance test plan 238. Acceptance testing is a technique that may be performed to determine if a software system meets requirements. The requirements specification may describe functional and non-functional characteristics required of the software system.

[0033] In some examples, one or more virtual nodes may be configured to perform black-box testing according to a black-box test plan 246, which may be defined as a type of software testing where the functionality of the software is not known. The testing may be performed without internal knowledge of the product. In some examples, black-box testing may include syntax-driven testing and equivalence partitioning. Syntax-driven testing may be applied to systems that may be syntactically represented by some language, such as a compiler, or a language that may be represented by a context-free grammar. In the latter example, tests may be generated such that each grammar rule is used at least once. Equivalence partitioning may include partitioning the input domain of the system into several equivalence classes, such that each element of the class functions in a similar manner. Many types of inputs may behave in a similar manner, and similar inputs may be grouped together, and tests may be performed on only one input of each group. For example, each element of an equivalence class may behave similarly, and if a test item performed on one element of the class incurs some error, other elements of the class may be expected to respond to the test with the same error.

[0034] In some examples, one or more virtual nodes may be configured to perform end-to-end testing (E2E testing) according to an end-to-end test plan 240. E2E testing may refer to a software testing methodology that involves testing an application workflow from beginning to end. This methodology aims to replicate real-world scenarios to validate the system for integration and data integrity. The virtual nodes perform E2E testing by executing all operations configured for the application to account for how the application communicates with hardware, network connectivity, external dependencies, databases, and other applications. Typically, E2E testing is performed after the completion of functional and system testing.

[0035] In some examples, one or more virtual nodes may be configured to perform integration testing according to an integration test plan 236. Integration testing may be performed to test individual components to determine how they function together. Integration testing may be performed to test modules that have been determined to function properly as individual or standalone elements and that have no obvious bugs when integrated. Integration testing is the most common type of functional testing and may be performed using automated testing. Typically, different developers build different modules of a system or software simultaneously in parallel without reference to other modules. The developers may perform extensive unit testing on the individual modules, including black-box and white-box functional validation. Integration testing is performed by flowing data and operational commands between modules, and the modules are operated as parts of the overall system rather than as individual components. Integration testing may reveal issues with user interface operations, data formats, operational timing, application programming interface (API) calls, and database access and user interface operations.

[0036] In some examples, one or more virtual nodes may be configured to perform interoperability testing according to the interoperability test plan 252. Interoperability testing may be performed to test the ability of individual components to function within a group of components. Interoperability may be configured to test the response of a virtual node or functional component to performance variations. For example, a collocated node or functional component accessed by a high-speed communication link may respond with lower latency (i.e., more quickly) than a remotely located node accessed over a slower communication link. Interoperability testing may be performed to ensure that the virtual device under test can respond appropriately to transactions and events without being hindered by timing skews. Interoperability testing may be performed by flowing data and operational commands between modules, which are operated as parts of an overall system rather than as individual components. Interoperability testing may address timing variations and other variations related to the set of issues addressed by integration testing.

[0037] In some examples, one or more virtual nodes may be configured to perform interface testing between the nodes according to an interface test plan 250. An interface may include a connection that integrates two components and may include APIs, web services, and other types of connectivity services. An interface may include software that includes a set of commands, messages, and other attributes that enable communication between a device and a user. An interface test may be performed as a type of software test that verifies whether communication between two different software systems occurs correctly.

[0038] In some examples, one or more virtual nodes may be configured to perform regression testing according to a regression test plan 234. Regression testing may include iteratively performing tests on functional and non-functional requirements to ensure that a previously developed and tested network of virtual nodes functions as expected after changes to one or more nodes. A fault or error may be referred to as a regression on functionality. Changes that require regression testing include bug fixes, software enhancements, configuration changes, and replacements of electronic or other hardware components. Test automation is often involved, since regression test suites tend to grow as defects are found. Sometimes, a ripple effect analysis is performed to determine an appropriate subset of tests (non-regression analysis).

[0039] In some examples, one or more virtual nodes may be configured to perform sanity testing according to a sanity test plan 248. Sanity testing may be a subset of regression testing. Sanity testing is performed to ensure that code changes made are functioning as expected. Sanity testing is a stoppage to check whether testing of a software build can proceed. One purpose of sanity testing is to verify the functionality of an application without extensive testing. Sanity testing is typically performed on builds that require immediate production deployment, such as after a critical bug fix has been made.

[0040] In some examples, one or more virtual nodes may be configured to perform smoke testing according to a smoke test plan 244. Smoke testing may be performed on new or updated software builds to verify that basic functionality is working. Smoke testing is typically the first test performed on any new build. In smoke testing, test cases are chosen to cover the most important functions or components of the system. The objective is not to perform exhaustive testing, but to verify that critical functions of the system are working as expected.

[0041] In some examples, one or more virtual nodes may be configured to perform unit testing according to a unit test plan 232. Unit testing is performed to ensure that each piece of code within a component gives the desired or expected output. Unit testing typically covers the interfaces and specifications for the component. Unit testing may provide documentation of code development, as each unit of code is tested independently and thoroughly before proceeding to another unit. Unit testing supports functional testing by exercising the code that is most likely to break.

[0042] In some examples, one or more virtual nodes may be configured to perform white-box testing according to a white-box test plan 242. White-box testing goes beyond black-box testing and uses techniques to analyze the internal structure, including data structures, internal design, code structure, and functionality of the software. White-box testing may also be referred to as glass-box testing, clear-box testing, or structural testing.

[0043] DIDO TE may be based on a reference architecture, referred to herein as the DIDO Reference Architecture (DIDO-RA). DIDO TE provides a unique, dynamic, distributed, testing environment as a service (TEaaS) that can be made widely accessible and available. DIDO TE addresses a wide range of user scenarios for platforms that include or use DIDO. DIDO TE can rapidly deploy a network of virtual nodes for use cases including peer-to-peer, blockchain with smart contracts, and clustered applications.

[0044] In some cases, the core functionalities of the product include the following DIDO TE features: Ability to create multiple types of nodes in a virtualized network with all necessary resources allocated for interoperability testing. Ability to quickly scale from one virtualized node to thousands of virtualized nodes on a test-ready virtualized network.

[0045] In some examples, the test environment manager 204 is the core engine and controller of the DIDO TE framework 200. The test environment manager 204 manages and controls the entire lifecycle of the DIDO virtual network, applications, test case execution, and resource management. The DIDO test manager API can include custom integral node command maps to ensure scalability and native protocol support for any distributed application. Custom resource orchestration and management protocols allow configuration of virtual nodes for various operating systems and other environment selection and support parameters. In some examples, the virtual nodes can be virtual machines or containers.

[0046] FIG. 3 illustrates an example of a virtual node 302 providing a platform 304 that can be used to implement a portion of the DIDO test framework 200 illustrated in FIG. 2. The virtual node 302 can be implemented using different virtual node types 312, 322 that can be configured according to certain aspects of the disclosure. The first virtual node type 312 can be implemented using a virtual machine, such as an Erlang virtual machine. The platform 314 provided by the first virtual node type 312 can be composed of components that fit the virtual machine selected. The second virtual node type 322 can be implemented using containers. The containers can wrap DIDO applications using a custom framework to interact with a container orchestration platform, such as Kubernetes. In one example, a node of an open source distributed ledger cryptocurrency network (such as the IOTA network) can be implemented on the virtual node by providing context and other information to the virtual node before starting the instantiation of the IOTA virtual node. The platform 324 provided by the second virtual node type 322 can be configured to support the operation of containers.

[0047] 4 illustrates certain configurable features of a DIDO TE system provided in accordance with certain aspects of the disclosure. DIDO TE platform 400 includes example components that may be implemented using any combination of hardware and software modules. DIDO TE platform 400 may employ one or more processing circuits including a processor that may be configured to execute a desired runtime system in a manner that emulates a target DIDO production environment. In some examples, the processing circuitry may include a processor type used in the target DIDO production environment. In some examples, the processing circuitry may operate under the control of a virtual machine that may emulate a processor type used in the target DIDO production environment.

[0048] For example, as further shown in FIG. 5, the DIDO TE platform 400 can be defined by a TE manager. The DIDO TE platform 400 includes a combination of components 402 that can be implemented using any combination of hardware and software modules. The TE manager can define the type of hardware platform 404 used, including the version 406 or age of the hardware employed by the hardware platform 404. In one example, the hardware platform 404 can be characterized by a processor type and associated resources and peripherals. In another example, the hardware platform 404 can be characterized by a type of device, such as a wireless modem with a processor and associated resources and peripherals, a user interface, and / or a mobile phone with a set of functionality provided by a system-on-chip. The TE manager can define a runtime environment that includes a runtime platform 408 that can have a specified runtime version 410. The runtime platform 408 can represent an operating system, device drivers, interrupt schemes, and other modules that control or interact with elements of the hardware platform 404. The runtime platform 408 can include a standard-defined, commercially available, or proprietary operating system. In some examples, the operating system may be a version of Unix, Linux, Windows, Java VM, etc., or a derivative thereof. The TE manager may define one or more libraries, including runtime libraries (e.g., Solidity) for a defined operating system or for a programming language.

[0049] The TE manager can define a DIDO platform 412, and in some examples, a version 414 of the DIDO platform 412. The DIDO platform 412 can include a commercially available or proprietary DIDO environment defined by standards such as Ethereum, Polkadot, Iota, Interplanetary File System (IPFS), etc.

[0050] DIDO TE node platform types 420 include combinations of components 422 that may be implemented using any combination of hardware and software modules. DIDO TE node platform types 420 represent various ways in which DIDO nodes may be configured or used within a DIDO TE. Each DIDO TE node may be associated with a DIDO TE platform 424 and may be configured according to the version 426 or instantiation of the TE, the platform type 428 implemented by the DIDO TE node, and the role 430 the DIDO TE node plays. DIDO TE platform 424 may represent or correspond to an instantiation of a DIDO TE node platform type 420 and may be identified or categorized as a virtual machine, container, or other type of virtual node. DIDO TE nodes may be operated in roles 430 such as wallets, full nodes, light nodes, archive nodes, or other nodes.

[0051] A DIDO TE node set 440 defines a set of reusable nodes that may be uniquely identified within a TE. A DIDO TE node set 440 includes a combination of components 442 that may be implemented using any combination of hardware and software modules. A DIDO TE node set 440 may be identified by a node set name 444 and may relate to a set of nodes that includes a specified number 446 of nodes. A DIDO TE node set 440 may indicate one or more configuration types 448 of the nodes. In some examples, each DIDO TE node set 440 defines a set of nodes that are all of the same type.

[0052] DIDO TE virtual network 460 defines a collection of DIDO TE node sets 440. DIDO TE virtual network 460 includes a combination of components 462 that may be implemented using any combination of hardware and software modules. DIDO TE virtual network 460 may be identified using a virtual network name 464 and may include a defined number of nodes 466 in the collection.

[0053] FIG. 5 is a flow diagram 500 illustrating a process for defining a DIDO TE platform. This process may include the participation of a DIDO TE administrator and a DIDO TE manager. The DIDO TE administrator may be responsible for creating a list of new DIDO TE platforms. The DIDO TE administrator does not need to create and verify DIDO TE platforms. In some examples, the DIDO TE administrator may create and verify some of the DIDO TE platforms. The DIDO TE manager maintains a list of DIDO TE node types available to DIDO TE. Once a DIDO TE platform is authenticated and approved, an entry may be created in a DIDO node TE platform definition data store 502 maintained by the DIDO TE administrator. The DIDO node TE platform definition data store 502 or a local data store 522, which is a copy of the data store 502, may be accessible to the DIDO TE manager. In one example, the data stores 502, 522 maintain references to platform components, including, for example, URLs.

[0054] A DIDO TE administrator may identify a need for a DIDO TE platform or receive a request 504 for a DIDO TE platform. Metadata 506 describing the requested TE platform may be provided by the administrator and a query made to the DIDO TE node platform definition data store 502 allows the DIDO TE administrator to determine whether a suitable platform exists in block 508. In some examples, a suitable or matching DIDO TE platform may exist in the DIDO TE node platform definition data store 502 and the process ends. In some examples, a suitable or matching DIDO TE platform has not yet been defined and a request 510 is made to a DIDO TE manager to create a definition of the platform. The DIDO TE manager may create or configure the platform and create an entry for the new DIDO TE platform definition in the DIDO node TE platform definition data store 502, 522. Upon completion of the task, the manager may notify the administrator of the status of the request or the request may time out.

[0055] FIG. 6 is a flow diagram 600 illustrating a process for defining a DIDO TE node set. This process may include the participation of a DIDO TE administrator and a DIDO TE manager. The DIDO TE administrator may be responsible for managing and maintaining a list of DIDO TE node sets. The DIDO TE administrator is not required to create and verify DIDO TE node sets, although in some examples the DIDO TE administrator may create and verify some of the DIDO TE node sets. The DIDO TE manager maintains a list of DIDO TE node sets available for DIDO TE. Once a DIDO TE node set is authenticated and approved, an entry can be created for the DIDO TE platform in the DIDO TE node set type definition data store 602.

[0056] A DIDO TE administrator may identify or receive a request 604 or need for a DIDO TE node set definition. Metadata 606 describing the DIDO TE node set may be provided by the administrator, and a query may be made to the DIDO TE node set type definition data store 602 to determine if a suitable or matching node set definition already exists at block 608. In some instances, a suitable or matching DIDO TE node set may already be defined and the process may end. If a suitable or matching DIDO TE node set has not yet been defined, a request 610 may be sent to the DIDO TE manager to create a node set definition and to create an entry for the new DIDO TE node set definition in the DIDO node set type definition data store 602. The manager may notify the administrator of the status of the request upon completion of the task, or the request may time out. The administrator is typically informed if the created node set definition cannot be confirmed at blocks 622 and 624. In one example, a validation failure may result from a request by an administrator to create an Iota archive node on an iOS device.

[0057] FIG. 7 is a flow diagram 700 illustrating a process for defining a DIDO TE virtual network. This process may include the participation of a DIDO TE administrator and a DIDO TE manager. The DIDO TE administrator may be responsible for managing and maintaining a list of DIDO virtual networks. The DIDO TE administrator is not required to create and verify DIDO TE virtual networks, although in some implementations the DIDO TE administrator may be configured to create and verify certain DIDO TE virtual networks. The DIDO TE manager may keep a list of DIDO TE virtual networks available for use with DIDO TE in a DIDO TE virtual network definition data store 702. Once a DIDO TE virtual network is authenticated and approved, an entry may be created in the DIDO TE virtual network definition data store 702.

[0058] A DIDO TE administrator may receive or identify a need or request 704 for a DIDO TE virtual network definition. Metadata 706 describing the requested DIDO TE virtual nodes may be generated by or provided to the administrator, and a response to a query made to the DIDO TE virtual network definition datastore 702 may be used in block 708 to determine if a suitable virtual network definition exists. In some cases, a suitable or matching DIDO TE virtual network may already be defined and the process ends. If a DIDO TE virtual network is not yet defined, a request 710 is made to a DIDO TE manager to create a DIDO TE virtual network and to create an entry in the DIDO TE virtual network definition datastore 702 for the new DIDO TE virtual network definition. The manager notifies the administrator of the status of the request upon completion of the task, or the request may time out. The administrator is typically informed if the created node set definition cannot be confirmed in blocks 722 and 724. In one example, a validation failure may occur following a request by an administrator to create a virtual network using Ethereum in the absence of miners.

[0059] 8 illustrates certain configuration settings 800 for initializing DIDO according to certain aspects of the disclosure. The test environment manager 204 is adapted to handle a large number of different types of configuration settings. For example, the test environment can be configured such that the number of nodes, node types, and platforms are tracked within the topography. In this scenario, all nodes are based on a new initial ledger (e.g., genesis block).

[0060] Figure 9 illustrates a definitional example of a code debugging setting 900 that may be configured within a test environment according to certain aspects of the present disclosure. Figure 10 illustrates a definitional example of a DIDO TE monitoring 1000 according to certain aspects of the present disclosure. Figure 11 illustrates a definitional example of a malicious agent test according to certain aspects of the present disclosure. Figure 12 illustrates an example of a DIDO TE manager configuration 1200 according to certain aspects of the present disclosure.

[0061] The DIDO Editor environment can be configured to manage and control the entire lifecycle of a web-based Integrated Development Environment (IDE). The DIDO Editor environment can be comparable to the DIDO TE 202 shown in FIG. 2 and can include an equivalent of the Test Environment Manager 204. Each IDE is created and managed for each user who wants to edit, deploy, and test smart contracts for DIDOs within the DIDO TE, among other things. This IDE is customized using multiple plugins to provide enhanced usability and support unique features for each DIDO.

[0062] DIDO TE Node Command and Control is a supplemental application for DIDO that provides an additional vector for commanding or controlling the lifecycle of DIDO or its environment. In one example, to fully support user desires and demands, the DIDO TE Node Command and Control application provides an API to manage and configure distributed applications that otherwise do not have internal APIs or require additional control.

[0063] In some examples, the test environment can be hosted on a cloud service such as Google Cloud®, Amazon AWS®, and Microsoft Azure®. Virtualizing the test environment allows for testing of malicious or malicious agents without raising concerns about accidentally releasing the agent into a computer network (e.g., into commercial, public, or proprietary cyberspace, etc.).

[0064] 13 illustrates an example of the structure of a test 1300 and a set of test plans 1302 provided in accordance with certain aspects of the disclosure. The set of test plans 1302 may correspond to the set of test plans 230 shown in FIG. 2. In various examples, a static or dynamic test includes eleven or more test plans, including: Unit testing Integration Testing End-to-end testing (E2E testing) Smoke Testing Sanity Testing Regression testing Acceptance testing ·White Box Testing Black box testing Interface testing Interoperability testing Malicious actor testing (see Figure 11)

[0065] Each test plan may define a test set consisting of multiple test steps that generate test results for comparison with expected results.

[0066] DIDO TE enables structured and reproducible testing that is performed using a DIDO TE virtual network of nodes. A test set 1304 can exercise a single DIDO virtual node, a subset of virtual nodes, or all virtual nodes in the test network. Tests can be associated with or controlled using a test definition that uniquely identifies a test plan and test set 1304 in a set of test plans 1302. A test set 1304 can include one or more test steps 1306, each of which defines an expected test result for the test step 1306. The test step 1306 is expected to result in a test result that captures the outcome of the test step 1306.

[0067] FIG. 14 is a flow diagram 1400 illustrating a process for defining tests to be performed using a DIDO TE. This process may include the participation of a DIDO TE administrator and a DIDO TE manager. The DIDO TE administrator is responsible for managing and maintaining a list of DIDO TE tests. The DIDO TE administrator does not need to create and verify DIDO TE tests. In some examples, the DIDO TE administrator may be configured to create and verify some of the DIDO TE tests. The DIDO TE manager may keep a list of DIDO TEs available for use with the DIDO TE in a DIDO TE test definition data store 1402. Once a DIDO TE test is authenticated and approved, an entry is created in the DIDO TE test definition data store 1402.

[0068] In one example, a DIDO TE administrator may receive a request 1404 for a DIDO TE test definition or may identify the need for a DIDO TE test definition. Metadata 1406 describing the DIDO TE test may be generated by or provided to the administrator, and a response to a query made to the DIDO TE test definition data store 1402 may be used in block 1408 to determine if a suitable test definition has already been defined. If the DIDO TE test has not yet been defined, a request 1410 is made to the DIDO TE manager to create the test and to create an entry in the DIDO test data store for the new test definition. Upon completion of the task, the DIDO TE manager may notify the DIDO TE administrator of the status of the request or the request may time out. The administrator is typically informed if the created DIDO TE test definition cannot be confirmed in blocks 1422 and 1424. In one example, a validation failure may occur after a request to create a test is provided by an administrator without producing the expected results.

[0069] FIG. 15 is a flow diagram 1500 illustrating a process for performing testing using DIDO TE. With continued reference to FIG. 2, this process may include the participation of a DIDO TE repository manager, a DIDO TE quality manager, and a DIDO TE manager. The DIDO TE repository manager or the DIDO TE administrator do not need to create or perform a quality assessment. The DIDO TE quality manager analyzes the modifications to the base code and defines a list of tests that need to be performed on the new base code to ensure quality before the modifications are released. An example of a change to the base code that does not require a quality assessment may be the addition or modification of a comment in the code base. The DIDO TE test manager is responsible for running all of the tests required to ensure the quality of the product before it is released. At the end of the testing process, a quality assessment report is generated from the individual test reports generated for each of the required tests.

[0070] In one example, the DIDO TE Repository Manager identifies the need for a quality assessment to be performed. To initiate a quality assessment of the repository, metadata describing the test procedure is provided by the DIDO TE Repository Manager to the DIDO TE Quality Manager. The DIDO TE Quality Manager analyzes the base code repository and determines which tests need to be performed to ensure the quality of the base code. The DIDO TE Test Manager processes the list of tests end-to-end and creates a test report that covers each individual test that was performed and summarizes the overall results. The DIDO TE Test Manager notifies the DIDO TE Quality Manager of the status of the request upon completion of the task or the test request times out.

[0071] In various examples, multiple types of nodes are created in a virtualized network with all necessary resources allocated for interoperability testing. The system can be scaled from one virtualized node to thousands of virtualized nodes on the virtualized network. In some examples, the test platform can be configured and operationalized in a time frame measurable in seconds or subseconds. The test environment can be virtualized to enable testing of malicious or nefarious agents without concern for accidentally launching an agent. Container orchestration can be used to enable comprehensive static and dynamic testing of distributed systems using container orchestration.

[0072] FIG. 16 shows an example of a DIDO TE framework 1600. DIDO TE 1602 provides a framework that can operate within any environment that includes concepts inherent to the DIDO solution as well as a specialized aggregation of interfaces that facilitate the control and management of a distributed network of nodes within DIDO. The DIDO TE framework 1600 can orchestrate on-demand work by a series of distributed workflow processes 1610 that can use, for example, a mailbox messaging paradigm or interface to communicate status or control requests. Throughout the stages of a defined workflow, processes are spawned, messages are acted upon, responses are received, and resources are released after the work is completed. Because these processes are distributed in a manner similar to any distributed application, the framework can even host itself, allowing the system to grow indefinitely.

[0073] In one example, the DIDO TE 1602 is built on a functional programming language, such as the Elixir functional programming language, which may be configured for use with an Erlang virtual machine. The illustrated DIDO TE 1602 includes a test environment manager 1604 capable of running applications that manage the messaging layer through custom application programming interfaces (APIs). In some examples, interaction with these APIs can be through public REST interfaces or through a simple, easy-to-use graphical user interface (GUI). The term "REST interface" may refer to a distributed systems framework that uses web protocols and technologies. The REST architecture may involve client and server interactions. Applications may be packaged and distributed for any host operating system, thereby enabling portability and use of isolated operating environments.

[0074] In some examples, a secure HTTP(S) web server (DIDO Web) may be used to host a user-friendly web GUI for interacting with and configuring the DIDO TE1602 test environment. This application also hosts a REST interface for client and third-party application access. These two servers can be tightly coupled for flexibility of use cases and ease of feature implementation. Third-party authentication gives the application multiple integration possibilities for additional plug-in functionality in the future.

[0075] Each node in the system may be a physical node or a virtual node. Every node in the system may be identified by an assigned unique identifier (node ​​ID). According to certain aspects, multiple nodes may be defined with the same node ID, and a pair of nodes sharing a common node ID may be referred to as a twin. A twin cannot be active in an active system at the same time. That is, only one instance of a node with a twin's address may be active and addressed at any particular time in an active or running system. In some implementations, a switching circuit or component may be configured to route traffic to a twin selected from a physical instantiation of a node (real-world twin 1618) or a virtual instance (e.g., a node in a virtual twin pool 1614). The switch may be operated to allow multiple instances of a node with the same node ID to exist in an active or running system. Virtual nodes with the same node ID may exist as part of the virtual twin pool 1614. The virtual twin switch 1616 may be configured to select from the instances in the virtual twin pool 1614 for use during runtime. In addition to the virtual twin pool 1614, instances of physical nodes 1612 may exist and operate as twins. The virtual twin switch 1616 may be configured to allow the use of the physical node 1612 in place of any of the virtual nodes in the virtual twin pool 1614. During initialization, the controller or the virtual twin switch 1616 may determine whether to use the physical node 1612 or a virtual twin in the virtual twin pool 1614. The physical node 1612 may require additional configuration parameters to bridge the gap between the virtual and physical worlds.

[0076] FIG. 17 is a flow diagram 1700 illustrating a method for configuring a test platform that may include the test environment manager 204 shown in the DIDO TE framework 200 of FIG. 2 or the test environment manager 1604 shown in the DIDO TE framework 1600 of FIG. 16. At block 1702, the test environment manager 204 or 1604 may configure a virtual network of virtual nodes in response to a request for testing, simulation, or training on the distributed system. At block 1704, the test environment manager 204 or 1604 may deploy the distributed system among multiple virtual nodes in the virtual network. At block 1706, the test environment manager 204 or 1604 may receive test, simulation, or training results from multiple virtual nodes. At block 1708, the test environment manager 204 or 1604 may provide a response to the request by combining the test, simulation, or training results received from multiple nodes.

[0077] In some examples, the test environment manager 204 or 1604 can configure each of a plurality of physical nodes to implement a virtual node of a plurality of virtual nodes. The plurality of physical nodes may include physical nodes that are geographically distant from one another. In some examples, a virtual network is associated with at least one physical node. For example, the test, simulation, or training may be performed using real-world devices, nodes, or systems that interact with the virtual network. In the latter example, the real-world devices, nodes, or systems may include a cryptographic server or a device, node, or system under test.

[0078] In one example, at least one node of the virtual network includes a container. In one example, at least one node of the virtual network includes a virtual machine.

[0079] In one example, the test environment manager 204 or 1604 can configure a plan with a set of test procedures when a request relates to a test, allocate portions of the test procedures among the nodes of the virtual network, receive results of the test procedures, and provide test results by combining results of the test procedures received from multiple nodes.

[0080] In various examples, the test environment manager 204 or 1604 can configure the test plan 230, 1622 to cause at least one virtual node to modify a DIDO and provide the modified DIDO in the results of the test. A DIDO can include a distributed ledger configured to secure one or more transaction records.

[0081] In various examples, the test environment manager 204 or 1604 can configure a timeline for executing the test plan 230, 1622 on the virtual network of virtual nodes. The timeline can be executed in line with real time. The timeline can be executed at a rate faster than real time. The timeline can be executed at a rate slower than real time. The test environment manager 204 or 1604 can repeat the test, simulation, or training based execution. The timeline for executing the test plan 230, 1622 on the virtual network of virtual nodes is restarted for each test, simulation, or training execution. The test environment manager 204 or 1604 can configure the test plan 230, 1622 to cause at least one virtual node to modify the distributed ledger of the DIDO. Modifying the distributed ledger can include using time-dependent cryptography to encrypt information maintained by the distributed ledger. The timeline can be configured to affect the time-dependent cryptography.

[0082] In one example, the test environment manager 204 or 1604 can generate baseline results for the test plan 230, 1622 by running a test, simulation, or training where each virtual node in the virtual network includes a verified node, comparing the baseline results with additional results obtained by running the test, simulation, or training after adding or modifying at least one virtual node in the virtual network, and validating the added or modified virtual node or virtual network based on the comparison of the additional results with the baseline results.

[0083] FIG. 18 is a flow diagram 1800 illustrating a method for testing nodes in a distributed system. The distributed system may include a test environment manager 204 as shown in the DIDO TE framework 200 of FIG. 2 or a DIDO test manager 1604 as shown in the DIDO TE framework 1600 of FIG. 16. At block 1802, the test environment manager 204 or 1604 may configure a plurality of virtual nodes to emulate a distributed system used to process and manage DIDOs. At block 1804, the test environment manager 204 or 1604 may initiate testing of the emulated distributed system, where the testing includes performing one or more transactions on a first DIDO to obtain a modified first DIDO. At block 1806, the test environment manager 204 or 1604 may validate the emulated distributed system by comparing the modified first DIDO with a second DIDO that has been generated from the first DIDO by a verified system. In some examples, at least one of the transactions is performed by the node under test in the emulated distributed system. The node under test may be verified if the content of the modified first DIDO corresponds to the content of the second DIDO.

[0084] In certain examples, the first DIDO is a copy of the reference DIDO. The second DIDO can be obtained by having a verified system perform one or more transactions on an identical copy of the first DIDO. The reference DIDO can include or be associated with a first distributed ledger configured to secure one or more transaction records. The second DIDO can include a second distributed ledger. The modified first DIDO includes a third distributed ledger. Each of these distributed ledgers can be kept independent. The content of the modified first DIDO can be considered to correspond to the content of the second DIDO when the decryption information in the second distributed ledger matches the corresponding decryption information in the DIDO of the third distributed ledger. The verified system can be a real-world system. The reference DIDO, the second DIDO, and the one or more transactions can be copied from the real-world system at the corresponding record time.

[0085] In some implementations, the test environment manager 204 or 1604 may configure a start point of a test timeline for executing the test plan 230, 1622 using the emulated distributed system. One or more transactions are defined by the test plan 230, 1622. In a first mode of operation, the test timeline runs at a rate consistent with real time. In a second mode of operation, the test timeline runs at a rate different from real time. After resetting the test timeline to the start point, the test environment manager 204 or 1604 may resume testing with an unmodified version of the first DIDO. The test environment manager 204 or 1604 may configure the test plan 230, 1622 to have at least one virtual node modify the distributed ledger of the first DIDO. Modifying the distributed ledger may include using time-dependent cryptography to encrypt information maintained by the distributed ledger according to the test timeline.

[0086] In some examples, the test environment manager 204 or 1604 can configure the physical nodes to implement various virtual nodes in the virtual nodes. The physical nodes may include physical nodes that are geographically distant from one another. In one example, at least one node of the virtual nodes includes a container.

[0087] In certain implementations, the test environment manager 204 or 1604 may be configured to generate a baseline result for the test plan 230, 1622 by running a test where each virtual node in the emulated distributed system includes a verified node, compare the baseline result with an additional result obtained by running the test after adding or modifying at least one virtual node in the emulated distributed system, and validate the added or modified virtual node or emulated distributed system if the baseline result matches the additional result. A virtual twin node may be included in the emulated distributed system, and the virtual twin node is addressed using an identifier shared by one or more other twin nodes in the emulated distributed system. The test environment manager 204 or 1604 may compare an additional result obtained by running the test after adding or modifying a virtual twin node with the baseline result, and validate the at least one added or modified virtual twin node or emulated distributed system if the baseline result matches the additional result. The test environment manager 204 or 1604 can compare the baseline results with additional results obtained by running a test after adding or modifying a physical twin node within one or more other twin nodes, and validate the added or modified physical twin node or emulated distributed system if the baseline results match the additional results.

[0088] The method of claims 17 and 18 may be implemented as a test platform or environment implemented using a distributed system. The test platform or environment may include a plurality of processing systems, each configured to operate one or more of a plurality of virtual nodes, a network coupled to the plurality of processing systems and configured to communicatively couple the plurality of virtual nodes such that the plurality of virtual nodes are configured to emulate a distributed system used to process and manage DIDOs, and a test management processor. The test management processor may be configured to initiate testing of the emulated distributed system, the testing including performing one or more transactions on the first DIDO to obtain a modified first DIDO. The test management processor may be further configured to validate the emulated distributed system by comparing the modified first DIDO with a second DIDO that has been generated from the first DIDO by a verified system.

[0089] In some examples, at least one of the one or more transactions is performed by a virtual node under test in the emulated distributed system. The test management processor may be further configured to validate the virtual node under test if the content of the modified first DIDO corresponds to the content of the second DIDO.

[0090] The first DIDO may be a copy of a reference DIDO, and / or the second DIDO may be obtained by having a verified system perform one or more transactions on an identical copy of the first DIDO. The verified system may include a real-world system. The reference DIDO, the second DIDO, and the one or more transactions may be copied from a real-world system at a corresponding recording time.

[0091] In some implementations, the test management processor is further configured to generate baseline results for the test plan 230, 1622 by running tests where each virtual node in the emulated distributed system includes a verified node, compare the baseline results with additional results obtained by running tests after including a twin node in the emulated distributed system and adding or modifying the virtual twin node, and verify at least one added or modified twin node or the emulated distributed system if the baseline results match the additional results. A twin node may be addressed using an identifier shared by one or more other twin nodes in the emulated distributed system.

[0092] Some implementation examples are described in the following numbered clauses: 1. A method for testing nodes in a distributed system, comprising: configuring a plurality of virtual nodes to emulate a distributed system used to process and manage distributed immutable data objects (DIDOs); initiating testing of the emulated distributed system, the testing including executing one or more transactions against the first DIDO to obtain a modified first DIDO; and validating the emulated distributed system by comparing the modified first DIDO with a second DIDO that has been generated from the first DIDO by a verified system. 2. The method of claim 1, wherein at least one of the one or more transactions is executed by a node under test within the emulated distributed system. 3. The method of any one of clauses 1 to 2, further comprising validating the node under test if the modified first DIDO content corresponds to the second DIDO content. 4. A method according to any one of clauses 1 to 3, wherein the first DIDO is a copy of a reference DIDO and the second DIDO is obtained by having a verified system perform one or more transactions against an identical copy of the first DIDO. 5. The method of claim 4, wherein the reference DIDO includes a first distributed ledger configured to secure one or more transaction records, the second DIDO includes a second distributed ledger, and the amended first DIDO includes a third distributed ledger, each of which is maintained independently. 6. The method of claim 5, wherein the modified content of the first DIDO corresponds to the content of the second DIDO when the decryption information in the second distributed ledger matches the corresponding decryption information in the DIDO of the third distributed ledger. 7. A method according to any one of clauses 4 to 6, wherein the verified system is a real-world system and the reference DIDO, the second DIDO and one or more transactions are copied from the real-world system at the corresponding recording points. 8. The method of any one of clauses 1 to 7, further comprising configuring a starting point of a test timeline for executing a test plan using the emulated distributed system, where one or more transactions are defined by the test plan. 9. The method of claim 8, wherein in a first operational mode the test timeline runs at a rate consistent with real time and in a second operational mode the test timeline runs at a rate different from real time. 10. The method of any one of clauses 8 to 9, further comprising resuming testing using an unmodified version of the first DIDO after resetting the test timeline to the starting point. 11. The method of any one of clauses 8 to 10, further comprising: configuring a test plan to cause at least one virtual node to modify the distributed ledger of the first DIDO, wherein modifying the distributed ledger includes using time-dependent cryptography to encrypt information held by the distributed ledger according to the test timeline. 12. The method of any one of clauses 1 to 11, further comprising configuring a plurality of physical nodes to implement various virtual nodes within a plurality of virtual nodes, the plurality of physical nodes including physical nodes that are geographically distant from one another. 13. The method of any one of clauses 1 to 12, wherein at least one node of the plurality of virtual nodes comprises a container. 14. A method according to any one of clauses 1 to 13, further comprising: generating a baseline result of the test by running a test where each virtual node in the emulated distributed system includes a verified node; and comparing the baseline result with an additional result obtained by running the test after adding or modifying at least one virtual node in the emulated distributed system; and verifying the added or modified virtual node or the emulated distributed system if the baseline result matches the additional result. 15. The method of clause 14, further comprising including a virtual twin node in the emulated distributed system, the virtual twin node being addressed using an identifier shared by one or more other twin nodes in the emulated distributed system, comparing the baseline results with additional results obtained by running tests after adding or modifying the virtual twin node, and validating the at least one added or modified virtual twin node or the emulated distributed system if the baseline results match the additional results. 16. The method of clause 14 or 15, further comprising comparing the baseline results with additional results obtained by running tests after adding or modifying a physical twin node within one or more other twin nodes, and validating the added or modified physical twin node or emulated distributed system if the baseline results match the additional results. 17. A test platform for a distributed system, comprising: a plurality of processing systems, each configured to operate one or more of a plurality of virtual nodes; a network coupled to the plurality of processing systems and configured to communicatively couple the plurality of virtual nodes such that the plurality of virtual nodes are configured to emulate a distributed system used to process and manage distributed immutable data objects (DIDOs); and a test management processor, wherein the test management processor is configured to: initiate testing of the emulated distributed system, the testing including performing one or more transactions on the first DIDO to obtain a modified first DIDO, and validate the emulated distributed system by comparing the modified first DIDO with a second DIDO that has been generated from the first DIDO by a verified system. 18. The testing platform described in clause 17, wherein the test management processor is further configured to validate the virtual node under test if at least one of the one or more transactions is executed by the virtual node under test in the emulated distributed system and the content of the modified first DIDO corresponds to the content of the second DIDO. 19. A testing platform as described in clause 17 or 18, wherein the first DIDO is a copy of a reference DIDO and the second DIDO is obtained by having a verified system perform one or more transactions on an identical copy of the first DIDO, the verified system being a real-world system, and the reference DIDO, the second DIDO, and the one or more transactions are copied from the real-world system at the corresponding recording time. 20. A test platform as described in any one of clauses 17 to 19, wherein the test management processor is further configured to generate baseline results for the test plan by running tests where each virtual node in the emulated distributed system includes a verified node, include a twin node in the emulated distributed system, where the twin node is addressed using an identifier shared by one or more other twin nodes in the emulated distributed system, compare the baseline results with additional results obtained by running tests after adding or modifying the virtual twin node, and verify at least one added or modified twin node or the emulated distributed system if the baseline results match the additional results.

[0093] The foregoing description is provided to enable one of ordinary skill in the art to practice the various embodiments described herein. Various modifications to those embodiments will readily occur to those skilled in the art, and the general principles defined herein may be applied to other embodiments. Thus, the claims are not intended to be limited to the embodiments shown herein, but are to be accorded the full scope consistent with the claims in their language, and reference to an element in the singular is intended to mean "one or more" and not "only one" unless otherwise specified. The term "some" refers to one or more, unless otherwise specified. All structural and functional equivalents known or later known to those of ordinary skill in the art to the elements of the various embodiments described throughout this disclosure are expressly incorporated herein by reference and are intended to be encompassed by the claims. Furthermore, nothing disclosed herein is intended to be made public, whether or not such disclosure is expressly recited in the claims. No element of a claim shall be construed under 35 U.S.C. ...

Claims

1. A method for testing nodes in a distributed system, performed by one or more communicatively coupled processing systems, comprising: configuring a plurality of virtual nodes to emulate a distributed system used to process and manage distributed immutable data objects (DIDOs) within said distributed system; initiating testing of the emulated distributed system with a test environment, the testing including executing one or more transactions against a first DIDO to obtain a modified first DIDO in the emulated distributed system; validating the emulated distributed system by comparing the modified first DIDO with a second DIDO generated from the first DIDO by a verified system using the test environment; generating a baseline result of the test with the test environment by executing the test where each virtual node in the emulated distributed system includes a verified node; comparing the baseline results with additional results obtained by running the test after adding or modifying at least one virtual node in the emulated distributed system using the test environment; validating the added or modified at least one virtual node or the emulated distributed system with the test environment if the baseline results match the added results; A method comprising:

2. The method of claim 1 , wherein at least one of the one or more transactions is executed by a node under test in the emulated distributed system.

3. Validating the node under test if the content of the modified first DIDO corresponds to the content of the second DIDO. The method of claim 2 further comprising:

4. 2. The method of claim 1, wherein the first DIDO is a copy of a reference DIDO, and the second DIDO is obtained by having the verified system perform the one or more transactions on an identical copy of the first DIDO.

5. 5. The method of claim 4, wherein the reference DIDO includes a first distributed ledger configured to secure one or more transaction records, the second DIDO includes a second distributed ledger, and the modified first DIDO includes a third distributed ledger, each of which is kept independently.

6. 6. The method of claim 5, wherein the modified first DIDO content corresponds to the second DIDO content when the decryption information in the second distributed ledger matches corresponding decryption information in the third distributed ledger.

7. 5. The method of claim 4, wherein the verified system is a real-world system, and the reference DIDO, the second DIDO, and the one or more transactions are copied from the real-world system at corresponding recording points in time.

8. configuring a starting point of a test timeline for executing a test plan using the emulated distributed system, the one or more transactions being defined by the test plan; The method of claim 1 further comprising:

9. 9. The method of claim 8, wherein in a first mode of operation the test timeline runs at a rate consistent with real time and in a second mode of operation the test timeline runs at a rate different from real time.

10. and resuming the test using an unmodified version of the first DIDO after resetting the test timeline to the starting point. The method of claim 8 further comprising:

11. configuring the test plan to cause at least one virtual node to modify a distributed ledger of the first DIDO, the modifying the distributed ledger including using time-dependent cryptography to encrypt information maintained by the distributed ledger according to the test timeline; The method of claim 8 further comprising:

12. configuring a plurality of physical nodes to implement various virtual nodes within the plurality of virtual nodes, the plurality of physical nodes including physical nodes that are geographically distant from one another; The method of claim 1 further comprising:

13. The method of claim 1 , wherein at least one node of the plurality of virtual nodes comprises a container.

14. including a virtual twin node within the emulated distributed system, the virtual twin node being addressed using an identifier shared by one or more other twin nodes within the emulated distributed system; Comparing the additional results obtained by performing the test after adding or modifying the virtual twin node with the baseline results; verifying the added or modified virtual twin node or the emulated distributed system if the baseline result matches the additional result obtained by adding or modifying the virtual twin node and then running the test; The method of claim 1 further comprising:

15. including a physical twin node in the emulated distributed system, the physical twin node being addressed using an identifier shared by one or more other twin nodes in the emulated distributed system; comparing the additional results obtained by performing the test after adding or modifying the physical twin node with the baseline results; verifying the added or modified physical twin node or the emulated distributed system if the baseline result matches the additional result obtained by performing the test after adding or modifying the physical twin node; The method of claim 1 further comprising:

Images