Systems and methods for decentralized autonomous processing and / or control

Abstract

A system and method are provided for interactions between processors and devices to provides stigmergy among the processors without processors-processors communication. The processors execute steps in a script in response to receiving an event message from a device, and the processors transmit operation messages encoding first state information and / or operations generated by executing a step of the script. The devices generate event messages encoding data representing second state information and / or representing acknowledgments. The devices also perform operations indicated in the operation messages received from the processors. Each of the devices multicasts the event messages to all reachable processors (e.g., the subset of processors that ate communicably connected with the device), and the second state information in the multicast event messages provides stigmergy among the processors without communication among the processors.

Description

SYSTEMS AND METHODS FOR DECENTRALIZED AUTONOMOUS PROCESSING AND / OR CONTROL[1] The patent application relates to systems and methods for a distributed, decentralized resilient system with respective controllers acting independently and autonomously. The controllers autonomously send messages to the devices, and devices multicast messages to the respective controllers to ensure the controllers act in concert, providing stigmergy throughout the system.CROSS-REFERENCE TO RELATED APPLICATIONS[2] This application claims priority to U.S. provisional application number 63 / 519,499, filed on August 14, 2023, which is expressly incorporated by reference herein in its entirety.BACKGROUND[3] Society's dependence on control systems is growing with the advent of smart cities and smart homes, industrial processes, the Internet of Things, and space missions. Such systems drive the need for highly available, highly reliable, and resilient control systems. Traditional control systems have often relied on centralized architectures, where a single control unit manages the operational aspects of a system. While effective in certain applications, centralized systems can suffer from limitations related to scalability, flexibility, and resilience.[4] Resilience can be provided by employing a distributed control system (DCS) in which multiple controllers are distributed throughout the system. One of the controllers is elected to be the active controller and when it fails another controller will be elected to replace it in controlling operations..[5] More sophisticated DCS disperses control functions across multiple control units, often termed as "controllers" or "nodes," that divide the workload among them, and communicate and collaborate to manage the overall system. This distributed approach provides advantages such as improved fault tolerance, reduced risk of single points of failure, and enhanced scalability. However, managing and coordinating these distributed elements requires a highly redundant and highly available controllers, wired network, and infrastructure. The network must be a strongly connected and always available network, which becomes expensive and complex to install and maintain, particularly as the number of nodes increases. Even such a DCS will fail if the network fails and it is subject to degradation of service during failovers, when a controller fails.[6] Supervisory control and data acquisition (SCADA) is related to DCS. SCADA is a control system architecture comprising computers, networked data communications and graphical userinterfaces for high-level supervision of machines and processes. It also covers sensors and other devices, such as programmable logic controllers, which interface with process plant or machinery. The operator interfaces enable the monitoring and issuing of process commands. For example, controller setpoint changes are handled through the SCADA computer system.[7] The subordinated operations, e.g. the real-time control logic or controller calculations, are performed by networked modules connected to the field sensors and actuators. Today the functionality of SCADA and DCS systems are very similar, but DCS tends to be used on large continuous process plants where high reliability and security is important, and the control room is not geographically remote. Many machine control systems exhibit similar properties as plant and process control systems do.[8] Some of the key challenges in distributed control systems is achieving smooth and reasonably short failovers and coherent operations upon network partitions and other natural perturbations. Unfortunately, the detection of the need for a failover, and the failover itself do take time. If the system is configured to respond to it too fast, the system may degrade to frequent flipflop of control, which will degrade its performance and reliability. And numerous network perturbations such as a network partition and electronic interferences, can lead to operational inconsistencies, duplicate actuations, and coordination issues (which may render failovers impossible) if not managed properly. In DCS, managing such periods properly often means that the control system identifies this state and deliberately renders its services unavailable or degraded, for example, to prevent duplicate actuation and maintaining of inconsistent state by controllers in the different network partitions.[9] In view of the above, the advantages can be appreciated for a wired, highly redundant (including network switches and other infrastructure, and powerful reliable computers as controllers) and strongly connected network. However, there are numerous operating environments where redundant and strongly connected networks are impractical, or impossible. For example, we cannot expect redundant and wired networks and infrastructure to be installed in smart homes. When physical space or weight of wires is a design consideration, we will often consider wireless communication. However, by nature, wireless is subject to electronic and environmental interference and generally renders inconsistent quality of service. At homes, offices, or industrial environments - building materials and passing objects and people impose additional interference on wireless networks. In homes, the predominant network is Wi-Fi, which is neither wired nor redundant. In medical settings, we cannot expect ambulatory patients to carry redundant equipment.Wireless, and therefore unreliable network, is the appropriate network in these use cases. Physical space considerations and moving machinery may preclude laying out redundant cables or a wired network, which is the case in manufacturing and other plants. For control systems installed in remote areas, wireless may be the practical choice. Wireless may be the correct choice for a site to remain accessible for command and control after a natural disaster because wires may be severed. In space missions, weight consideration may render redundant or wired infrastructure unattainable.

[0010] While distributed control systems offer significant advantages there remains a need for an improved approach that is not susceptible to failures in the communication network between the controllers.BRIEF DESCRIPTION OF THE SEVERAL VIEWS OF THE DRAWINGS

[0011] In order to describe the manner in which the above-recited and other advantages and features of the disclosure can be obtained, a more particular description of the principles briefly described above will be rendered by reference to specific embodiments thereof which are illustrated in the appended drawings. Understanding that these drawings depict only exemplary embodiments of the disclosure and are not therefore to be considered to be limiting of its scope, the principles herein are described and explained with additional specificity and detail through the use of the accompanying drawings in which:

[0012] FIG. 1 A illustrates a distributed controller-device system that requires a network between the controllers.

[0013] FIG. IB illustrates a decentralized controller-device system that is autonomous without using a controller network (CNET) in accordance with some embodiments.

[0014] FIG. 2A illustrates an example of an automation system and script in accordance with some embodiments.

[0015] FIG. 2B illustrates another example of an automation system and script in accordance with some embodiments.

[0016] FIG. 3 illustrates event messages and operation messages in accordance with some embodiments.

[0017] FIG. 4A illustrates an automation script exemplifying the steps corresponding to the events that occur in the system as shown in FIG. 4B in accordance with some embodiments.

[0018] FIG. 4B illustrates an example of events that occur in the environment, triggering the automation script of FIG. 4A, and the flow of execution among controllers in accordance with some embodiments.

[0019] FIG. 5A illustrates an example of a script / manifest in accordance with some embodiments.

[0020] FIG. 5B illustrates an example of an event message in accordance with some embodiments. Rl] FIG. 6 illustrates a flow diagram of an example of a method for obtaining a shared constant value in accordance with some embodiments.

[0022] FIG. 7 illustrates a flow diagram of an example of an alternative method for obtaining a shared constant value in accordance with some embodiments.

[0023] FIG. 8 illustrates a plot of a the probability as a function of time that an object tagged with the value v is still in the system, and the probability that v is regenerated in accordance with some embodiments.

[0024] FIG. 9 illustrates a flow diagram of an example of a method for or executing a script in a Decentralized and Autonomous Control (DAC) system in accordance with some embodiments.

[0025] FIG. 10A illustrates the knowledge-horizon problem of distributed systems, illustrating that controllers may make control decisions using stale datum in accordance with some embodiments.

[0026] FIG. 10B illustrates how processors can autonomously determine if a datum is fresh in a DAC system in accordance with some embodiments.

[0027] FIG. 11 illustrates a computing device in accordance with some embodiments.DESCRIPTION OF EXAMPLE EMBODIMENTS

[0028] Various embodiments of the disclosure are discussed in detail below. While specific implementations are discussed, it should be understood that this is done for illustration purposes only. A person skilled in the relevant art will recognize that other components and configurations may be used without parting from the spirit and scope of the disclosure.OVERVIEW

[0029] In some aspects, the techniques described herein relate to a system including: processors configured to execute, in response to receiving an event message, one or more steps in a script, and configured to transmit operation messages encoding first state information and / or operations generated by executing a step of the script; devices configured to generate event messages encodingdata representing second state information and / or representing acknowledgments, and the devices being configured to perform (i) one or more tasks and / or (ii) operations of one or more operation messages received from one or more of the processors, wherein, for each of the devices, a device multicasts the event messages including the second state information to a subset of the processors that is communicably connected with the device, and the second state information provides stigmergy among the processors without communication among the processors.

[0030] In some aspects, the techniques described herein relate to a method including: receiving, at respective processors, event messages that encode data representing second state information; executing, by the processors, instructions of steps in a script; transmitting, from the processor, operation messages encoding first state information and / or operations generated by executing the respective instructions; generating, at respective devices, the event messages, wherein some of the event messages encode acknowledgments; performing, at the devices, (i) one or more tasks and / or (ii) operations of one or more operation messages received from one or more of the processors, wherein, for each of the devices, a device multicasts the event messages including the second state information to a subset of the processors that is communicably connected with the device, and the second state information provides stigmergy among the processors without communication among the processors.EXAMPLE EMBODIMENTS

[0031] Additional features and advantages of the disclosure will be set forth in the description which follows, and in part will be obvious from the description, or can be learned by practice of the herein disclosed principles. The features and advantages of the disclosure can be realized and obtained by means of the instruments and combinations particularly pointed out in the appended claims. These and other features of the disclosure will become more fully apparent from the following description and appended claims, or can be learned by the practice of the principles set forth herein. Systems and methods disclosed herein.132] The disclosed technology addresses the need in the art for robust and resilient systems (e.g., control systems) that continue to function properly when part of the system fails. A new paradigm that will provide redundancy of routes from devices to controllers while endowing data coherency, and redundancy of control by allowing plurality of controllers to operate autonomously while preventing duplicate actuation is needed to satisfy the requirements of our modern societal systems. Using a decentralized system can mitigate vulnerabilities in the system due to the failure ofcontrollers and segments of the communication subsystem. We describe a decentralized system of autonomous processors, that does not depend on a network or coordination among the processors. We provide the mechanisms for organizing the functioning of discrete controllers that ensures lockstep operation, operational and data consistency and coherency, and prevents duplicate actuations without direct communication among the controllers. This property of organizing the behavior of the system without communication among the controllers or centralized control of the controllers is referred to as “stigmergy.” Stigmergy is more commonly observed in biological systems, and stigmergy is a mechanism of indirect coordination, through the environment, between agents or actions. According to certain non-limiting examples, the systems and methods disclosed herein realize stigmergy among the controllers via shared values. For example, the devices can determine these shared values (e.g., based on operation messages received from one or more controllers), and the devices can multicast messages, which include the shared values, to the controllers, and the shared values are then used by the controllers, autonomously and independently, to determine which step in a script to execute / run.

[0033] The above-described features provide significant advantages by transforming s system that depend on each other into a parallel system, with much lower probability of failure.

[0034] According to certain non-limiting examples, controllers or processes distributed for fault tolerance strive to share very little, allowing them to fail independently. Lowering the dependence on shared resources such as network switches and power sources and the desire to have the processes send messages from diverse provenances toward target devices over diverse routes to improve the redundancies using spatial dispersal. Spatially decentralized autonomous controllers that send copies of redundant messages over diverse routes, further endow temporal diversity that may ameliorate electronic interference affecting the network. This property is provided because the messages traverse routes of different lengths. Although it might seem surprising that discrete, autonomous controllers can individually generate the same value without having to communicate between the controllers, the systems and methods disclosed herein show how such equal values (referred to as shared values) can be constructed without communication or coordination among the processes. These shared values provide a mechanism for lockstep operations among the controllers while the controllers maintain autonomy and independence.

[0035] The systems and methods disclosed herein enable flexibility regarding which processors / controllers participate in the respective steps in a script. A processor / controller may fail and recover and rejoin in the execution of a script without coordinating with the otherprocessors / controllers when the processor / controller rejoins. Note, the term “processor” and the term “controller” can be used interchangeably. A recovering processor does not need approval to join, nor does the processor need to query the active cohort of processes for the latest values already used by the cohort of processors. Yet, a rejoined process can construct a value equal to the other processes without coordinating with them.

[0036] As an example, consider the following design of a decentralized control system that runs an automation script in response to an impetus event. Different from previous topologies that require a network among the controllers, using the systems and methods disclosed herein, the network among the controllers (CNET) can be eliminated. In such a control system, a script is invoked when an event arrives at a controller process. For resiliency, all controllers that receive the event may invoke the script. The participating controllers process the event and send a command embedded in a message that includes a command (or message) ID to the target device. It is required that without coordination, all participating processes will include the same ID in their message.

[0037] The above-noted technique provides processors and communication redundancies. Multiple controllers are active; each message traverses diverse routes towards the target device. Because all copies of the redundant command carry the same ID, the receiving device can use it to execute only one (or use however many it needs for a purpose). After a command executes, the device multicasts a notification message (e.g., an acknowledgment (ACK) message) to a subset or all controllers to notify the controllers regarding what command was executed. For example, the device may only be able to communicate with a subset of all controllers, and therefore the device is limited to multicasting the notification message to only the subset of all controllers with which the device can communicate. The systems and methods disclosed herein do not require that the subset of controllers be the same subset that sent the command, and all controllers that receive the ACK use the ACK as an impetus to run the subsequent automation step and use and the information communicated in the ACK (e.g., state information) to select which step of the script to perform as the subsequent automation step. The cycle continues as the specific automation script prescribes, and each batch of redundant commands (corresponding to each automation step) carries an ID identifying the batch without direct communication or coordination among the participating processors.

[0038] A challenge of the systems and methods disclosed herein is how the processors can construct a shared value (such as an ID) without coordination. A non-limiting example of how this is achieved is provided here with additional details and examples being provided below.

[0039] Setup: In the case of a control system, automation scripts can be fixed objects. These automation scripts can be made available to all controllers that may participate. If a script changes, it is updated on all controllers. Each script can be identified by a Script ID (sid), and each step in a script can be identified by a Script Function Pointer (sjp). For example, in FIG. 2A and FIG. 2B, which are discussed below, the Script IDs are sid="gentle wake up script" and sid="natural gas detection script", respectively. Further, in FIG. 2A and FIG. 2B, the Script Function Pointers aresfP= ffi, f js} and sjp=tfi,f2}, respectively.

[0040] On-going use (the shared value method): A participating processor receives an ACK identifying the command that was executed. From this identification, controllers independently construct a Program Counter (PC) equal to the 2-tuple <sid, sfp>. This PC is the thought-after shared value because for a given ACK, each of the controllers independently constructed the same PC. From the PC the controllers determine the subsequent control step, and those controllers that successfully complete it, send a command to the subsequent target device. The PC of the subsequent command is included in the messages that are sent from participating processors to a target device. Even though the processors do not communicate among themselves (i.e., no processor-to-processor communications), the PC that is sent with commands and ACK, enables the processors to operate in lockstep, thereby effectively satisfying the functionality that processor-to-processor communications and coordination would often be used to provide. For example, the PC can be used to uniquely identify a command and has the same value no matter which controller constructed it. Here, program counters (PCs) are being used to provide a novel utility. Whereas other methods would rely on a consensus protocol to agree on an same value for an ID of a command (for example) that is sent from more than one controller, the systems and methods disclosed herein can use the PC generation process to have the processors each autonomously and independently generate the same value. Thus, a consensus protocol is not required.

[0041] All copies of the redundant batch of commands carry the same PC; therefore, the target device can decide to execute any of the commands and ignore the others. After it executes what is prescribed in the message, the target device includes the PC as part of its ACK. Any process receiving the ACK can use it to obtain another unique value (the subsequent PC) that identifies the subsequent automation step. Without coordinating with the other processes, a receiving processor can update sjp to point to the next automation step, forming a new unique identifying value for the next command batch.

[0042] According to certain non-limiting examples, when a processor fails and later recovers and receives an ACK, the processor is immediately privy to the latest PC from which it may construct the subsequent value and thereby rejoin and contribute to the system operations. Therefore, no coordination with the other processes is needed to rejoin the active cohort of processors. A newly joined processor is not required to learn which IDs were used while it was not operational. It also does not need additional information about the next step in the automation, which otherwise would require additional communication with the other controllers to learn for each of the running scripts, which steps were already performed. With such a method, a freshly rejoined process may participate in the next automation step without synchronizing or communicating with the other processes.

[0043] According to certain non-limiting examples, even though the PC can uniquely identify a step of a script, another value might be relied on to distinguish a command prepared by a processor during run r of the script and another command prepared by the processor during a subsequent run r'. When such a qualification is desired, a run number can be added to the tuple, such that the PC becomes the three-tuple <r, sid, sp>.

[0044] A new nomenclature is needed to distinguish from otherPCs a new nomenclature is used herein by collectively naming the two-tuple <sid, sjp> and the three-tuple <r, sid, sfp> as Distributed Program Counter (DPC). In general, as we establish more methods in the following sections, the Distributed Reference, DR.

[0045] Another challenge is obtaining the same r value from autonomous. To address this challenge, a method is presented to construct a shared ID from a constant source (the automation script). The following method shows how decentralized and autonomous processes can obtain an equal value from dynamic sources.

[0046] One suitable fountain and corresponding methods producing shared values become available when considering the attributes of a received impetus or event. For example, one may use the event's time. Such an event may arrive from some sensor that measures a change in its environment or another event source. When the sensor generates the event (and similarly other event sources), it timestamps the event with its local time is or uses an arbitrary time-like counter. It then multicasts the event to the processes. Emanating from a single source, it follows that all receiving processes see the same value, ts and can set r = ts.

[0047] A Decentralized and Autonomous Control (DAC) system can have several advantages over systems that rely on a leader to coordinate the actions of the controllers. Con sider, e g., systems that use external or dynamic sources of values, such as systems that elect a leader to issue an ID (or othershared values). The leader multicasts a value it determines to the participating processes or allows them to query it for the value. The leader, however, is a single point of failure. This is correct, even if the system maintains plurality of backup leaders, because to enable failover the leader must synchronize its state (e.g. which IDs it issued) with all backup computers, which requires the network, and it is not always available. Such an architecture imposes superfluous communication and coordination on the system, subjecting the leader-based system to various problems as discussed above. The vulnerabilities of the leader-based system are exacerbated if multiple processes are active, complicating the system's implementation and lowering its resiliency, availability, and reliability. Compared to such architectures, the systems and methods disclosed herein improve all properties: resiliency, availability, and reliability. To see that,, an impetus must arrive at a processor to trigger any work being performed. Nevertheless, using the systems and methods disclosed herein, no coordination and communication between the processes is required, no query for an ID is needed, and there is no single point of failure.

[0048] A used herein the term “configured” shall be considered to interchangeably be used to refer to configured and configurable, unless the term “configurable” is explicitly used to distinguish from “configured”. The proper understanding of the term will be apparent to persons of ordinary skill in the art in the context in which the term is used.

[0049] A used herein the term “shared value" refers to a shared equal autonomous value , which is a value produced when needed (when there is an impetus to construct it). The terms shared and equal indicate that all distributed processes that produce a value construct the same equal value. The term autonomous indicates that each process independently produced it without coordination or communication with the other processes.

[0050] A used herein the term “timestamp" refers to a data field that increases in value at a constant frequency. Oxford Dictionary defines the noun as “a digital record of the time of occurrence of a particular event” and the verb “record the time or date of’.

[0051] A used herein the term “manifest" refers to a list of objects (or equally, operations and other artifacts), and optional metadata such as the objects descriptions, events associated with the manifest {et, natural and data dependent sequences {G} of the objects listed in the manifest, objects traversal rules OTR (given a reference to an object, a rule describes how to obtain a reference to the next object), last access time, and other data elements applicable to the domain described by the manifest. A manifest has an ID that uniquely qualifies it in the system. Each object in a manifest has an ID that uniquely qualifies it in the manifest. Manifests can be immutable or have a sporadic rateof update. Updated manifests are made available to all controllers. The natural sequence of objects in a manifest may be implicitly defined by their listing order or be explicitly enumerated. A system may have one or more manifests.

[0052] A used herein the terms “impetus" and "event" are interchangeably and refer to an event that causes something to occur. Events have a value (that could be NULL , an explicit (direct) or implicit (indirect) reference oID to an object in a manifest, optional metadata such as a timestamp ef, association with a manifest mID, and event source. An example of an explicit reference to an object oID is an event that carries an object ID, and implicit reference is a NULL oID referring to the first object in the manifest (implicit rules that reference the zzth object are usable, and each system may define them to satisfy its specific requirement)

[0053] A used herein the term “P / v / " refers to the probability that a value v is generated by an event source at time t. When v is drawn from a probability distribution, such as a Normal distribution, P[v] increases over time as more values are drawn.

[0054] A used herein the term “P[objv]" refers to the probability that the value v is used to reference an object or an instance of an object at time t. In many systems, P[objv] lowers as time passes because objects referenced in the past are no longer pertinent or are discarded.

[0055] A used herein the term “fountain of values (FOV)" refers to a source from which shared values can be obtained or derived. Examples of fountains are a Manifest (defined above), or objects that naturally exist in the system (different systems have different objects depending on its functionality, but generally we assume that any system that does useful work is operating using one or another form of objects. Concrete examples will be discussed in detail in the specifications that follow).

[0056] FIG. 1 A illustrates an example embodiment of system 100 in which a network or central memory (e.g., network infrastructure 106 and CNET 108) is used to coordinate operations among the various controllers (e.g., controller 104a, controller 104b, and controller 104c, which are collectively referred to as controllers 104). The controllers 104 can control processes and measurements among the devices of system 100 (e.g., device 102a, device 102b, device 102c, device 102d, device 102e, device 102f, device 102g, device 102h, device 102i, and device 102j, which are collectively referred to as devices 102).

[0057] A controller network (e.g., CNET 108) provides communication between the controllers 104. A controllers-devices network (e.g., CDNET 110) provides communication between the controllers 104 and the devices 102. The controllers communicate using CNET 108, and controllers-devicescommunication occurs using CDNET 110. CNET 108 and CDNET 110 can be the same type of network or different types of networks. For example, both CNET 108 and CDNET 110 can be a Local Area Network (LAN), or one can be a LAN such as Ethernet and the other a Low Power Lossy Network (LLN) such as IEEE 1102.15.4.

[0058] System 100 can include a cohort of distributed processors and devices that can communicate by sending messages (e.g., messages can be sent using a network). Additionally or alternatively, processors and devices that can communicate by saving and retrieving data from a shared electronic memory, for example. The embodiment of system 100 in FIG. 1A includes a network (e.g., CNET 108) or shared memory providing communication among the processors. In the figure, the example system provides distributed control. The processors are controllers, and the devices are sensors and actuators.

[0059] The embodiment of system 100 illustrated in FIG. 1 A can be implemented as primarybackup distributed systems. In a primary-backup distributed system, a spatially distributed group of controllers elect one leader responsible for making or approving operations. To enable failovers, the state of the system state is continuously synchronized among the cohort of controllers 104 using CNET 108 and other network infrastructure 106. To the extent that the network was available to keep the synchronization up to date, and if the failover and transfer of control were successful, the system achieves the desired characteristics (one or more of high availability, reliability, and resiliency to failures). Such systems critically depend on the CNET availability, bandwidth, and continuous, reliable operation for state synchronization and other coordination. In practice, these requirements from the network are addressed by using redundant wired networks and redundant enabling components such as network switches. Primary-backup systems may degrade functionality during network partitions because no controller knows what controllers in the other partitions are doing, nor does it know whether the leader has failed or is in another partition. The leader may not be able to operate because it does not know whether the network partition was interpreted as a leader failure by controllers in one of the partitions, which may result in a new leader election in that partition and duplication of control. Furthermore, upon CNET failures, synchronization ceases, so failovers are not possible.

[0060] The embodiment of system 100 illustrated in FIG. 1A can be implemented as load-balanced distributed systems. In a load-balanced distributed system, work is divided among the controllers. Such load-balanced systems are not fault-tolerant at the system level, but when one of the active controllers fails, the failure is limited to the part of the system controlled / affected by the failingcontroller. For example, some load-balanced distributed system can forward traffic designated to a failed controller to another controller. Notably, the resilience in load-balanced distributed systems depends on and assumes that CNET 108 and network infrastructure 106 do not fail because CNET 108 and all of the network switches and other infrastructure 106 are relied on, for the above-noted forwarding functionality. Any failure in 106 or network infrastructure 108 may prevent the traffic originally designated to a failed controller to be forwarded to another controller. That is, load- balanced systems are dependent on CNET 108. And if CNET 108 does not operate correctly, then upon each controller failure, all devices connected to that controller fail. Furthermore, with each controller failure, the increased load makes the failure of additional controllers likely. Moreover, the entire apparatus becomes dependent on the forwarder, which may itself fail.

[0061] The embodiment of system 100 illustrated in FIG. 1A can be implemented as multi-primary distributed systems. Multi-primary distributed systems provide a higher degree of sophistication than the above-discussed distributed systems. For example, a multi-primary distributed system may be used in space missions and similar systems requiring very high fault tolerance. In multi-primary distributed system, a majority of controllers arrive at a consensus on what is the correct data and subsequent operation, at each control step. Once a consensus is reached, one of the controllers may be responsible for that operation. As with the other distributed-system classifications discussed above, multi-primary distributed systems also depend on CNET 108 to function properly (e.g., to reach a consensus and to synchronize data).

[0062] Each of the above-discussed distributed systems depends on the ability of controllers to communicate among themselves using CNET 108 and network infrastructure 106. It is given that the CDNET must be operational as well, because an active controller must also be able to receive data and send commands to the devices it controls, or no progress can be made. The latter leads to an inherent predicament in all classes of known systems, excluding multi-primary. Contrarily, the controllers must be spatially distributed to decrease the probability of related failures. However, in real physical systems, this often leads to controllers too far to be connected to all devices. For example, construction materials and the architectural structure of a plant or a home can prevent controllers from being able to send and receive data from all devices, sensors and actuators.

[0063] Distributed systems that depend on CNET 108 can work well under certain assumptions and for specific theoretical system models. For example, one assumption in these models is that CNET 108 is highly connected among the controllers. Satisfying this assumption requires redundant wired networks (e.g., CNET 108) and redundant infrastructure embodied by network infrastructure 106 tosatisfy model requirements. However, there can be systems and physical deployments where such redundancies are not possible or may be prohibited by the use case. For example, in home automation systems (e g., smart homes) can use Wi-Fi networks, which are not wired nor redundant.

[0064] Further, spatial constraints and moving machinery may preclude laying out redundant cables or a wired network in manufacturing and other plants. For industrial installations in remote areas, wireless may be the practical choice. Wireless may be the correct choice to remain operational after a natural disaster because wires may be severed. In space missions, weight consideration may render redundant infrastructure unattainable. In those examples and numerous other use cases, a preferable system architecture is illustrated in FIG. IB, which uses distributed controllers sans controller-to- controller coordination and sans data synchronization over CNET 108 and does not require network infrastructure 106.

[0065] FIG. IB illustrates a system 120 that does not depend on CNET 108 nor network infrastructure 106 to function properly. In fact, the embodiment of system 120 operates without infrastructure (e,g„ network infrastructure 106) and without CNET 108. Absent network infrastructure 106 and CNET 108, other mechanism are used to keep the controllers operating in lockstep and ensure the coherency and consistency of the system.

[0066] The controllers 104 are processors and can be referred to as processors, such that the function of the processors is not limited to being used for controlling the devices. For example, the topology of system 120 can be used as a communication system, rather than a control system, and the processors (e.g., controllers 104) can be communication processors, rather than control processors, as would be understood by a person of ordinary skill in the art. For clarity of discourse, thenon-limiting example of a control system is used to illustrate aspects of the systems and methods disclosed herein without always explicitly specifying it, but the scope of the disclosure is not limited to control systems.

[0067] Each of the above-discussed distributed systems depend on CNET 108 to enable the controllers to communicate among themselves, making them susceptible to CNET 108 and 106 failures. In contrast, the systems and methods disclosed herein that use system 120, are more robust / resilient than those using system 100 because the systems and methods disclosed herein do not depend on CNET 108 nor network infrastructure 106. System 120 has the advantage that it only depends on CDNET 110 being operational. Further, there are several reasons that a channel in a communication network might fail. For example, in wireless networks that depend on line-of-sight communication, or low power networks, channels are susceptible to multipath interference andobstructions. In the decentralized control system and methods we describe here, even when failures of one or more links in CDNET 110 prevent communication between a device and one of the controllers, the device may nevertheless still be in communication with another of the controllers, allowing system 120 to continue functioning in spite of a partial failure of CDNET 110. In contrast, failures of CNET 108 with respect to communication between controllers 104 can have a much more detrimental impact.

[0068] It is notable that none of the controllers directly communicate with each other. In FIG. IB, devices 102 may directly communicate with at least one controller, and can also relay / pass through signals through other devices and controllers, but the devices do not necessarily relay messages originating one controller to another controller.

[0069] The systems and methods disclosed herein use system 120 to realize Decentralized and Autonomous Control (DAC), which provides control of system 120 without requiring control ler-to- controller communication. DAC may use the CNET 108 for auxiliary services but does not need it for its core services. Consequently, it continues to provide very high availability and reliability when CNET 108 falters. According to certain non-limiting examples, CNET 108 can be relinquished in system 120. In the DAC paradigm, control is decentralized, because the controllers operate autonomously and do not require coordination with the other controllers. At any given time, a subset of DAC controllers may be active in response to network dynamics, interference, and controller and CDNET 110 perturbations and failures. Consequently, control operations are decentralized and not only physically distributed. DAC allows multiple controllers to contribute to running automation scripts at opportune points during the script run and control flows among the controllers to attain high availability and reliability. The resilience of DAC to failures is innate because more than one controller can participate in each control operation. This positive redundancy may lead to duplicate actuation of devices, and a coordination-less, Universal Idempotent Scheme (UIS) is presented to prevent failure.

[0070] Many applications benefit from high availability, reliability, and failure resiliency. One or more of these characteristics are often required in simple to complex systems, such as home automation (e.g., a smart home), industrial control, and space missions. For example, high availability, reliability, and failure resiliency can be attained using the distributed system illustrated in FIG. IB.

[0071] As discussed above, the system architecture is illustrated in FIG. IB can provide controller redundancy in cases where communication among the controllers cannot be guaranteed, is not plausible, or is impossible.

[0072] There are many use cases and systems where the system architecture is illustrated in FIG. IB provides advantages over other system architectures. Consider For example, the non-limiting use case of a smart home as an exemplary reference system. Although not limited to this example, the smart-home example is used extensively to illustrate aspects of the systems and methods disclosed herein. In addition to the smart-home example, a person of ordinary skill in the art will understand that the disclosed aspects in the context of the smart-home example can be straightforwardly generalized to other applications. Thus, the smart-home example is not limiting.

[0073] As supervisors of considerable home functions, smart home systems are required to provide continuous availability and swift response to spontaneous events. Robust and reliable functioning of a smart home system is required if we wish to provide a safety critical operation and not only operate light bulbs. Psychologically, for example, the smart home is a physical system in front of the user's eyes, and the user will find it undesirable when an event occurs, or they tap a button, but devices do not actuate. Nor does the system appear smart or reliable compared to users’ experience with standard physical switches.

[0074] In many systems, safety depends on control operations not being missed. Numerous reasons, however, may render the home controller unavailable, transiently, or for an extensive duration. The system design must recognize that controllers will fail, disconnect from the network, lose power, or restart for multiple reasons. Redundant controllers with failover may provide some relief, and the setup of such a system appears in FIG. 1 A. Controllers are spatially dispersed, so more than one controller can interact with each device via CDNET 110. CDNET 110 can be, e.g., a Low Power IP mesh network. CNET 108 in a smart home predominantly uses Wi-Fi.

[0075] The operating environment at home, however, is surprisingly harsh and poses an abundance of challenges. Despite significant advances in CDNET technologies, these networks are sensitive to the home structure and building materials, passing objects, and the presence of people and are susceptible to electronic interference and nearby networks. Environmental and electronic perturbations may sever network routes, partition subsets of nodes, and the network may lose messages. Multiple empirical studies show that a CDNET failure rate is between 0.01 to 0.20 in such operating environments. An empirical study of the modem Thread protocol showed that its reliability is near 99% when it is available. However, its average availability is only close to 88%,with 6% up and down fluctuations. Empirical studies have shown that these characteristics make achieving high reliability at the systems level challenging.

[0076] The CNET, which today is predominantly Wi-Fi, is also subject to transient interferences and periods of unavailability. Significantly, Wi-Fi may fail over extended periods in residential settings because there is no redundancy nor resources to maintain high-availability solutions such as those deployed in enterprise Wi-Fi. When the CNET is unavailable, a smart home system that requires coordination among the controllers will not work. Even a distributed system explicitly designed to cope with CNET partitioning must degrade to interacting with only some devices. Moreover, distributed controllers cannot synchronize the state during these periods, which is necessary to allow failovers. Classical distributed computing commonly solves this by employing redundant infrastructure and reliable networks. Unfortunately, such infrastructure is absent in typical homes.

[0077] Thus, in many use cases, a fundamental mismatch exists between the de-facto infrastructure and operating conditions and what a distributed system requires to operate. This fundamental disparity is present in numerous systems and deployments. Similar to the smart-home example discussed above, this fundamental disparity is also present for chemical plants, industrial plants, Internet of Things (loT) applications, and space exploration applications.

[0078] The systems and methods disclosed herein provide Decentralized Autonomous Control (DAC). CDNET 110 can use a low-power and lossy network (LLN). Although LLNs may lose messages and may be susceptible to network interference and people in the vicinity. DAC addresses these fragilities by having each device send its messages to its set of reachable controllers. Because the controllers are already spatially distributed this communication pattern causes the messages to traverse diverse routes toward the controller, and can ameliorate the CDNET brittleness.

[0079] DAC’s design is principled in a decentralized architecture in which distributed controllers operate autonomously and in parallel (with favorable timeshift). Controllers may join the run of an automation script at will, leave, and rejoin as execution flows between controllers in response to environmental and network dynamics. Each device provides route redundancy by communicating with its subset of reachable controllers. In turn, the decentralized controllers that receive data provide redundancy by participating when they have sufficient data to make the next control step. The communication pattern is that each device multicasts its data to its subset of reachable controllers, and all controllers that decide to participate in this control step send redundant commands to the next target device.

[0080] Message redundancy, however, concludes its utility after a command executes on a device, and its copies must be ignored. Typically in distributed computing, a protocol can assign this responsibility to the sending controllers or receiving device. Nevertheless, because the CNET is not always available, novel techniques are used with DAC. For example, the novel techniques can effectively coordinate controller operations using a unique identity (ID) to include in the message copies and shifting the responsibility to the device would not be a general solution in the decentralized case.

[0081] The systems and methods disclosed herein address the challenges arising from a decentralized architecture that lacks a predictable CNET. According to certain non-limiting examples, controllers 104 process data with some time shift (the time shift can be beneficial for averting transient interference) and include auxiliary data in the message (e.g., the auxiliary data can include the message generation time or message counter). Only some controllers participate in each control step, so each controller may have a different message counter value and clock variation. Consequently, another mechanism than the message counter value and clock variation from the controller can be used by the device to uniquely identify each message. DAC uses a new coordination-less Universal Idempotent Scheme (UIS) to enable decentralization. The UIS is universal in that the UIS does not impose that devices will be innately idempotent. The UIS works in the decentralized case because it uses a fixed object instead of a dynamically generated value. By way of example and not limitation, one such object that can be utilized is the program counter (PC) of the function that generates the message. It follows that a redundant message set stems from the same PC, regardless of which controllers sent the messages, and there is no need for the controllers to communicate to coordinate to manufacture uniqueness.

[0082] FIG. 2A illustrates an example of an automation process. In the illustration, the input parameters of a function enter it from its left, the output exits on its right or bottom-right, and the trigger to run it enters the dark segment from the top. In this script example, in response to a timed event, the gentle wake-up automation turns the bedroom lights ON at a 20% dim level if it is 07:00, orders breakfast varied by weekday, and brews coffee.

[0083] The process illustrated in 200 includes an automation script 210 that executes step 230, step 232, and step 234. The devices (i.e., device 212, device 214, device 216, device 218, and device 220) generate various event messages, which separately or in combination trigger controllers that receive one or more of these messages to perform the steps of the script. As articulated in thisspecification, using the described methods, any of the steps can execute on different controllers, adding to the system resilience.

[0084] FIG. 2B illustrates an example of an automation process. In the illustration, the input parameters of a function enter it from its left, the output exits on its right or bottom-right, and the trigger to run it enters the dark segment from the top. System 250 performs a script for natural gas detection automation by checking if the gas level datum is over a threshold that requires closing the main gas valve and sounding an alarm.

[0085] System 250 includes devices (i.e., device 260, and device 264) and controller 252, which executes a script that includes step 270 and step 272. The devices (i.e., device 260, and device 264) generate various event messages, which separately or in combination trigger controller 210 to perform steps of the script.

[0086] DAC is a reliable automation orchestration service. Using automation scripts, home occupants, industrial plant operations, and many other control systems benefit from a higher utility, as illustrated in FIG. 2A and 2B. A script has one or more associated invocation event and a sequence of small control steps. It is natural for automation to be triggered by events, such as a sensory event driven by physical phenomena, a value passing over or below a threshold, periodic events, or a user pushing a button. Thereupon, automation may need to run. In the example in FIG. 2A, the invoking event of the gentle wake-up script is an alarm clock, and the natural gas script in FIG. 2B may be invoked by periodic event levels sent from the sensor, or the sensor may send an event when the sensed level surpasses a configurable threshold.

[0087] In DAC, each automation script is divided into fine-grain control steps implemented by functions. The model defines a function as non-blocking, short-running code that may interact with a single device. A function may have 0 to n input parameters, which fully provide its state. These characteristics of function support the migration of execution between the controllers (the flux property). Typically, a function will prepare a command for its target device or conclude that the automation has come to fruition and stop the script run. A function has a default target device, which it may override (unless it is the first function of a script) when it runs. The gentle wake-up script in FIG. 2A has three functions in this example. The first function has one parameter, the time of the event, and it will turn on the bedroom lights at a 20% dim level if the time is 07:00, and otherwise stop the script from running. The second function orders breakfast varied by the day input parameter. The last has no parameters and starts brewing coffee when the ACK is received. The natural gas example script has two functions. The first function may stop the automation if the gaslevel is benign; otherwise, it will turn OFF the main gas valve. The second function sounds the alarm.

[0088] A script is invoked when its invoking event is received. In essence, however, what runs is the first function of the script. The trigger to run any subsequent functions is receiving an acknowledgment message, ACK. Unlike the standard definition of an ACK or NAK belonging to a discrete sender that runs on a sending computer, DAC semantically associates these messages with a script that may run on any controller. As a result, multiple controllers may use the information an ACK carries and then use it as a trigger to run the next function.

[0089] Controllers may contribute to running the next function when they satisfy three conditions: They receive an ACK, have valid data for all of the function parameters, and the function's target device is reachable. In a decentralized system, however, there is difficulty in determining the validity of subsets of data. A function may have multiple parameters, but at most one can arrive with a message from a discrete device. Consequently, the remaining arguments must come from the controller memory or be received in the future. All parameters must be current, but independent event sources, such as sensors, may produce an update at any given time; hence controllers may indefinitely wait for the next update. A controller may query for an update, but another parameter may be outdated when the response is received. The challenge emerges because it takes non-zero time for data to spread across the system, and a decision-making process cannot discern if a device has no update or if an update is on its way. In such situations, we say that knowledge is below the controller knowledge horizon, and DAC solves it with the idea of freshness. Each datum in the systems and methods disclosed herein has a freshness period during which it is valid for use, and controllers may run a function if they can satisfy all its parameters with fresh data.

[0090] In DAC, the subset of controllers that work on a specific function satisfy two control cycle requirements: (1) Controllers must work on the function with a timeshift among them to ameliorate a transient CDNET fragility; (2) they must do so within a reasonably short time window.

[0091] The first requirement is satisfied because controllers are spatially dispersed. However, given a CNET that may not be available to resolve this frequently occurring coordination, it is unclear how the cohort would guarantee they are working in lockstep. Control steps may occur at any given time, but the CNET is not guaranteed to be always available. Since the system model provides that if the message is received, it is received within Amet, it follows that controllers that receive the acknowledgment will start working on the next function within a time window that is no longer than Ainetbetween the first and last controller, satisfying both requirements.

[0092] A CNET that is not always available presents a complex challenge in preventing duplicate actuation. Known techniques, such as having the controllers coordinate a unique id for the message would not work, and having the destination hash the message is not a general solution in a decentralized system. The former is because the CNET may not be available when needed, and the latter is because discrete controllers may include in the message varied auxiliary data, such as a message generation time. DAC obtains a unique command identifier with the new idea of the Distributed Program Counter (DPC) presented here. Since scripts are generally fixed objects that can be made available on all controllers, a reference identifying each script function the DPC is unique and equal among all controllers. It is constructed from the script id and function pointer: DPC = <sid, sfp>. It turned out, however, that the DPC is a necessary component but is insufficient alone for some purposes. It uniquely identifies a script's functions but cannot distinguish a command prepared by that function in run r of the script and the command in a subsequent run r'. Therefore, in those cases, objects that must be uniquely qualified are indexed with a 3 -tuple that includes the run number and DPC: <r, DPC>.

[0093] This DAC paradigm of design gives rise to several desired properties and we provide a short description the reason for these properties is provided:1) Authoring automation scripts using non-blocking, short-running functions support control flow among decentralized controllers.2) The novel idea that acknowledgment messages, ACK, are correlated with scripts (and not with a sender on a particular controller). This may trigger multiple controllers to contribute to the run of automation.3) Using the concept of freshness, controllers determine if events are fresh without using a real-world clock. Neither devices nor controllers are required to have a clock.4) Participating controllers execute functions with temporal diversity but in lockstep within a short time window, enabled without synchronization or coordination. This is not possible leader-based systems without communication and coordination.5) Since more than one controller may work on a control step (a function), each is capable of sending a control command to the function target device. This endows execution redundancy.6) Since the controllers are spatially distributed, this also endows route diversity over a CDNET that is subject to electronic and other interference.7) Each device has diverse routes to more than one controller, forming its set of reachable controllers. A device multicasts its messages to its set of reachable controllers. Because the controllers are spatially distributed the message traverses diverse CDNET links, with varying latencies, which may ameliorate electronic and other interferences, and the fragilities of low power networks (CDNET today or commonly low power and lossy networks, also called LLN).

[0094] FIG. 3 illustrates various type of messages can be sent from the devices and from the controllers / processors. The event messages 302 include data messages 302a, heartbeat messages 302b, occurrence messages 302c, and acknowledgement messages 302d. Each type of event message can include state information of the device (e.g., a tuple representing the step in the script that corresponds to the most recently executed operation). The event messages 302 is multicast to all reachable processors / controllers.

[0095] Operation messages 304 include registration messages 304a and command messages 304b. Operation messages 304 include commands that can be determined by executing a step of the script on a processor / controller, and operation messages 304 includes a tuple encoding state information of the processor / controller (e.g., a tuple representing the step in the script that generated operation messages 304).

[0096] Decentralized and Autonomous Control (DAC) uses a decentralized protocol with corresponding parts of the algorithm running on controllers and devices. Fig. 3 shows the devicecontroller communication patterns. The names of the communication messages may change, split into sub-messages, and messages may be added without loss of applicability of the patent protection we seek.

[0097] When a device wakes up normally, or after it recovers from a crash, it announces its presence by multicasting its configuration and current state as an event. Heartbeats periodically refresh this data during normal operations. A controller receiving either message will confirm its awareness of the device registration by sending a registration message. Depending on their capabilities, devices may send periodic events when a configurable period elapses or a threshold event when a phenomenon the device measures passes a configurable value. After executing a command, devices send an acknowledgment that includes their updated state. All device messages are multicast to its set of reachable controllers. Over time devices may update their set of reachablecontrollers according to the frequency by which they receive registration confirmations and commands from controllers. Complementary to our work, and without limiting the patent protection we seek, other schemes may be used, more or less sophisticated, to keep devices and controllers aware of each other.SCRIPTS INVOCATION

[0098] Starting from event genesis, the steps the protocol follows are described below (the letters d and c to the left of each step depict whether a device or controller performs the step). Participating controllers and a device take the following steps: d : Measures its state as e after normal wake-up or recovery from a crash, or a regular threshold or periodic event e occurs, d : Respectively DevicelnfoMsg or DeviceEventMsg message m that embed e, is multicast to all reachable controllers, c : Controllers that receive m update FE with e data, c : These controllers attempt to invoke all scripts s x.) with X = ecode whose inter-run duration Airdelapsed since 5, last ran. c : Initializes script s, data structure to denote the beginning of a new run. In particular it sets the run number r = 0. c : The controller increments v.sfp and since s, data structure was just initialized, it runs the first function / ; of s,, if it can satisfy its parameters and its target device is reachable. Upon fi exit, the cmd the function generated is sent to the target device, proviso to / exit status beingQ = SEND CMD. d : The target device approves the invocation request (identified by r = 0) if 5 inter-run duration indeed elapsed. The invocation time eot is memorized by the device, and it mints a run number r is minted. One way to mint r is by drawing it from a random distribution that does not repeat the last n numbers (in all cases known to us, it is sufficient to have n 3). d : cmd is executed, and r and the command unique identification tuple <r, sid, sfp> are memorized. The device sends the first acknowledgment ACKi to its reachable controllers.RUNNING SCRIPTS

[0099] After scripts are invoked, participating controllers (which, by DAC design, are allowed to change with each step) will be triggered to run the next function each time an ACK arrives. In other words, after its invocation, s, continues to run, driven by cycles of ACK and cmd can be seen at the bottom of Fig. 3.

[0100] We continue to describe the protocol steps from where we left it. The target device executed the first command, and we commence here upon the receipt of the first acknowledgment, ACKi, by a controller: c : The subset of controllers that receive an ACK update FE with e data, c : On each such controller in the subset, if the run number r in the ACK is new for st on c, the controller latches onto this run and remembers the r value. If r is not new on c, meaning c participated in this runbefore, there are two cases: the ACK is current, or from the past of c. Controller ignores the past ACK; Otherwise, it updates the run information in s, data structure with the following data: the run message counter rmc, script function pointer sfp, and the invoking event time eot. c : Runs the next f of st if it wasn't the last, and conditioned on / exit status Q = SEND CMD, sends the next cmd to / target device, which includes the r value extracted from the ACK. d : Upon receiving the message, the target device is activated. If the command DPC, which is (r, sid, sfp) was not seen by d before, then the cmd is executed, and r and the unique identification tuple are memorized by the device. An ACK is sent, d may receive redundant cmd, but in such cases <r, sid, sfp> is already seen because it was memorized, so it will be rejected as DUP command.

[0101] This cycle continues until one of the functions of st decides to stop its run or the automation runs to completion.

[0102] FIG. 4A and 4B illustrates an example of how devices and processors interact to realize a process, such as an automation process. In example interaction 400, two devices (i.e., device 402 and device 406) interact with processor 412. Processor 412 includes script 404 showing two of its steps / and / . In response to receiving event message 410, processor 412 executes a step of script 404, resulting in operation message 408, which is then sent to device 406. In the operation messages, the processors communicate state information indicating what step / part of the script generated the respective operation message, and the event messages can serve various functions, one of which is to multicast, upon receipt of an operation messages, and ACK message to all processors that the device can reach - not just the processor originating the operation message. The A CK message includes the state information indicating what step / part of the script generated the operation message that triggered the ACK message. This state information ensures the processors that are reachable by the device are informed where in the script the process is, and thus, the processors can participate in a next / sub sequent step of the process by executing the next / sub sequent step of the script. Even if a processor did not participate in the current step, failed and restarted, or is rejoining the process for some other reason, the processor is informed by the state information at what step to rejoin the process.

[0103] FIG. 4B illustrates how the execution of this process can flow between processors.. In this example, there are two processors (i.e., processor 418 and processor 424) that interact with three devices (i.e., device 422, device 436, and device 438). Script 434 is present on processor 418 and processor 424. Device 422 sends event message 416 to processor 418 indicating the an alarm event occurred. In response to event message 416, processor 418 executes step of script 434, therebygenerating operation message 420, which includes state information (e.g., a tuple of the script ID and a script function pointer representing step fi). our

[0104] Step fi of script 434 includes instructions that operation message 420 is transmitted to device 436. Upon receiving operation message 420, device 436 performs the indicated operation (e.g., sounding the outside bell) and generates an event message which includes ACKi, which includes the state information from operation message 420. Copies of ACKi are multicast to all processors that can be reached by event message device 436. Receiving event messages 426 and 428, allows processors 418 and 424, respectively, to determine that the subsequent step in script 434 is step f2 and provides the trigger / impetus for processors 418 and 424 to consider executing the next automation step

[0105] In response to event message 428, processor 424 executes step f2 of script 434, thereby generating operation message 432, which includes state information (e.g., a tuple of the script ID and a script function pointer representing step _ )). Processor 418 can also perform step fi of script 434, but processor 418 is not connected to device 438 and therefore cannot send an operation message to device 438.

[0106] Upon receiving operation message 432, device 438 performs the indicated operation (e.g., flashing the bedroom bulb) and generates an event message which includes ACK2, which includes the state information from operation message 432. Copies of ACK2 would be multicast to all processors that can be reached by device 438, which in this example is processor 424.

[0107] Thus the process is carried out with the processors acting independently and autonomously and without communication among the processors. Automation flow 414 provides a simple example of Decentralized and Autonomous Control (DAC).

[0108] In a DAC system model, a CDNET connects decentralized controllers and devices. The CDNET may be subject to continuous electronic and environmental interferences, which may cause routes to be unstable. Furthermore, a DAC system may be deployed within a physical structure where the architecture, passing objects, or people may form transient network partitions or zones. Devices are installed where they are functionally needed, and controllers are spatially dispersed to zones such that more than one controller can interact with each device.

[0109] The controllers and devices form a decentralized system and communicate by exchanging messages using the CDNET. According to certain non-limiting examples, the network delivers a message m within a reasonably short duration Amet, or it is dropped.[HO] Devices in the operating environment may be miniature sensing or actuating objects or any other sensor, actuator, or a computing device. Each device has routes through the CDNET to at least one controller, as illustrated in FIG. IB.[Hl] FIG. 5A illustrates an example of a script (e.g., script 502) that can be used to perform a process such as an automation process. Script 502 includes various steps (e.g., step 524, step 526, and step 534) and metadata 528. Each of the steps can include a step ID and instructions. For example, step 524 includes step ID 504 and instruction(s) 506. The metadata 528 can include, among other things, associated events 508, object sequences 510, traversal rules 530, and last access time 532. Script 502 can be identified / labeled by script ID 512. A script can be a manifest, and the terms “script” and "manifest" are used herein interchangeably.

[0112] A script / manifest is a list of objects / steps (or equally, operations and other artifacts), and optional metadata. For example, the objects / steps can be operations of an automation process or can be other artifacts. The metadata can include object / step descriptions, events associated with the manifest {et, natural and data dependent sequences {G} of the steps listed in the manifest, step traversal rules OTR, a last access time, and other data elements applicable to the domain described by the manifest. An example of an OTR rule might be: given a reference to a step, a rule describes how to obtain a reference to the subsequent step. A manifest has an ID that uniquely qualifies it in the system. Each object / step in a manifest has an ID that uniquely qualifies it in the manifest. Manifests can be immutable or have a sporadic rate of update. Updated manifests can be made available to all devices. The natural sequence of objects in a manifest may be implicitly defined by their listing order or be explicitly enumerated. A system can have one or more manifests.

[0113] A script can be stored on a processor / controller and the steps / objects in the script can be used to generate operation messages. These operation messages are sent to one or more devices, which perform one or more operations indicated by a given operation message. In addition to operations to be performed by a device, the operation messages can also include a field that encodes script ID 512 and the step ID that was executed on the processor to generate the operations. The field that encodes the script ID and the step ID can be a two-tuple <mID,oID> such as a Program Counter (PC) that notifies the device where the system is within the process (e.g., the automation process). Because multiple processors can autonomously and independently execute the script / manifest, a device may receive duplicates of a given operation message, and these duplicates may arrive at the device at different times. To avoid erroneously repeating the operation of a duplicate operation message, thedevice can, e.g., record in its state information the two-tuples corresponding to the operations the the device performs, and the device can omit operations from those operation messages having two- tuples that match any of the two-tuples stored in the state information of the device.

[0114] FIG. 5B illustrates an example of an event message (e.g. event message 514). Event messages can also be referred to as “events” or as "impetuses," and these terms can be used interchangeably. Event message 514 can include data 516, timestamp 518, script ID 520, and step ID 522. The script ID 520 and step ID 522 can identify a step of the script that determined the operation that was performed by a device. For example, as a device receives operation messages, the two-tuple of the received operation message can be added to the top of a stack of two-tuples that is stored in the device's state information. Then the two-tuple on the top of the stack can be used by a device to populate the fields script ID 520 and step ID 522 for event messages that are transmitted by the device.

[0115] According to certain non-limiting examples, an impetus / event causes something to occur. Events can have a value (e.g., data 516, which could be NULL). Further, events can have an explicit or implicit (e.g., direct or indirect) reference oID to an object / step in a manifest. Additionally, events can optionally have metadata such as a timestamp ef, association with the time the event occurred, and the event source. An example of an explicit reference to an object oID is an event that carries an object ID, and implicit reference is a NULL oID referring to the first object in the manifest (implicit rules that reference the nth object are usable, and each system may define them to satisfy its specific requirement).

[0116] Shared values are used to provide stigmergy among the processors without communication among the processors. First consider construction a shared constant value. Constant values can be used to identify objects. The value produced for object x is a constant oIDx, and object y is the constant oIDy. In other words, the ID of an object is constant. This is a beneficial method when the need to identify an object is impartial to the frequency of events. For example, when a system is required to produce the same value in a manner unrelated to the occurrence of events or produce the same value upon each and any event. Such shared values can be constructed from any list we inclusively name manifest. Many artifacts are manifests, depending on the use case or domain of the system. For example, scripts or other code listing and any meaningful list according to the use case.

[0117] FIG. 6 illustrates an example routine for a method of obtaining a shared constant value. Although the example routine depicts a particular sequence of operations, the sequence may be altered without departing from the scope of the present disclosure. For example, some of theoperations depicted may be performed in parallel or in a different sequence that does not materially affect the function of the routine. In other examples, different components of an example device or system that implements the routine may perform functions at substantially the same time or in a specific sequence.

[0118] When an event e is perceived or received by a processor: (1) Use the manifest associated events {e, , data element or mID field of the event to reference the manifest associated with the event. (2) Use the event oID field of the event to reference an object in the manifest. If oID = NULL, the referenced object is the first (or generally the nth if the system defines it) in the manifest. (3) The tuple <mID, oID> obtained by the above two operations is a unique value that qualifies the current object. It is a shared value produced by all participating devices. (4) If what is needed is a unique value that qualifies the subsequent object, then use the manifest object sequences G, or Object Traversal Rules OTR, to traverse from <mID, oID> to <mID, oID ’>. (5) The tuple <mID, oID ’> is a shared value that each participating processors produces independently.

[0119] According to some examples, the method includes using the manifest associated events , ...} data element (see example in FIG. 6) or mID field of the event to reference the manifest associated with the event at block 602.

[0120] According to some examples, the method includes using the event oID field to reference an object in the manifest. If oID = NULL, the referenced object is the first (or generally the nth if the system defines it) in the manifest at block 604.

[0121] According to some examples, the method includes encoding the tuple <mID, oID> obtained by the above two operations is a unique value that qualifies the current object, it is a shared value produced by all participating devices at block 606.

[0122] According to some examples, the method includes using the manifest object sequences G, or object traversal rules OTR, to traverse from <mID, oID> to <mID, oID ’>, when a unique value that qualifies the subsequent object is needed at block 608.

[0123] According to some examples, the method includes using the tuple <mID, oID "■> as a shared value that each participating device produces independently at block 610.

[0124] Method 600 is one non-limiting example for generating a shared value, and other values can uniquely identify or tag an object. For example, in the case where the system has one manifest, the object oID and oID ’ by themself are unique and may be used, and mID may be omitted from the tuple.

[0125] Additionally, there are use cases where, upon each occurrence, the identified object requires a value unique to that episode. In other words, we would like to provide methods that produce a value that does not only qualify only the object but of every instance of the object. This is a much more complicated endeavor because instances of objects are dynamic or ephemeral creatures, and listing them in a manifest is not a reasonable task.

[0126] FIG. 7 illustrates an example routine for a method of obtaining a shared constant value that includes an episodic value. Although the example routine depicts a particular sequence of operations, the sequence may be altered without departing from the scope of the present disclosure. For example, some of the operations depicted may be performed in parallel or in a different sequence that does not materially affect the function of the routine. In other examples, different components of an example device or system that implements the routine may perform functions at substantially the same time or in a specific sequence.

[0127] Method 700 is like method 600, except another step is added to provide an episodic value. The step to generate the episodic value can occur after different points in the method. FIG. 7 shows the case where the episodic value is generated between block 608 and block 610. That is, block 712 is new.

[0128] This method works because either a process received the event or not, and all processes that receive it are privy to the same value of the ts field. Generally, an event may carry other unique fields, such as a message sequence or event number. Depending on the use case and the system design, they can substitute for the ts field. Such fields are non-repeatable (for very long periods, as we will soon show), and in the following, we further provide methods that produce a unique shared value from fields that may contain repeatable values.

[0129] According to some examples, the method includes using the manifest associated events {e,, ...} data element (see example in FIG. 7) or mID field of the event to reference the manifest associated with the event at block 702.

[0130] According to some examples, the method includes using the event oID field to reference an object in the manifest. If oID - NULL, the referenced object is the first (or generally the / / th if the system defines it) in the manifest at block 704.

[0131] According to some examples, the method includes encoding the tuple <mID, oID> obtained by the above two operations is a unique value that qualifies the current object, it is a shared value produced by all participating devices at block 706.

[0132] According to some examples, the method includes using the manifest object sequences G, or object traversal rules OTR, to traverse from <mID, oil) to <mID, oID ’>, when a unique value that qualifies the subsequent object is needed at block 708.

[0133] According to some examples, the method includes using the tuple <mID, oID ’> as a shared value that each participating device produces independently at block 710.

[0134] There are various other alternative implementations for generating the episodic value. Here, three alternative implementations are presented: (1) obtaining a shared episodic value, unique with probability P at time T; (2) obtaining a shared episodic value, with value provenance being the process; and (3) obtaining a shared episodic value, with value provenance being a device.Obtaining a shared episodic value, unique with probability P at time T

[0135] When a real-world clock is used as the timestamp A, it will not repeat. If a real-world clock is not available, a simple 32-bit timestamp counter that advances every second will overflow after a little longer than 136 years, and until then, it will provide a unique fountain of values. Data elements like these are worth considering for use by the methods we describe. Timestamp counters may be readily available in many systems or easily implemented. Nevertheless, when a system does not have them or other fields with similar non-repeatability characteristics, we provide the following method.

[0136] These methods utilize the idea from probability theory that if a value v is drawn, For example, from a Normal probability distribution, then as more values are drawn the higher is the probability that the value will repeat. Nevertheless, in many real systems, objects become irrelevant over time and are dereferenced or naturally fade. FIG. 8 illustrates this idea. The plotted probabilities show that after some time (the gray area), a value v can be reused. This occurs when the probability that v is used in a reference to an object is lower than the probability that an event source will regenerate it: P[objv] < P[v], System designers can determine when this time T starts based on the frequency that any candidate field v is drawn from its probability distribution and how long objects remain relevant before dereferenced. The method to generate a freshly new shared value is:• If time > t, add the field holding v from the event to the tuple, resulting in <mID, oID’, v>.• In systems that require the processes to determine the tuple in periods that are shorter than T, multiple fields v, v’, vncan be used to lower the probability to a desired level: Add the fields v, v ’, v" to the tuple, resulting in < mID, oID ’, v, v vn>.Obtaining a shared episodic value, with value provenance being the process:

[0137] The previous methods used a source of values that came from the impetus. An impetus / event occurs, and the processes must decide on a value they will share. It sounds unreasonable that the source of values can emanate from the processes themselves because previously it was commonly thought that each independent process will generate a different value without coordination. This notion is correct, but there are countless use cases where the need for the shared value can be delayed. We use that delay in the following methods.

[0138] Consider again the example of a control system. Using the following method, a plurality of controllers send a command towards an actuator, and each includes their own locally determined value r in the message. So we have a command cmd that is tagged: <cmd, r ’>, <cmd, r”>,<cmd, rn>, ... with different r values. After the target device (plausibly an actuator in this case) runs one of the redundant copies of cmd, it sends an ACK to the device and includes the r-value of the command it elected to execute. Consequently, the receiving devices have a shared tuple that uniquely references cmd. <cmd, r>. This method uses the fact that controllers must send commands for any work to be done in a control system. Nevertheless, the method removes the need for the controllers to a priori communicate to agree on the shared value with which they will identify the command. The general method in this case is:

[0139] The process generates a value r (it can be a random number or any arbitrary number, including 0 and NULL). The process sends a message m to a target device, containing a reference to the object that needs identification and the value r, resulting in <mID, oID r>. The target device does its function and multicasts an ACK that includes the reference to the object it elected to use: <mID, oID rn>. The ACK for contains a shared value <mID, oID / '" > that is shared among all receiving processes.

[0140] Note that the target device can also remember <mID, oID rn>, so if it receives a late <mID, oID f > due to some network anomaly, it can simply ignore it because mID, oID ’> was already processed by it.Obtaining a shared episodic value, with value provenance being the target device:

[0141] Block 712 can alternatively be implemented by:• Each process sends a message m to a target device containing a reference to the object <mID, oID ’> for which a shared value is required.• The target device generates r and includes it in its ACK, resulting in <mID, oID r> sent to the processes.• The ACK for m contains a shared value <mID, oID r>.

[0142] Note that using fields from a received event or ACK is similar because both are values generated by entities external to the process that uses them. Compared to that, the source of values is internal when the provenance is a process on the same computer. The latter provides more flexibility in terms of range, type, structure, and other properties that may be desired for the values to have.

[0143] Additionally, note that not all use cases and systems require a manifest. It depends on what needs to be referenced. For example, when an event arrives at a plurality of devices, a shared value identifying the event is the ts field from the event, and other fields, such as an event sequence number, should be considered. Similarly, any field v can be used when P[v] is sufficient for the use case.

[0144] Depending on the use case, an impetus may carry a reference to the object or object instance that needs identification. For example, if the event carries <oID >, then the receiving devices can use ts from the event to construct a <oID, ts> tuple or oID, v > when appropriate, without using a manifest.

[0145] There are various other alternative implementations for ensuring the shared value / number is within a desired range, and these alternative implementations are now discussed.

[0146] There are many use cases and systems that require selecting a number from a range. For example, between 0 to n - 1 or more generally between m to + n - 7. Once the number is chosen, it can be used for decision-making to achieve any desired task. In the other systems, communication and consensus protocols are necessary when multiple processors are required to use the same number. Methods are now introduced to obtain such a number without communicating among the processes producing the value.

[0147] In this method a shared number between 0 to n - 1 is determined by distributed processors without coordination of the processors by:1. A processor perceives an impetus (such as a message specifying the desired value ri) to determine the shared number.2. Each processor uses an agreed-upon field from the impetus event. For example, use the message counter me.3. Each processor calculates y = me mod n.4. The value y is a shared value between 0 to n - 1 as required.

[0148] Any algebraic or other transformation on this value will provide different forms, numeric, alphanumeric, or other forms. Furthermore, the produced value can further be used to drive additional decisions.

[0149] The method of obtaining a number in range 0 to n - 7, can be generalized to obtaining a number in any desired range, such as m to m + n - 7, by adding a known or determined m to each produced number.

[0150] The methods we introduced are highly resilient to failures because all participating processes produce the same value while remaining fully independent. In particular, they do not depend on communication to reach a consensus on the produced value, and communication is one of the critical components that compromise consensus protocols (see, for example, the FLP (Fischer, Lynch, and Paterson) Impossibility theorem. The FLP Impossibility Theorem is a fundamental theorem of distributed systems proving that no consensus protocol can tolerate even a single undetected failure, and Molina proved that with some communication failures, it may be impossible to guarantee that any process can become a leader.

[0151] Suppose there is no applicable field to use in step (2) above. The following method determines a number between 0 to n - 7 in such cases:1. A processor receives a message that a shared value is required.2. Calculate the hash of the incoming message: h = hash(message) .3. Calculate: y = h mod n.4. The value y is a shared value between 0 to n - 7 as required.

[0152] FIG. 8 illustrates a plot of the probabilities for obtaining a shared episodic value, unique with probability P at time T. As discussed above, the plotted probabilities show that after some time (the gray area), a value v can be reused. This occurs when the probability that v is used in a reference to an object is lower than the probability that an event source will regenerate it: P[objv] < P[v], System designers can determine when this time T starts based on the frequency that any candidate field v is drawn from its probability distribution and how long objects remain relevant before dereferenced.

[0153] FIG. 9 illustrates an example method 900 for executing a script in a DAC system. Although the example method 900 depicts a particular sequence of operations, the sequence may be altered without departing from the scope of the present disclosure. For example, some of the operations depicted may be performed in parallel or in a different sequence that does not materially affect the function of the method 900. In other examples, different components of an example device orsystem that implements the method 900 may perform functions at substantially the same time or in a specific sequence.

[0154] According to some examples, the method includes receiving, at respective processors, event messages that encode data representing second state information at block 902.

[0155] According to some examples, the method includes determining which scripts are invoked by the arriving event and executing, by the processors, instructions of steps in a script, and decide what operation to send to a target device at block 904.

[0156] According to some examples, the method includes transmitting, from the processor, operation messages encoding first state information and / or operations generated by executing the respective instructions at block 906.

[0157] According to some examples, the method includes generating, at respective devices, the event messages, wherein some of the event messages encode acknowledgments at block 908.

[0158] According to some examples, the method includes performing, at the devices, (i) one or more tasks and / or (ii) operations of one or more operation messages received from one or more of the processors, wherein at block 910.

[0159] According to some examples, the method includes for each of the devices, a device multicasts the event messages include the second state information to a subset of the processors that is communicably connected with the device, and the second state information provides stigmergy among the processors without communication among the processors at block 912.

[0160] The timeline in FIG. 10A illustrates aspects of the concept of freshness. The concept of freshness allows a distributed system to operate in a correct state. To illustrate where problems may lurk, see FIG. 10A and imagine the following scenario: At time ti controller c received the most recent update and learned that device d state is ev. A reasonable time later, at ts, the controller is about to decide based on this value. Fractionally earlier, however, at / ?, the state of d changed to ev'. The device immediately sent the update, but c did not receive it yet because of communication time. We say the most recent data is below the controller knowledge-horizon. As omnipotent observers outside the system, we know that c is about to decide using not the most recent value, but that edge- of-knowledge has not yet reached it.

[0161] After initialization, each controller may build their set of Fresh Events, FE, by sending a query to devices D', the set of devices it commonly controls, or wait for events from devices D, the set of all devices. The latter is sufficient, and the only necessary condition because scripts aredescribed as a set of mathematical functions, and the entire state a function needs to run is its arguments. We have already discussed the benefits of mathematical functions to the flow of execution. They also make a good design choice because they provide the significant advantage of allowing a controller to contribute to a script run, even if it has sufficient data to execute only an occasional function. At the same time, FE is assembled.

[0162] To address the knowledge-horizon challenge, a process closest to the ground truth defines a freshness period AFrfor each datum e it measures or generates. This period should be related to the physics of the phenomenon and its expected rate of change.

[0163] Definition of Freshness: The freshness period AFrof each event e modality on device d, registered using t / local clock, defines that the value d.evis correct (fresh) from its genesis time d.etto d.et+ AFr. In interval notation, e is fresh during: [d.et, d.et + AFr],

[0164] Controllers know AFra priori for all devices D and their modalities, and we require AFr» 2A'netotherwise, evwould not be of much use in a distributed system (considering communication and processing time). When queried, a device may send a newly obtained measurement state. However, when generating a new event is expensive on a particular device, it may send the current state if less than the half-life of e has elapsed. The equation they use for this purpose is: now () - et< ' 2AF'.

[0165] Controllers and devices C U D = Pi may not have a synchronized clock system, so the same point in time will likely have a different value on each. Furthermore, the processes are distributed, so communication routes have varying distances and latencies; therefore, the same message m sent from dPi to a reachable subset of controllers C’ -C c Pi may arrive to controllers at different world times. DAC design guarantees that: (1) All events arriving at C are fresh. (2) Without a synchronized clock system, any controller CC"can determine a remaining valid freshness period for an event e generated on d, using c discrete local clock. To see why it works, refer to FIG. 10B and the explanation below.

[0166] FIG. 10B illustrates how decentralized controllers can determine freshness of a datum autonomously, using only the data each of the controllers have locally.. An event e genesis etis timestamped using d clock. Shortly afterward, e is sent to reachable controllers CC"at time m,. The first and last controllers to receive it, respectively caafter e, and - after Ametcan determine aremaining valid freshness period using the freshness equation shown below. Therefore, any controller receiving the event between e and Ametcan determine a valid freshness period.

[0167] A controller CC"can calculate the minimum remaining freshness period 3Frof an event e using the freshness equation. In real systems, 3Frwill be close to the maximum remaining freshness because communication consumes a non-zero but negligible duration relative to AFr. The sending device d timestamps the message sending time m, using its discrete local clock, each message m carrying event e, and e tuple contains its genesis time et.The minimum remaining freshness equation:= AFr- (mt - et) -A,net

[0168] The minimum remaining freshness equation removes the reference to device d clock when subtracting (mf- et). Therefore, the resultis a clock-less length that any controller can use to bound the freshness period to its local clock, regardless of the controller clock skew relative to the generating device clock. This method removes the requirement to have a real-world clock in the system and clock synchronization among the processors, which when required in non DAC systems subject them to the problem we discussed earlier in this disclosure.

[0169] Let ctbe the time in the local controller clock when message becomes available on c. A freshness interval of event e on any controller CC"is given by [ct, ct- 8Fr],

[0170] We conclude the discussion on freshness with a few remarks for practitioners. First, some systems in the operating environment will have a negligible upper bound for A,net. Examples of such systems would be a small smart home or zone in the smart city or a small spacecraft. We can assume, therefore, that the time Ct when e is received by c is close in terms of real-world time, to event genesis etfor messages that are sent immediately upon e generation (this will not be correct upon a query of d state). We symbolize this relation ct / / etand caution that when the processes do not synchronize clocks, it is very likely that ct* et. In other words, even though the events occur nearly simultaneously, they may have very different time values, and as a result, one cannot replace the other in calculations.

[0171] As controllers receive data from devices, they build their local state in a set FE of fresh events. Controllers can take two approaches that keep FE correct. Either evict a datum e when it is no longer fresh (it is stale) or skip participating in running a function if e is stale when c reads it from FE. In either case, c may wait for another round or query d for an update. The evicting approach requires a timer and computing resources but may allow c to contribute to the runs of more scripts.OPERATING AT A DESIRED RELIABILITY:

[0172] The DAC architecture allows a system to operate at user-defined required reliability by adding controllers. The equation to calculate the number of needed controllers to support a desired reliability level is presented below. This equation can be used for the whole network or on a zone- by-zone basis. When the system spans a large area, some zones have heavy pedestrian traffic or are architecturally (structurally) separated to floors or by walls and other objects.

[0173] Let Pcbe the probability that a controller cC fails, and let Pi be the probability that a CDNET route from or to c fails. Because controllers and the CDNET are discrete cyber-physical objects, their failure probabilities are independent and likely to differ. For example, controllers made by a particular manufacturer can be favorably reliable; therefore, a couple of them may be sufficient to run automation with the desired reliability. Nevertheless, interference may be high in that zone, requiring more controllers to maintain an adequate route diversity. This can occur in space flights, where cosmic interference may be high.

[0174] In an operating environment where controllers and network may fail with probabilities 0 < Pc< 1 and 0 < Pi < 1, respectively; The number of controllers, n, that are needed to support a desired operating reliability (liveness) of 0 < L < 1 is given by:

[0175] In practice, we expect the system reliability to be higher than the specified liveness L because of the ceiling and max operations.CONTROL FLUX

[0176] DAC architecture allows script execution to flow between controllers in response to environmental dynamics. We name this design Control Flux. We logically divided control scripts into fine-grain components to support flux, each interacting with a single device and representing the smallest possible control step, interacting with a single device. We modeled the steps as mathematical functions which allows multiple controllers to contribute to a run of scripts even if a controller crashes and recovers into the middle of runs. This includes new runs, a controller did not participate before crashing and it just learned about when it recovers. The novel principle of a Flow function drives this quality. See FIG.4A and 4B for reference for the explanation of control flux below.

[0177] Definition of Consecutive functions. Functions f and fi of script 5 are consecutive if fi is the function that runs immediately after the fi function.

[0178] The code of a script 5 determines whether functions are consecutive. Without loss of generality, let cmd, be the command that sends to device di and ACKi be the acknowledgment d, sends in response. Similarly, let cmdj be the command fi sends to dj and ACKj be its acknowledgment.

[0179] Definition of a Flow function: Let fi and fi be consecutive functions. Function fi is a flow function when the controller it runs on receives A CKtand can send cmdj, and the controller has the necessary and sufficient data to run fi.

[0180] The flow characteristic is temporal, determined by the real-time network and environment dynamics. In a topology with healthy routes, more than one controller can hear ACK, and send cmdj at any given time. It is also spatial, therefore, it may further improve the system's reliability. For example, because of environmental dynamics, / may flow on controller cain one run and cyin another. For liveness, it is sufficient to assume that when a script s is invoked, its functions can flow on a subset of controllers C C. Generally, any distributed system model must accept two rudimentary properties for any work to be done: a function can receive a trigger to run and interact with its specified device. Consequently, our flow concept is the weakest connectivity assumption any system can hope for in a distributed system. It has an excellent affinity to the harsh operating environment and supports decentralization.

[0181] In previous systems, the required connectivity is more demanding, leading to some of their lackluster reliability and availability when the CNET is not redundant and strongly connected and because CDNET are low power and lossy. Leader-based systems require all devices to be connected to the leader that runs s. This is a demanding connectivity assumption, even before considering interferences and leader failures. For example, when a controller cannot physically reach all devices with consistency and reliably, any interference may fail the automation. Furthermore, such a system is exposed to the number of communication hops between the devices and the controller. Systems that support more than one leader may provide some remedy, but without supporting dynamic transfer of control, a controller that started to run s must complete it. This usually means that each subset of devices in such a system is tethered to a controller, and when it fails, some portion of the system, small or large, is out of service. As a flux system, DAC functions natively enable the transfer of control, and functions may flow between controllers. It may therefore, operate systemswhose devices are spread over large distances, and may ameliorate network interferences by diverting control from a compromised area to another.

[0182] The introduction of flow of control by DAC is critically important to ongoing reliable operations. In DAC, flow of control is core to the architecture. Unlike failover in current art architectures, which is a feature summoned upon failure, DAC flow of control is part of regular operations. This provides ongoing smoother, jitter free operation. Nevertheless, this behavior of DAC also provides a solution to operating conditions that are not possible in current art systems. When there are transient interferences precluding controller cp from working, a DAC cohort of controllers will simply allow another controller cato execute the operation. In leader-based system, when controller cp is the active controller, no number of attempts to deliver a message would remedy the problem. Controller cp will be simply wasting resources and time, and the operation will still fail. Worse, since controller cp is not failing in such situations, it will not be replaced by another leader.PROTRACTED OPERATIONS AND RELIABILITY

[0183] In some use cases, minimizing the duration from controller decision to sensing or actuation is important. The issue at hand may be a human expectation or an essential non-functional requirement. It certainly affects the perceived quality of the system when devices are in front of the user's eyes, such are the cases of the smart home or human operated machinery in a plant. And the issue can be momentous if the control system has safety-critical or time-sensitive devices. Shortening the time from decision to operation and doing so with high reliability, may therefore be vital if we want to control more than a chandelier. We will explain in this section, how DAC provides both.

[0184] Let the optimal decision-to-actuation time be denoted t*. Let the round-trip time of a message be RTT, and the number of rounds be R. Let the probability that a message survives a one way traversal through the network be P[A],

[0185] In today's networks, the shortest decision-to-operation time we can hope for is t* = 0.5RTT (the time it takes a message, sent from a controller to a target device to arrive). Since a network may lose messages, important messages are usually sent using a reliable protocol such as TCP / IP. TCP will construct a session with the target and will then guarantee message delivery towards that target. The session construction requires no less than 2 messages, before an application can send a message, therefore, the earliest time that a command can execute is 3t* (three times longer than the optimal),and the earliest that a controller can know whether a command actually executed is at time 4t* (when a confirmation is received). If the message is not delivered, a reliable protocol will retransmit it until it is received, or the number of attempts or period was exceeded. It follows that, upon failures the decision to operation time will be longer than 3t*. In the general case the command executes at time R X t*, R >3 and the probability that it executes is P[A]R. The salient observation is that because P[A] < 1.0, in environments with interference more round trips (higher J?) may be required to deliver a message but the probability of delivering the message decreases as R grows.

[0186] DAC architecture is different. Because DAC controllers are autonomous, they are independent, and each message provenance is different. Consequently, each of the messages traverses different routes toward a target device. As a result it may overcome interference. The decentralized controllers and routes form a parallel system. It is known from the theory of probability that the probability that n independent controllers will successfully complete a task is 7 - P[all controllers fail] , which is equal to q .

[0187] To exemplify the differences, let a DAC architecture use the same energy budget that a reliable protocol requires to deliver a message. In a leader architecture, it is prudent to use a reliable protocol because of the challenges we have already discussed throughout this paper. Theenergy budget is therefore 3 messages (in environments with regular interferences the average number of messages needed to deliver a message may be more). We will use this energy budget to configure 3 DAC controllers to send 1 message each, using a protocol that does not guarantee delivery. For an example, let the probability of message delivery be P[A] = 0.9. The probability that a leader based system will deliver the message is P[A]R= P[A]3= 0. 729 and the control command may execute at time 3t* at the earliest. Contrary to that, the probability that a DAC architecture will deliver the message is P[A]3= 0.999 and it may execute at time t* at the earliest. What we are observing is that a DAC based system is remarkably more reliable and provides a faster command-to-operation time.

[0188] The desire to build parallel machines is not new. The challenge prior to DAC, however, was that there was no known way to guarantee that only one of the redundant commands will execute. Although techniques to coordinate among the controllers can work, these techniques require redundant and highly reliable CNET but they are not plausible or possible in many use cases. Moreover, the synchronization and coordination over a CNET decrease the probability of success. For an architecture to form a parallel system, the controllers must be autonomous. Therefore, new methods of identifying a redundant batch of messages are required and DAC provides that using the universal idempotent scheme.FAR FIELD COMMUNICATION:

[0189] The very high reliability of DAC architecture and its asymptotically optimal control -to- operation time may be advantageous for controlling distant devices. It may be a chemical plant in a desolated area, an area during a natural disaster, or any other use case where the controllers are distant from the devices they control. Remote control of space missions provides a more picturesque motivation.

[0190] Imagine controllers on Earth, intermediary IPv6 nodes aptly scattered on the path to Mars, and devices on the planet Mars. A command is sent from Earth to a device on Mars, using a reliable transport protocol such as TCP / IP to guarantee its delivery. Using the speed of light it is known that one-way communication takes approximately t* = J 2.5 minutes. If there are no failures, the command will execute after 37.5 minutes. If the command fails to arrive, or the device doesn't execute it successfully, the controller on Earth (a human or a computer) will know about it after 50 minutes. The command may be automatically retransmitted but it will execute 1 hour and 2.5 minutes after the controller first issued it, or longer if there is more than 1 failure.

[0191] In a terrestrial use case, users regularly experience tardy actuation when their controller is in the cloud, and numerous systems are cloud based these days. Remote control of far industrial loT sites will encounter similar delays.

[0192] A DAC system changes that at the core of the system architecture. A number of independent controllers may send the command over diverse routes offered by the same intermediary IPv6 nodes aptly scattered on the path to Mars. The reliability of delivery will be higher. If a higher reliability is desired, simply add a controller. Importantly, adding DAC controllers does not increase the command delivery time, which remains t* at that set probability of success.

[0193] FIG. 11 shows an example of computing system 1100. The computing system 1100 can be a device or a controller / processor. The computing system 1100 can perform the functions of one or more of devices 102, controllers 104, or any of the devices or processor illustrated in FIG. 2 A, FIG. 2B, FIG. 4A, or FIG. 4B. The computing system 1100 can be part of a distributed computing network in which several computers perform respective steps in method 600, method 700, or method 900 and / or the functions of the processors or devices in a DAC system. The computing system 1100 can be connected to the other parts of the distributed computing network via connection 1102 or communication interface 1124. Connection 1102 can be a physical connection via a bus, or a direct connection into processor 1104, such as in a chipset architecture. Connection 1102 can also be a virtual connection, networked connection, or logical connection.

[0194] In some embodiments, computing system 1100 is a decentralized processing system in which the functions described in this disclosure can be distributed within a home, building, plant, city, datacenter, multiple data centers, a peer network, etc. In some embodiments, one or more of the described system components represents many such components each performing some or all of the function for which the component is described. In some embodiments, the components can be physical or virtual devices.

[0195] Example computing system 1100 includes at least one processing unit (CPU, microprocessor, or other processor) 1104 and connection 1102 that couples various system components including system memory 1108, such as read-only memory (ROM) 1110 and random access memory (RAM) 1112 to processor 1104. Computing system 1100 can include a cache of high-speed memory 1106 connected directly with, in close proximity to, or integrated as part of processor 1104. Processor 1104 may essentially be a completely self-contained computing system, containing multiple cores or processors, a bus, memory controller, cache, etc. A multi-core processor may be symmetric or asymmetric.

[0196] Processor 1104 can include any general-purpose processor and a hardware service or software service, such as services 1116, 1118, and 1120 stored in storage device 1114, configured to control processor 1104 as well as a special -purpose processor where software instructions are incorporated into the actual processor design.

[0197] To enable user interaction, computing system 1100 includes an input device 1126, which can represent any number of input mechanisms, such as a microphone for speech, a touch-sensitive screen for gesture or graphical input, keyboard, mouse, motion input, speech, etc. Computing system 1100 can also include output device 1122, which can be one or more of a number of output mechanisms known to those of skill in the art. In some instances, multimodal systems can enable a user to provide multiple types of input / output to communicate with computing system 1100. Computing system 1100 can include a communication interface 1124, which can generally govern and manage the user input and system output. There is no restriction on operating on any particular hardware arrangement, and therefore the basic features here may easily be substituted for improved hardware or firmware arrangements as they are developed.

[0198] Storage device 1114 can be a non-volatile memory device and can be a hard disk or other types of computer-readable media that can store data that are accessible by a computer, such as magnetic cassettes, flash memory cards, solid state memory devices, digital versatile disks,cartridges, random access memories (RAMs), read-only memory (ROM), and / or some combination of these devices.

[0199] The storage device 1114 can include software services, servers, services, etc., that when the code that defines such software is executed by the processor 1104, it causes the system to perform a function. In some embodiments, a hardware service that performs a particular function can include the software component stored in a computer-readable medium in connection with the necessary hardware components, such processor 1104, connection 1102, output device 1122, etc., to carry out the function.

[0200] For clarity of explanation, in some instances, the present technology may be presented as including individual functional blocks including functional blocks comprising devices, device components, steps or routines in a method embodied in software, or combinations of hardware and software.

[0201] Any of the steps, operations, functions, or processes described herein may be performed or implemented by a combination of hardware and software services or services, alone or in combination with other devices. In some embodiments, a service can be software that resides in memory of a device or a processor and performs one or more functions of a DAC when a processor executes the software associated with the service. In some embodiments, a service is a program or a collection of programs that carry out a specific function. In some embodiments, a service can be considered a server. The memory can be a non-transitory computer-readable medium.

[0202] In some embodiments, the computer-readable storage devices, mediums, and memories can include a cable or wireless signal containing a bit stream and the like. However, when mentioned, non-transitory computer-readable storage media expressly exclude media such as energy, carrier signals, electromagnetic waves, and signals per se.

[0203] Methods according to the above-described examples can be implemented using computerexecutable instructions that are stored or otherwise available from computer-readable media. Such instructions can comprise, For example, instructions and data that cause or otherwise configure a general-purpose computer, special-purpose computer, or special-purpose processing device to perform a certain function or group of functions. Portions of computer resources used can be accessible over a network. The executable computer instructions may be, For example, binaries, intermediate format instructions such as assembly language, firmware, or source code. Examples of computer-readable media that may be used to store instructions, information used, and / or information created during methods according to described examples include magnetic or opticaldisks, solid-state memory devices, flash memory, USB devices provided with non-volatile memory, networked storage devices, and so on.R04] Devices implementing methods according to these disclosures can comprise hardware, firmware and / or software, and can take any of a variety of form factors. Typical examples of such form factors include servers, laptops, smartphones, small form factor personal computers, personal digital assistants, and so on. The functionality described herein also can be embodied in peripherals or add-in cards. Such functionality can also be implemented on a circuit board among different chips or different processes executing in a single device, by way of further example.

[0205] The instructions, media for conveying such instructions, computing resources for executing them, and other structures for supporting such computing resources are means for providing the functions described in these disclosures.

[0206] For clarity of explanation, in some instances the present technology may be presented as including individual functional blocks including functional blocks comprising devices, device components, steps or routines in a method embodied in software, or combinations of hardware and software.

[0207] Any of the steps, operations, functions, or processes described herein may be performed or implemented by a combination of hardware and software services or services, alone or in combination with other devices. In some embodiments, a service can be software that resides in memory of a client device and / or one or more servers of a content management system and perform one or more functions when a processor executes the software associated with the service. In some embodiments, a service is a program, or a collection of programs that carry out a specific function. In some embodiments, a service can be considered a server. The memory can be a non- transitory computer-readable medium.

[0208] In some embodiments the computer-readable storage devices, mediums, and memories can include a cable or wireless signal containing a bit stream and the like. However, when mentioned, non-transitoiy computer-readable storage media expressly exclude media such as energy, carrier signals, electromagnetic waves, and signals per se.

[0209] Methods according to the above-described examples can be implemented using computerexecutable instructions that are stored or otherwise available from computer readable media. Such instructions can comprise, For example, instructions and data which cause or otherwise configure a general purpose computer, special purpose computer, or special purpose processing device to perform a certain function or group of functions. Portions of computer resources used can beaccessible over a network. The computer executable instructions may be, For example, binaries, intermediate format instructions such as assembly language, firmware, or source code. Examples of computer-readable media that may be used to store instructions, information used, and / or information created during methods according to described examples include magnetic or optical disks, solid state memory devices, flash memory, USB devices provided with non-volatile memory, networked storage devices, and so on.

[0210] Devices implementing methods according to these disclosures can comprise hardware, firmware and / or software, and can take any of a variety of form factors. Typical examples of such form factors include servers, laptops, smart phones, small form factor personal computers, personal digital assistants, and so on. Functionality described herein also can be embodied in peripherals or add-in cards. Such functionality can also be implemented on a circuit board among different chips or different processes executing in a single device, by way of further example.[2H] The instructions, media for conveying such instructions, computing resources for executing them, and other structures for supporting such computing resources are means for providing the functions described in these disclosures.

[0212] Although a variety of examples and other information was used to explain aspects within the scope of the appended claims, no limitation of the claims should be implied based on particular features or arrangements in such examples, as one of ordinary skill would be able to use these examples to derive a wide variety of implementations. Further and although some subject matter may have been described in language specific to examples of structural features and / or method steps, it is to be understood that the subject matter defined in the appended claims is not necessarily limited to these described features or acts. For example, such functionality can be distributed differently or performed in components other than those identified herein. Rather, the described features and steps are disclosed as examples of components of systems and methods within the scope of the appended claims.Aspects:

[0213] The present technology includes computer-readable storage mediums for storing instructions, and systems for executing any one of the methods embodied in the instructions addressed in the aspects of the present technology presented below:

[0214] Aspect 1. A system comprising: processors configured to execute, in response to receiving an event message, one or more steps in a script, and configured to transmit operation messages encoding first state information and / or operations generated by executing a step of the script;devices configured to generate event messages encoding data representing second state information and / or representing acknowledgments, and the devices being configured to perform (i) one or more tasks and / or (ii) operations of one or more operation messages received from one or more of the processors, wherein, for each of the devices, a device multicasts the event messages including the second state information to a subset of the processors that is communicably connected with the device, and the second state information provides stigmergy among the processors without communication among the processors.

[0215] Aspect 2. The system of Aspect 1, wherein the second state information, which is received in a multicast event message from a device, includes process-step information indicating what step in the script informed an operation performed by the device, and the processors are configured to determine a subsequent step in the script using the process-step information in the second state information, resulting in the subset of the processors that receive the multicast event message each handling a same subsequent step in the script without communication among the subset of the processors.

[0216] Aspect 3. The system of any one of Aspects 1-2, wherein upon a subsequent step in the script being triggered, a processor elects to not execute the subsequent step and to not generate an operation message corresponding to the subsequent step, when the processor determines that a datum at the processor that is used in the subsequent step is not fresh.

[0217] Aspect 4. The system of any one of Aspects 1-3, wherein the processor is configured to determine the datum is fresh using the freshness equation or its derivatives.

[0218] Aspect 5. The system of any one of Aspects 1-4, wherein for each of the devices, a device is configured to compare a state of the device with the first state information received in an operation message to determine a comparison result, and the device is further configured to use the comparison result to determine whether to perform an operation of the operation message, wherein the state of the device is based on the first state information received in prior operation messages.

[0219] Aspect 6. The system of any one of Aspects 1-5, wherein the device is configured to not perform an operation of the operation message when the comparison result indicates that the operation message is obsolete.

[0220] Aspect 7. The system of any one of Aspects 1-6, wherein the device is configured to not perform an operation of the operation message when the comparison result indicates that the first state information of the operation message matches the first state information of another operation message for which the operations have been performed on the device.

[0221] Aspect 8. The system of any one of Aspects 1-7, wherein: the first state information encoded in a given operation message includes a first identifier that identifies which step in the script was executed to generate the given operation message, and the second state information that is multicast, in a given event message, from a given device to a given subset of the processors includes a second identifier representing a state of the given device including information of the first identifier of a most recent operation that was performed in response to an operation message.

[0222] Aspect 9. The system of any one of Aspects 1-8, wherein the first state information of the operation messages includes a first identifier comprising two or more shared indices including a first index representing the script and a second index representing the step of the script corresponding to an operation.

[0223] Aspect 10. The system of any one of Aspects 1-9, wherein two or more shared indices further include an episode indicator.

[0224] Aspect 11. The system of any one of Aspects 1-10, wherein the episode indicator is a value selected by a device based on proposed values from two or more processors.

[0225] Aspect 12. The system of any one of Aspects 1-11, wherein the episode indicator is encoded from the arrival of a common object that is external to the processors, the common object being an object that is referenced by each of the processors generating a same value for the episode indicator of the two or more shared indices.

[0226] Aspect 13. The system of any one of Aspects 1-12, wherein the common object is an event message that was multicast to the processors and the processors that receive the event message encode the episode indicator using one or more fields of the event message.

[0227] Aspect 14. The system of any one of Aspects 1-13, wherein the episode indicator is determined using a random number generator to generate one or more random numbers, wherein a combined probability of the one or more random numbers provides a predefined confidence that a combination of the one or more random numbers uniquely identifies the episode indicator.

[0228] Aspect 15. The system of any one of Aspects 1-14, wherein the episode indicator originates on a device.

[0229] Aspect 16. The system of any one of Aspects 1-15, wherein one or more values of the first state information of an operation message originates from one of the devices.

[0230] Aspect 17. The system of any one of Aspects 1-16, wherein one or more values of the second state information of an event message originates from one of the processors.

[0231] Aspect 18. The system of any one of Aspects 1-17, wherein the first state information includes an index that is an integer in a range [0, n-1], and each of the processors receiving a multicast event message independently generates a same value for the index by generating the index using information of the multicast event message.

[0232] Aspect 19. The system of any one of Aspects 1-18, wherein: the event messages include acknowledgment messages, measurement messages, heartbeat messages, occurrence messages and / or combinations thereof, the acknowledgment messages including the second state information and an acknowledgment of handling at a device of another message, the measurement messages including the second state information and one or more values of measurements obtained by the device, the heartbeat messages being a periodic report of a status of the device, the status including the second state information, and the occurrence messages including the second state information and indicia that predefined criteria have occurred.

[0233] Aspect 20. The system of any one of Aspects 1-19, wherein the second state information provides stigmergy among the processors by facilitating lockstep execution of the script among the subset of the processors, wherein the subset of the processors independently use the second state information to select a subsequent step of the script to execute.

[0234] Aspect 21. The system of any one of Aspects 1-20, wherein: the processors encode, in the first state information of an operation message, one or more values representing a step in the script that was used to generate the operation message, and the devices use the one or more values received in the operation message to select which operations to perform, omitting operations of the operation message when the one or more values indicate that the operation message duplicates a previously received operation message and that the device performed the operations of the previously received operation message.

[0235] Aspect 22. The system of any one of Aspects 1-21, wherein: the first state information designates the operation messages from a processor in accordance with which steps in the scripts were executed to generate the operation messages, the second state information designates the event messages from a device in accordance with which operation messages directed the operations performed by the device, and a combination of the first state information and the second state information provides lockstep execution among the processors by uniquely identifying to the devices which steps of the script are associated with the respective operation messages and informing the processors to which steps of the script the event messages correspond.

[0236] Aspect 23. The system of any one of Aspects 1-22, wherein: the script is an automation script; the steps are automation steps; the first state information is one or more shared indices are produced independently throughout the system; and the first state information of an operation message uniquely identifies a respective step of the script that generated the operation message.

[0237] Aspect 24. The system of any one of Aspects 1-23, wherein: the processors are configured such that, when a processor does not participates in executing one or more steps of the script, the processor uses the second state information of an event message from a device to rejoin an execution of a script by using the second state information to determine a subsequent step of the script to be performed, wherein the event message is generated by the device in response to performing an operation of the current step of the script.

[0238] Aspect 25. The system of any one of Aspects 1-24, wherein the devices are configured such that, in response to receiving one or more copies of an operation message and selecting to operate on one of the processor messages, the device multicasts the event messages including the second state information to the subset of the processors that is communicably connected and not only to the processor whose message was selected to be operated on.

[0239] Aspect 26. The system of any one of Aspects 1-25, wherein an event message includes an acknowledgment indicating that a device performed an operation of an operation message, and the second state information of the event message is identical to the first state information of the operation message.

[0240] Aspect 27. A method comprising: receiving, at respective processors, event messages that encode data representing second state information; executing, by the processors, instructions of steps in a script; transmitting, from the processor, operation messages encoding first state information and / or operations generated by executing the respective instructions; generating, at respective devices, the event messages, wherein some of the event messages encode acknowledgments; performing, at the devices, (i) one or more tasks and / or (ii) operations of one or more operation messages received from one or more of the processors, wherein, for each of the devices, a device multicasts the event messages including the second state information to a subset of the processors that is communicably connected with the device, and the second state information provides stigmergy among the processors without communication among the processors.

[0241] Aspect 28. The method of Aspect 27, wherein; the second state information, which is received in a multicast event message from a device, includes process-step information indicating what step in the script informed a most recent operation performed by the device, wherein theprocess-step information indicates a step of the process associated with an operation message causing the most recent operation, and the method further comprises: determining, the respective processors, a subsequent step in the process using the process-step information in the second state information, resulting in the subset of the processors that receive the multicast event message each executing a same subsequent step in the process without communication among the subset of the processors.

[0242] Aspect 29. The method of any one of Aspects 27-28, further comprising, when a subsequent step in the process is triggered: electing, by a processor, to not execute the subsequent step and to not generate an operation message corresponding to the subsequent step, when the processor determines that a datum at the processor that is used in the subsequent step is not fresh.

[0243] Aspect 30. The method of any one of Aspects 27-29, wherein the processor is configured to determine the datum is fresh when a difference of a current time and a time stamp corresponding to when the datum was acquired is within a predefined freshness period.

[0244] Aspect 31. A decentralized system comprising controllers that can make autonomous decisions without coordination among the controllers.

[0245] Aspect 32. A decentralized system comprising a cohort of controllers that can prepare and send redundant copies of messages to a destination.

[0246] Aspect 33. A decentralized system comprising a cohort of controllers that receive data directly from external devices, and do not require a network among the controllers to synchronize the data.

[0247] Aspect 34. The decentralized system of any of Aspects 31-33 wherein each received datum has an associated freshness period and a method by which controllers may determine whether a datum is fresh using the freshness equation= AFr- (mt- et) - Amet, or its derivatives.

[0248] Aspect 35. The decentralized system of any of Aspects 31-34 wherein each received datum has an associated freshness period and a method by which controllers may determine whether a datum is fresh without using a real-world clock.

[0249] Aspect 36. The decentralized system of any of Aspects 31-35, comprising a logical division automation scripts to non-blocking, short-running functions, where each function may be executed on a different subset of controllers.

[0250] Aspect 37. The decentralized system of any of Aspects 31-36, comprising a scheme by which controllers execute functions with temporal diversity but in lockstep within a short time window, without synchronization or coordination.

[0251] Aspect 38. The decentralized system of any of Aspects 31-37, comprising an idempotent scheme where messages or commands belonging to a redundant batch of copies has a unique identifier that was not determined by consensus or a leader.

[0252] Aspect 39. The decentralized system of any of Aspects 31-38, wherein the idempotent scheme where the unique identifier is a fixed object known and common to the participating computer processes, such as a related program counter.

[0253] Aspect 40. The decentralized system of any of Aspects 31-38, wherein the idempotent scheme includes that the unique identifier is a unique datum that is external to the participating computer processes, such as the time of an event that arrives at a participating cohort.

[0254] Aspect 41. The decentralized system of any of Aspects 31-38, wherein the idempotent scheme includes that the unique identifier is a tuple of a value and unique identifier, such as a run number and program counter.

[0255] Aspect 42. The decentralized system of any of Aspects 31-41, comprising a semantics of acknowledgments (ACK) and negative acknowledgments (NAK), where the ACK and NAK are not associated only with the originating sender; instead, they are semantically associated with the source (a script that may run on any controller).

Claims

CLAIMSWhat is claimed is:

1. A system comprising: processors configured to execute, in response to receiving an event message, one or more steps in a script, and configured to transmit operation messages encoding first state information and / or operations generated by executing a step of the script; devices configured to generate event messages encoding data representing second state information and / or representing acknowledgments, and the devices being configured to perform (i) one or more tasks and / or (ii) operations of one or more operation messages received from one or more of the processors, wherein, for each of the devices, a device multicasts the event messages including the second state information to a subset of the processors that is communicably connected with the device, and the second state information provides stigmergy among the processors without communication among the processors.

2. The system of claim 1, wherein the second state information, which is received in a multicast event message from a device, includes process-step information indicating what step in the script informed an operation performed by the device, and the processors are configured to determine a subsequent step in the script using the processstep information in the second state information, resulting in the subset of the processors that receive the multicast event message each handling a same subsequent step in the script without communication among the subset of the processors.

3. The system of claim 2, wherein upon a subsequent step in the script being triggered, a processor elects to not execute the subsequent step and to not generate an operation message corresponding to the subsequent step, when the processor determines that a datum at the processor that is used in the subsequent step is not fresh.

4. The system of claim 3, wherein the processor is configured to determine the datum is fresh using a timestamp corresponding to when the datum was generated, the event message timestamp in whichthe event was sent, the maximum time a message can reside in the network before it is no longer valid, and the datum freshness period.

5. The system of claim 1, wherein for each of the devices, a device is configured to compare a state of the device with the first state information received in an operation message to determine a comparison result, and the device is further configured to use the comparison result to determine whether to perform an operation of the operation message, wherein the state of the device is based on the first state information received in one or more prior operation messages.

6. The system of claim 5, wherein the device is configured to not perform an operation of the operation message when the comparison result indicates that the operation message is obsolete.

7. The system of claim 5, wherein the device is configured to not perform an operation of the operation message when the comparison result indicates that the first state information of the operation message matches the first state information of another operation message for which the operations have been performed on the device.

8. The system of claim 1, wherein: the first state information encoded in a given operation message includes a first identifier that identifies which step in the script was executed to generate the given operation message, and the second state information that is multicast, in a given event message, from a given device to a given subset of the processors includes a second identifier representing a state of the given device including information of the first identifier of a most recent operation that was performed in response to an operation message.

9. The system of claim 1, wherein the first state information of the operation messages includes a first identifier comprising two or more shared indices including a first index representing the script and a second index representing the step of the script corresponding to an operation.

10. The system of claim 9, wherein two or more shared indices further include an episode indicator.

11. The system of claim 10, wherein the episode indicator is a value selected by a device based on proposed values from two or more processors.

12. The system of claim 10, wherein the episode indicator is encoded from the arrival of a common object that is external to the processors, the common object being an object that is referenced byeach of the processors generating a same value for the episode indicator of the two or more shared indices.

13. The system of claim 12, wherein the common object is an event message that was multicast to the processors and the processors that receive the event message encode the episode indicator using one or more fields of the event message.

14. The system of claim 10, wherein the episode indicator is determined using a random number generator to generate one or more random numbers, wherein a combined probability of the one or more random numbers provides a predefined confidence that a combination of the one or more random numbers uniquely identifies the episode indicator.

15. The system of claim 10, wherein the episode indicator originates on a device.

16. The system of claim 1, wherein one or more values of the first state information of an operation message originates from one of the devices.

17. The system of claim 1, wherein one or more values of the second state information of an event message originates from one of the processors.

18. The system of claim 1, wherein the first state information includes an index that is an integer in a range [0, n-1], and each of the processors receiving a multicast event message independently generates a same value for the index by generating the index using information of the multicast event message.

19. The system of claim 1, wherein: the event messages include acknowledgment messages, measurement messages, heartbeat messages, occurrence messages and / or combinations thereof, the acknowledgment messages including the second state information and an acknowledgment of handling at a device of another message, the measurement messages including the second state information and one or more values of measurements obtained by the device, the heartbeat messages being a periodic report of a status of the device, the status including the second state information, and the occurrence messages including the second state information and indicia that predefined criteria have occurred.

20. The system of claim 1, wherein the second state information provides stigmergy among the processors by facilitating lockstep execution of the script among the subset of the processors, wherein the subset of the processors independently use the second state information to select a subsequent step of the script to execute.

21. The system of claim 1, wherein: the processors encode, in the first state information of an operation message, one or more values representing a step in the script that was used to generate the operation message, and the devices use the one or more values received in the operation message to select which operations to perform, omitting operations of the operation message when the one or more values indicate that the operation message duplicates a previously received operation message and that the device performed the operations of the previously received operation message.

22. The system of claim 1, wherein: the first state information designates the operation messages from a processor in accordance with which steps in the scripts were executed to generate the operation messages, the second state information designates the event messages from a device in accordance with which operation messages directed the operations performed by the device, and a combination of the first state information and the second state information provides lockstep execution among the processors by uniquely identifying to the devices which steps of the script are associated with the respective operation messages and informing the processors to which steps of the script the event messages correspond.

23. The system of claim 1, wherein: the script is an automation script; the steps are automation steps; the first state information is one or more shared indices are produced independently throughout the system; and the first state information of an operation message uniquely identifies a respective step of the script that generated the operation message.

24. The system of claim 1, wherein: the processors are configured such that, when a processor does not participates in executing one or more steps of the script, the processor uses the second state information of an event messagefrom a device to rejoin an execution of a script by using the second state information to determine a subsequent step of the script to be performed, and the event message is generated by the device in response to performing an operation of the current step of the script.

25. The system of claim 1, wherein the devices are configured such that, in response to receiving one or more copies of an operation message and selecting to operate on one of the processor messages, the device multicasts the event messages including the second state information to the subset of the processors that is communi cably connected and not only to the processor whose message was selected to be operated on.

26. The system of claim 1, wherein an event message includes an acknowledgment indicating that a device performed an operation of an operation message, and the second state information of the event message is identical to the first state information of the operation message.

27. A method comprising: receiving, at respective processors, event messages that encode data representing second state information; executing, by the processors, instructions of steps in a script; transmitting, from the processor, operation messages encoding first state information and / or operations generated by executing the respective instructions; generating, at respective devices, the event messages, wherein some of the event messages encode acknowledgments; performing, at the devices, (i) one or more tasks and / or (ii) operations of one or more operation messages received from one or more of the processors, wherein, for each of the devices, a device multicasts the event messages including the second state information to a subset of the processors that is communicably connected with the device, and the second state information provides stigmergy among the processors without communication among the processors.

28. The method of claim 27, wherein; the second state information, which is received in a multicast event message from a device, includes process-step information indicating what step in the script informed a most recent operationperformed by the device, wherein the process-step information indicates a step of the process associated with an operation message causing the most recent operation, and the method further comprises: determining, the respective processors, a subsequent step in the process using the processstep information in the second state information, resulting in the subset of the processors that receive the multicast event message each executing a same subsequent step in the process without communication among the subset of the processors.

29. The method of claim 27, further comprising, when a subsequent step in the process is triggered: electing, by a processor, to not execute the subsequent step and to not generate an operation message corresponding to the subsequent step, when the processor determines that a datum at the processor that is used in the subsequent step is not fresh.

30. The method of claim 29, wherein determining that the datum is not fresh includes determining, at the processor, that the datum is fresh when a difference of a current time and a timestamp corresponding to when the datum was generated, the event message timestamp in which the event was sent, the maximum time a message can reside in the network before it is no longer valid, and the datum freshness period.

31. The method of claim 27, further comprising: comparing, by the respective devices, a state of a device with the first state information received in an operation message received by the device to determine a comparison result, and using the comparison result to determine whether to perform an operation of the operation message, wherein the state of the device is based on the first state information received in one or more prior operation messages..

32. The method of claim 31, further comprising: determining, by the device, to not perform an operation of the operation message when the comparison result indicates that the operation message is obsolete.

33. The method of claim 31, further comprising: determining, by the device, to not perform an operation of the operation message when the comparison result indicates that the first state information of the operation message matches the firststate information of another operation message for which the operations have been performed on the device.

34. The method of claim 27, wherein: the first state information encoded in a given operation message includes a given operation message includes a first identifier of which step in the script and / or which instructions in the script were executed to generate the given operation message, and the second state information that is multicast, in a given event message, from a given device to a given subset of the processors includes a second identifier representing a state of the given device including information of the first identifier of a most recent operation that is based on an operation message,35. The method of claim 27, wherein the first state information of the operation messages includes a first identifier comprising two or more shared indices including a first index representing the script and a second index representing the step of the script corresponding to an operation.

36. The method of claim 35, wherein two or more shared indices further include an episode indicator.

37. The method of claim 36, wherein the episode indicator is: a value selected by a device based on proposed values from two or more processors; or determined using a random number generator to generate one or more random numbers, wherein a combined probability of the one or more random numbers provides a predefined confidence that a combination of the one or more random numbers uniquely identifies the episode indicator.

38. The method of claim 36, wherein the episode indicator is encoded from the arrival of a common object that is external to the processors, the common object being an object that is referenced by each of the processors generating a same value for the episode indicator of the two or more shared indices39. The method of claim 38, wherein the common object is an event message that was multicast to the processors and the processors that receive the event message encode the episode indicator using one or more fields of the event message.

40. The method of claim 36, wherein the episode indicator originates on a device.

41. The method of claim 27, wherein one or more values of the first state information of an operation message originates from one of the devices.

42. The method of claim 27, wherein one or more values of the second state information of an event message originates from one of the processors.

43. The method of claim 27, wherein the first state information includes an index that is an integer in a range [0, n-1], and each of the processors receiving a multicast event message independently generates a same value for the index by generating the index using information of the multicast event message.

44. The method of claim 27, wherein: the event messages include acknowledgment messages, measurement messages, heartbeat messages, occurrence messages and / or combinations thereof, the acknowledgment messages including the second state information and an acknowledgment of handling at a device of another message, the measurement messages including the second state information and one or more values of measurements obtained by the device, the heartbeat messages being a periodic report of a status of the device, the status including the second state information, and the occurrence messages including the second state information and indicia that predefined criteria have occurred.

45. The method of claim 27, wherein the second state information provides stigmergy among the processors by facilitating lockstep execution of the script among the subset of the processors, wherein the subset of the processors independently use the second state information to select a subsequent step of the script to execute46. The method of claim 27, further comprising: encoding, by the processors, one or more values in the first state information of an operation message, the one or more values representing a step in the script used to generate the operation message, and using the one or more values received at a device that received the operation message to determine whether to perform the operations of the operation message, andomitting operations of the operation message when the one or more values indicate that the operation message duplicates a previously received operation message and that the device performed the operations of the previously received operation message.

47. The method of claim 27, wherein: the first state information designates the operation messages from a processor in accordance with which steps in the scripts were executed to generate the operation messages, the second state information designates the event messages from a device in accordance with which operation messages directed the operations performed by the device, and a combination of the first state information and the second state information provides lockstep execution among the processors by uniquely identifying to the devices which steps of the script are associated with the respective operation messages and informing the processors to which steps of the script the event messages correspond.

48. The method of claim 27, wherein: the script is an automation script; the steps are automation steps; the first state information is one or more shared indices are produced independently throughout the system; and the first state information of an operation message uniquely identifies a respective step of the script that generated the operation message49. The method of claim 27, further comprising: rejoining, by a processor, an execution of a script, such that when a processor does not participates in executing one or more steps of the script, the processor uses the second state information of an event message from a device to rejoin an execution of a script by using the second state information to determine a subsequent step of the script to be performed, and the event message is generated by the device in response to performing an operation of the current step of the script.

50. The method of claim 27, further comprising: multicasting, by a device, the event messages including the second state information to the subset of the processors that is communi cably connected with the device, such that, in response to receiving one or more copies of an operation message and selecting to operate on one of theprocessor messages, the device multicasts the event messages including the second state information to the subset of the processors that is communicably connected and not only to the processor whose message was selected to be operated on.

51. The method of claim 27, wherein an event message includes an acknowledgment indicating that a device performed an operation of an operation message, and the second state information of the event message is identical to the first state information of the operation message.

Images