Troubleshooting in a scale-out system using a vector clock

Abstract

Determining a fault correction line in a distributed scale-out computing system. Each node or process in a distributed system has, or is associated with, a vector clock that includes a logical clock for each node in the distributed system. When a fault is detected, a correction operation can be performed using the vector clock. After the correction operation, the vector clock is updated so that the fault correction line is available in the computing system.

Description

AREA OF INVENTION

[0001] Embodiments of the present invention generally relate to troubleshooting operations. In particular, at least some embodiments of the invention relate to systems, hardware, software, computer-readable media, and methods for troubleshooting operations in systems that include distributed scale-out systems. BACKGROUND

[0002] Distributed scale-out systems, which may include replication environments, can experience various types of errors. Errors in these types of systems are often a characteristic of their scale. One possible solution to this problem, which is common in distributed scale-out systems, is the presence of redundancy. While redundancy can allow some errors to be transparent, there are many cases where it is necessary to repair or replace failed operations and resume them from a point before the error occurred. To resolve an error that is not transparent, it is often necessary to process the error and restart the task or operation.

[0003] For example, distributed scale-out systems can be used to perform replication operations. If an error occurs, it may be necessary to restart the replication operations from or before the point of the error. This ensures that the replication operation is executed correctly and that data is successfully replicated without loss. The difficulty often lies in identifying the point of the error and resuming from there. BRIEF DESCRIPTION OF THE DRAWINGS

[0004] To describe how at least some of the advantages and features of the invention can be achieved, a more detailed description of embodiments of the invention follows, with reference to specific embodiments illustrated in the accompanying drawings. It should be noted that these drawings only show typical embodiments of the invention and therefore are not intended to limit its scope of protection. Embodiments of the invention are described and explained with additional precision and further detail using the accompanying drawings, wherein: Fig. 1. Aspects of a distributed scale-out computing system or such environment are revealed; Fig. Two aspects of vector clocks in a distributed scale-out computing system are revealed; Fig. Three aspects of updating vector clocks with respect to processes occurring in the distributed system are revealed; Fig. Four aspects of determining a troubleshooting line and carrying out a corrective operation were revealed; Fig. 5 aspects of an exemplary procedure for determining a fault correction line and performing a corrective operation or a cascaded corrective operation are disclosed; and Fig. 6 aspects of a computing device, system or entity are revealed. DETAILED DESCRIPTION OF SOME EXAMPLE FORMS OF EXECUTION

[0005] Embodiments of the present invention generally relate to distributed systems that include distributed scale-out systems, which are generally referred to as distributed systems. In particular, at least some embodiments of the invention relate to systems, hardware, software, computer-readable media, and methods for error correction in distributed systems. An example may include performing error correction related to replication operations. In general, exemplary embodiments of the invention also relate to distributed and / or scale-out operations, correction operations, vector clock operations, or the like, or combinations thereof.

[0006] Due to the scale of distributed systems, the probability of errors is high. While distributed systems can have sufficient redundancy to make errors transparent, there are cases where it is necessary to repair or replace failed operations and resume from the point of failure. Error correction can involve performing some type of backup operation, such as taking snapshots, and / or logging operations. Snapshots generally involve the regular saving of a process state. Logging involves recording the operations that produced the current state of the process, so that the operations can be repeated if necessary.

[0007] For example, a node in a cluster may fail and revert to a previous state, or this state may be restored. Once the node's previous state is restored, the node can repeat the operations that were previously performed between the restored state and the point of failure. This may include retweeting remote procedure calls (PRCs) or other messages (generally referred to as messages) that may have been sent to other systems / nodes / processes.

[0008] Other nodes in the cluster should be able to tolerate these repeated messages (e.g., idempotent operations), detect these messages, and / or discard them. If the other nodes are unable to tolerate these messages, they must also be reset to a previous or earlier state.

[0009] A cascading reset occurs when resetting one node (or system) to a previous state causes another node or system to also be reset to a previous state, or requires another node or system to be reset to a previous state. Eventually, the nodes reach a state where all nodes can continue working together. This state, where all nodes can continue working together, is referred to herein as the resolution line.

[0010] In one example of troubleshooting, nodes in a cluster can be reset to a fix line. Embodiments of the invention relate to finding or identifying the fix line in a distributed scale-out system using a vector clock.

[0011] When executing distributed algorithms or operations, such as resource synchronization or replication, the operations may depend on a process or environment designed to order the processes taking place. For example, a system might include two processes and a disk. The processes can send messages to each other and to the disk to request access. The disk can then grant access to the disks in the order in which the messages were received.

[0012] For example, process A might send a message to the disk requesting write access and then send a read request message to process B. Process B receives the message and, as a result, sends a read request message to the disk. If there is a time delay that causes the disk to receive both messages from process A and process B simultaneously, the disk can determine which message occurred first. The message from process A preceded the message from process B if one of two types of sequences occurs from process A to process B: forward movement while remaining in the same process, and tracing a message from sending the operation to receiving the operation. A logical clock provides a mechanism for supplying information so that the order of operations can be determined.

[0013] Causality (also known as causal relationship or cause and effect) refers to the influence with which one event, process, condition, or object (a cause) contributes to the production of another event, process, condition, or object (an effect), where the cause is partly responsible for the effect and the effect is partly dependent on the cause. Generally, a process can have many causes, also called causal factors. All causal factors lie in the past with respect to the process. An effect, in turn, can be a cause or a causal factor of many other effects that lie in the future with respect to the process.

[0014] A vector clock is an example of a mechanism for capturing chronological and causal relationships in a distributed scale-out system. A vector clock maintains a vector of logic clocks for each communication node (or host, process, etc.). Each time a message is exchanged, the sending node transmits not only what it knows about the global time state, but also the time state it receives from the other nodes.

[0015] More generally, a scalable distributed system typically refers to a computing system or configuration that grows in response to demand. For example, a scalable system can add additional resources (such as nodes) as more users are added. A scalable system can add additional resources as the workload or resource demand increases. Scalability generally refers to the addition (or removal) of software / hardware resources, such as nodes, in a computing system.

[0016] Fig. Figure 1 reveals aspects of a distributed scale-out system or a scalable distributed system. Fig. Figure 1 illustrates a distributed computing system 100 at time 101 and time 103. At time 101, the distributed computing system 100 comprises one node (or process) 102 and one node (or process) 104. The nodes 102 and 104 can be controlled or coordinated by a centralized system 106 and can run a distributed application 108.

[0017] Time 103 illustrates that scaling has occurred in computing system 100. More precisely, a node (or process) 110 has been added to the distributed computing system 100. Node 110 may have been added due to load balancing, additional users, or other reasons.

[0018] Fig. Figure 1 further illustrates that each of the nodes 102, 104, and 110 is assigned to a vector clock 112, 114, and 116, respectively. The vector clocks 112, 114, and 116 are designed to capture chronological and causal relationships in the distributed scale-out system 100.

[0019] Fig. 2 reveals aspects of vector clocks. Fig. Figure 2 illustrates a distributed scale-out system 200, which includes nodes 202, 204, and 206, which may have been added to the distributed system 200 at different times. In this example, when a node is added to the system 200, a vector clock may be initiated, and the initial values ​​of this vector clock may all be zeros, indicating that the node is unaware of any events at other nodes and that no events have occurred at the node itself.

[0020] In this example, node 202 is assigned to vector clock 208. Nodes 204 and 206 are assigned to vector clocks 210 and 212, respectively. Vector clocks 208, 210, and 212 each contain one entry for every node in system 200. Thus, vector clock 208 contains one entry for every node in system 200. In this example, the first entry in vector clock 208 corresponds to node 202, and the other entries correspond to other nodes in sequence. Therefore, the first (208), second (210), and third (212) entries of the vector clocks correspond to the first (202), second (204), and third (206) nodes in system 200, respectively. For example, the second entry in node 210 corresponds to node 204, and the third entry in vector clock 212 corresponds to node 206.

[0021] At time 214, vector clocks 208, 210, and 212 are all zero. At time 216, events have occurred between time 214 and time 216. Vector clock 208 at time 216 indicates that node 202 is aware of two events that occurred at node 202 and that node 202 is not aware of any events at the other nodes 204 and 206. Vector clock 210 illustrates that node 204 is aware of two events that occurred at node 204 itself and two events that occurred at node 202. Node 204 is not aware of any events at node 206. Vector clock 212 at time 216 indicates that node 206 is only aware of one event that occurred at node 206.

[0022] Fig. Section 3 reveals aspects of vector clocks and changes to vector clocks in distributed computing systems. Fig. Figure 3 illustrates nodes 302, 304, and 306, which operate in a distributed system and are associated with a timeline 300. Fig. 3. Internal events that take place at a node are represented as circles, and events that are associated with messages between nodes are represented as rectangles.

[0023] For example, node 302 experiences internal events 308, 310, 312, and 316, and one external event 314. Node 304 experiences internal events 318, 324, and 326, and external events 320 and 322. Node 306 experiences internal events 328 and 330.

[0024] Fig. Figure 3 illustrates a vector clock for each of the three nodes 302, 304, and 306. The vector clock is shown next to each of the illustrated events. In this example, and because there are three nodes, the vector clock has the format [x,y,z], where x corresponds to node 302, y to node 304, and z to node 306.

[0025] Thus, each node contains or is associated with a vector clock. The number of entries in the vector clock corresponds to the number of nodes in the distributed system. Therefore, the number of entries in the vector clock can change as the system scales. Each entry in the vector clock is a logical clock of a specific node in the distributed system.

[0026] In an exemplary embodiment, the vector clock is updated according to the following rules, which are given as an example. The first rule is that every time an internal event occurs at a node, the value of that node's logical clock in the vector clock is incremented by 1. The second rule is that every time a node sends a message to another node, the value of the logical clock for the receiving node is incremented by one.

[0027] For example, when event 308 occurs, the logic clock of node 302 is incremented to 1 in the vector clock of node 302. At event 310, the node executes the first rule and increments the logic clock of node 302 from 1 to 2.

[0028] When event 318 occurs at node 304, the logic clock (at the second position) of node 304's vector clock is incremented to 1. With event 320, the second rule can also be executed because a message is received from node 302 and is associated with event 310. In this example, the event causes node 304's logic clock in its vector clock to be incremented from [0,1,0] to [0,2,0]. Simultaneously, the information shared by node 302 is used to update node 304's logic clock in its vector clock. Specifically, node 304's logic clock in its vector clock changes from [0,2,0] to [2,2,0] because the vector clock [2,0,0] is received from node 302.

[0029] For node 302, the logic clock of node 302 (first position of the vector clock) changes from 1 to 5 because 5 events are reported. The vector clock associated with event 316 illustrates that node 302 is informed about 4 events (from the logic clock of node 304 at the second position) of node 304 and 1 event of node 306. Fig. Figure 3 also illustrates that some nodes may initially be unaware of some events at other nodes. For example, node 302, whose vector clock is [5,4,1] at event 316, is not aware of event 330.

[0030] Fig. Figure 3 thus illustrates examples of vector clock changes for the vector clocks of each of the nodes 302, 304 and 306 contained in the distributed scaleout system 340.

[0031] Fig. Section 4 reveals aspects of determining a fault correction line in a distributed scale-out system. This section discusses... Fig. The illustrated example 4 demonstrates how to establish a troubleshooting approach in a cluster using snapshots and logging. In this example, the distributed system 440 comprises nodes 402, 404, and 406.

[0032] In Fig. In example 4, node 402 is started at 7:00 AM. A vector clock 414 represents event 408. In one example, a monotonically increasing atom generator number is used to track the event in vector clock 414. In this example, snapshot S1 is associated with the generation of the number 100. An internal reference can also be stored to set S1 to 100 at 7:00 AM (time). Thus, S1 is referenced by the generator number 100 in vector clock 414.

[0033] At 8:00 AM, the load on system 440 increases, and node 404 is added. As a result of the addition of node 404, a snapshot S2 is created. Snapshot S2 is identified by generator number 101 in the vector clock. Thus, vector clock 416, which represents or is associated with event 410 and is assigned to node 404, includes generator number 101 for the event of snapshot S2 and reflects the information received from node 402, namely generator number 100, which is assigned to snapshot S1 at node 402.

[0034] System 440 may have a strategy to take a snapshot every 2 hours. Thus, at 9:00 AM, a snapshot S3 is taken at node 402. Snapshot S3, which is associated with event 412, is identified by generator number 102. The vector clock 418 of node 402 is therefore updated to reflect that node 402 is associated with generator numbers 100 and 102. The vector clock 418 of node 402 is updated to include the information received from node 404, namely that node 404 is associated with generator number 101 (or snapshot S2). In this example, vector clocks 414 and 418 represent the same vector clock of node 402 at different times. At the time of event 412, the vector clock of node 404 is stationary [100,101,0].

[0035] At 10:00 AM, the load increases, so node 406 is added to system 440. A snapshot S4 is taken and identified by generator number 103. Thus, the vector clock of node 406, which is associated with event 422, reflects generator number 103 of event 422 and the information received from vector clock 418 of node 412. Fig. Figure 4 thus illustrates changes in a distributed system 440 and the effect of these changes or events on the vector clocks of nodes 402, 404 and 406 of the system 440.

[0036] In one example, it is assumed that snapshot S4 is taken when node 406 is added to system 440 (e.g., a cluster). However, adding node 406 resulted in an error, and it is necessary to restore system 440. In this example, vector 420 is evaluated. Vector 420 indicates that four snapshots were taken: two snapshots at node 402 with generator numbers 100 and 102, one snapshot at node 404 with generator number 101, and one snapshot at node 406 with generator number 103. Thus, vector 420 is: [100 & 102, 101, 103].

[0037] Since the error occurred when adding node 406, embodiments of the invention restore the last known state using the vector clock 420. The last known state is that the snapshot with generator number 102 was taken at node 402. Therefore, system 440 restores this state using snapshot S3, and the logs are repeated.

[0038] Since the logs are repeated starting with snapshot S3, an error is detected at 9:50 AM. System 440 can then determine that node 406 can be restarted by repeating the logs up to 9:50 AM (the time of the error). Snapshot S4, which is associated with generator number 103, is then discarded, and a new snapshot S5 is created on node 406. The internal reference relates snapshot S5 at 9:50 AM (time) to generator number 104. The vector clock of node 406 is also updated. At this point in the operation, a remediation line is created in system 440 and is represented by vector clock 424 with [100&102, 101, 104].

[0039] If a cascaded rollback becomes necessary (e.g., snapshot S3 is corrupted), the remediation process would start at node 404 with snapshot S2 and generator number 101. Logs would be re-run, and generator numbers would be decremented and replaced as needed with new generator numbers assigned to the new snapshots. Therefore, in the case of a cascaded rollback, snapshot S3 is discarded, logs are re-run from snapshot S2 onward, and snapshots S3 and S4 are created with new generator numbers to ensure a remediation line is available. In this example, the vector clock would be vector clock 426 with [100&103, 101, 104] after re-running the logs and re-capturing the snapshots at nodes 402 and 406.

[0040] Conventional snapshots taken in a cluster or distributed system do not capture the chronological and causal relationships in a distributed scale-out system. Embodiments of the invention provide a vector clock that captures chronological and causal relationships in a scale-out system and helps to create a troubleshooting line more quickly. Embodiments of the invention facilitate troubleshooting when cascaded rollbacks are involved, and creating a troubleshooting line involves rollbacks at multiple nodes in a cluster. A vector clock that considers chronological and causal relationships helps to create troubleshooting lines faster than with conventional snapshots.

[0041] Fig. Procedure 500 discloses aspects of determining a fix line and performing a fix operation in a distributed scale-out computing system. Procedure 500 involves assigning a vector clock to each node (or process, host, etc.) operating in the distributed system. The vector clock can be designed to change its size or length as nodes are added to or removed from the distributed system. Alternatively, the vector clock can have a size corresponding to the maximum number of possible nodes in the distributed system (thus, many of the entries would be 0 until the corresponding nodes are added to the system). Each entry in the clock vector is a logical clock for a corresponding node. Thus, each entry corresponds to a specific node in the distributed system. With respect to a given node, the clock vector includes a logical home clock and logical away clocks.Each of the nodes can receive a copy of the vector clock, all of which are initially set to zero.

[0042] When an event occurs at a node, that node's vector clock is updated. If the event is internal, only that node's vector clock is updated. If the event is associated with another node, or if a message is sent, the message contains a copy of that node's vector clock. This allows the receiving node to update its own vector clock to reflect the information contained in the sending node's vector clock. When a message is received, the receiving node also updates its own logical clock in the vector clock to reflect the event.

[0043] Next, the distributed system can determine 506 that a fault has occurred in the distributed system. In response to the determination that a fault has occurred, a reset operation is performed using the vector clock 508. The vector clock establishes a chronological and causal relationship between the nodes. For example, if a fault is determined at a first node, the vector clock of the first node is evaluated to identify a previous state of the distributed system, which may be associated with another node. In one embodiment, the vector clock can identify the node that had the most recent previous state.

[0044] After (or during) the remediation operation, the fault location can be determined and a fault line remediation is determined. Determining the fault line remediation can involve discarding a logic clock in the vector clock and replacing the discarded logic clock with a new one. This ensures that the vector clock still creates a fault line even after the fault has been remedied. Determining a fault line remediation can also involve performing the remediation operation. This can include restoring a snapshot, performing or repeating entries in a log, or the like, until the distributed system is restored. In one example, fault line remediation can also involve performing a cascaded reset operation.

[0045] Embodiments of the invention, such as the examples disclosed herein, can be advantageous in several respects. As is apparent from the present disclosure, for example, one or more embodiments of the invention can provide one or more advantageous and unexpected effects in any combination, some examples of which are set forth below. It should be noted that such effects are neither intended nor are to be seen as any limitation of the scope of protection of the claimed invention. It should also be noted that nothing herein is to be understood as an essential or indispensable element of the invention or of any embodiment. On the contrary, various aspects of the disclosed embodiments can be combined in various ways to define further embodiments. Such further embodiments are to be considered as included within the scope of protection of this disclosure.Furthermore, none of the embodiments included in the scope of protection of this disclosure should be considered as solving one or more specific problems. Nor should any such embodiment be seen as implementing, or being limited to implementing, several specific technical effects or solutions. Finally, it is not necessary for any embodiment to implement any of the advantageous and unexpected effects disclosed herein.

[0046] It should be noted that embodiments of the invention, whether claimed or not, cannot be practically or otherwise carried out in the mind of a person. Accordingly, nothing herein should be understood as teaching or suggesting that any aspect of any embodiment of the invention could be practically or otherwise carried out in the mind of a person. Furthermore, unless explicitly stated otherwise herein, the disclosed methods, processes, and operations are designed to be implemented by computing systems, which may include hardware and / or software. That is to say, such methods, processes, and operations are defined as computer-implemented.

[0047] The following is an explanation of aspects of exemplary operating environments for various embodiments of the invention. This explanation is not intended to limit the scope of protection of the invention or the applicability of the embodiments in any way.

[0048] In general, embodiments of the invention can be implemented in conjunction with systems, software, and components that individually and / or collectively implement and / or effect data protection operations, which may, without limitation, include data operations, data replication operations, scaling operations, vector clock-related operations, or the like. More generally, the scope of protection of the invention includes any operating environment in which the disclosed concepts may be useful.

[0049] At least some embodiments of the invention provide the implementation of the disclosed functionality in backup platforms, examples of which include the Dell-EMC NetWorker and Avamar platforms and associated backup software, and in storage environments such as the Dell-EMC DataDomain storage environment. In general, however, the scope of protection of the invention is not limited to a specific data backup platform or data storage environment.

[0050] New and / or modified data collected and / or generated in connection with certain embodiments may be stored in a data environment, which may take the form of a public or private cloud storage environment, an on-premises storage environment, or a hybrid storage environment that includes both public and private elements. Each of these exemplary storage environments may be partially or fully virtualized. The storage environment may include or consist of a data center that is operational to handle read, write, delete, backup, restore, and / or clone operations initiated by one or more clients or other elements of the operational environment.

[0051] Exemplary cloud computing environments, which may or may not be public, include storage environments that provide data protection functionality for one or more clients. Another example of a cloud computing environment is one in which processing, data protection, and other services are performed for one or more clients. Some exemplary cloud computing environments in connection with which embodiments of the invention can be implemented include, but are not limited to, Microsoft Azure, Amazon AWS, Dell EMC Cloud Storage Services, and Google Cloud. More generally, however, the scope of protection of the invention is not limited to the implementation of any particular type or implementation of a cloud computing environment.

[0052] In addition to the cloud environment, the operational environment can also include one or more clients capable of performing various operations, such as collecting, modifying, and generating data. Thus, a specific client can utilize one or more instances of each of one or more applications, or be otherwise associated with them, that perform such data operations. These clients can include physical machines, containers, or virtual machines (VMs).

[0053] It should be noted that all disclosed processes, operations, procedures, and / or any part thereof can be performed in response to, as results of, and / or based on the performance of one or more preceding processes, procedures, and / or operations. Accordingly, the performance of one or more processes can, for example, be a condition or a trigger for a subsequent performance of one or more processes, operations, and / or procedures. Thus, for instance, the various processes comprising a procedure can be connected or otherwise associated with one another through relationships such as those cited in the preceding examples.Finally, although this is not necessary, in some embodiments the individual processes comprising the various exemplary methods disclosed herein are carried out in the specific sequence given in these examples. In other embodiments, the individual processes comprising a disclosed method may be carried out in a different sequence than the one specifically given.

[0054] The following are some further exemplary embodiments of the invention. These are given only as examples and are not intended to limit the scope of protection of the invention in any way.

[0055] Embodiment 1. Method comprising: Assigning a vector clock to each node in a distributed scale-out system when an event occurs at a node, updating a vector clock assigned to the node, determining a fault in the distributed scale-out system, performing a reset operation using the vector clocks, wherein the vector clocks include a fault correction line for correcting a fault in the distributed scale-out system, and performing a corrective operation based on the fault correction line.

[0056] Embodiment 2. Method according to embodiment 1, wherein each of the vector clocks comprises a logical home clock for a corresponding node and one or more logical outgoing clocks for other nodes in the distributed scaleout system.

[0057] Embodiment 3. Method according to embodiment 1 and / or 2, wherein the event is an internal event, further comprising updating the logical home clock in the vector clock of the node that experienced the internal event.

[0058] Embodiment 4. Method according to embodiment 1, 2 and / or 3, wherein the event is an external event, further comprising updating a logical home clock in the vector clock of the node and updating the vector clock of the node using a vector clock of a sending node that sent the event to the node.

[0059] Embodiment 5. Method according to embodiments 1, 2, 3 and / or 4, further comprising performing a cascaded reset operation using the vector clocks.

[0060] Embodiment 6. Method according to embodiments 1, 2, 3, 4 and / or 5, wherein the remediation operation includes restoring from a snapshot and re-logging, the logs storing a chronological history of events in the distributed scale-out system.

[0061] Embodiment 7. Method according to embodiments 1, 2, 3, 4, 5 and / or 6, further comprising updating the vector clocks so that the vector clocks include a fault correction line which justifies the correction operation.

[0062] Embodiment 8. Method according to embodiments 1, 2, 3, 4, 5, 6 and / or 7, wherein some entries in the vector clocks are deleted and replaced by new entries.

[0063] Embodiment 9. Method according to embodiments 1, 2, 3, 4, 5, 6, 7 and / or 8, further comprising scaling the distributed computing system, wherein a vector clock of a new node added to the distributed computing system is initially set to zero.

[0064] Embodiment 10. Method according to embodiments 1, 2, 3, 4, 5, 6, 7, 8 and / or 9, further comprising moving several nodes forward from a rectification line after using the fault rectification line to identify the rectification line.

[0065] Embodiment 11. Method for carrying out any one of the operations, methods or processes, or any part thereof, or any combination thereof, as disclosed herein.

[0066] Embodiment 12. Non-transitory storage medium on which instructions are stored which can be executed by one or more hardware processors to perform operations which include the operations according to one or more of embodiments 1 to 11.

[0067] The embodiments disclosed herein may include the use of a specialized or general-purpose computer comprising various computer hardware or software modules, as further explained below. A computer may include a processor and a computer storage medium on which instructions are stored that, when executed by the processor and / or when caused by the processor to execute them, perform one or more of the methods disclosed herein or any part or part thereof of any one of the disclosed methods.

[0068] As noted above, embodiments within the scope of protection of the present invention also include computer storage media, which are physical media on which computer-executable instructions or data structures are stored or present. Such computer storage media can be any physical medium accessible to a general-purpose or specialized computer.

[0069] As an example, and without limitation, such a computer storage medium may include hardware memory such as a solid-state drive (SSD), RAM, ROM, EEPROM, CD-ROM, flash memory, phase-change memory (PCM), or other optical disk storage, magnetic disk storage, or other magnetic storage devices, or any other hardware storage devices that can be used to store program code in the form of computer-executable instructions or data structures and that a general-purpose or specialized computer system can access and execute in order to implement the disclosed functionality of the invention. Combinations of the above are also included in the scope of computer storage media.Such media are also examples of non-transitory storage media, and non-transitory storage media also include cloud-based storage systems and structures, although the scope of protection of the invention is not limited to these examples and non-transitory storage media.

[0070] Computer-executable instructions include, for example, instructions and data that, when executed, cause a general-purpose computer, a specialized computer, or a specialized processing device to perform a specific function or group of functions. As such, some embodiments of the invention may be downloadable to one or more systems or devices, for example, from a website, a mesh topology, or another source. Furthermore, the scope of protection of the invention includes all hardware systems or devices that comprise an application incorporating the disclosed executable instructions.

[0071] Although the subject matter has been described in a language specific to structural features and / or methodological processes, it is understood that the subject matter defined in the accompanying claims is not necessarily limited to the specific features or processes described above. Rather, the specific features and processes disclosed herein are disclosed as exemplary forms of implementing the claims.

[0072] As used herein, the terms module, component, engine, client, service, or the like may refer to software objects or routines that run on the computing system. These may be implemented as objects or processes that run on the computing system, for example, as separate threads. While the system and procedures as described herein may be implemented in software, implementations in hardware or a combination of software and hardware are also possible and intended. In the present disclosure, a 'computing entity' may be any computing system as defined herein, or any module or combination of modules that runs on a computing system.

[0073] In at least some cases, a hardware processor is provided that is operational to execute instructions in order to carry out a method or process, e.g., the methods and processes disclosed herein. The hardware processor may or may not include an element of other hardware, e.g., the computing devices and systems disclosed herein.

[0074] With regard to computing environments, embodiments of the invention can be executed in client-server environments, whether network or local environments, or in any other suitable environment. Suitable operating environments for at least some embodiments of the invention include cloud computing environments, in which one or more clients, servers, or other machines may be present and operate in a cloud environment.

[0075] With brief reference to Fig. 6. One or more of the entities disclosed or implied by the figures and / or elsewhere herein may take the form of, comprise, be implemented in, or be hosted on a physical computing device, with an example being designated as 600. If any of the foregoing elements comprises or consists of a virtual machine (VM) or container, such VM or container may be a virtualization of any combination of the entities described herein. Fig. 6 revealed physical components.

[0076] In the example of Fig.6. The physical computing device 600 comprises a memory 602, which may include one, some, or all of a random-access memory (RAM), non-volatile memory (NVM) 604, such as an NVRAM, a solid-state memory (ROM), and a persistent memory, one or more hardware processors 606, a non-transitory storage medium 608, a UI device 610, and a data storage 612. One or more of the memory components 602 of the physical computing device 600 may be in the form of a solid-state device (SSD) memory. Furthermore, one or more applications 614 may be provided, comprising instructions that can be executed by one or more hardware processors 606 to perform any of the operations disclosed herein, or parts thereof.

[0077] Such executable instructions can take various forms, for example, instructions that are executable to perform any procedure or part thereof disclosed herein, and / or that are executable by / at a storage location, whether on-premises at a company or a cloud computing location, a client, a data center, a data protection location that includes a cloud storage location, or a backup server, to perform any of the functions disclosed herein. Furthermore, instructions can be executable to perform any other operations and procedures disclosed herein, and any parts thereof.

[0078] The present invention can be implemented in other specific forms without departing from its spirit or essential features. The described embodiments are to be regarded in every respect as merely illustrative and not as limiting. The scope of protection of the invention is therefore specified by the accompanying claims and not by the preceding description. All modifications that fall within the meaning and equivalence of the claims are included within their scope of protection.

Claims

[1] Method comprising: Assigning a vector clock to each node in a distributed scaleout system; when an event occurs at a node, updating a vector clock associated with the node; Determine a fault in the distributed scaleout system; Performing a reset operation using the vector clocks, wherein the vector clocks include a fault recovery line for recovering from a fault of the distributed scaleout system, and Perform a remediation operation based on the troubleshooting line. [2] The method of claim 1, wherein each of the vector clocks comprises a logical home clock for a corresponding node and one or more logical away clocks for other nodes in the distributed scaleout system. [3] The method of claim 1, wherein the event is an internal event, further comprising updating the logical home clock in the vector clock of the node that experienced the internal event. [4] The method of claim 1, wherein the event is an external event, further comprising updating a logical home clock in the vector clock of the node and updating the vector clock of the node using a vector clock of a sending node that sent the event to the node. [5] The method of claim 1, further comprising performing a cascaded reset operation using the vector clocks. [6] The method of claim 5, wherein the remediation operation comprises restoring from a snapshot and replaying logs, the logs storing a chronological history of events in the distributed scaleout system. [7] The method of claim 6, further comprising updating the vector clocks such that the vector clocks include a debug line that establishes the debug operation. [8] A method according to claim 7, wherein some entries in the vector clocks are deleted and replaced by new entries. [9] The method of claim 1, further comprising scaling the distributed computing system, wherein a vector clock of a new node added to the distributed computing system is initially set to zero. [10] The method of claim 1, further comprising advancing a plurality of nodes from a fix line after using the fix line to identify the fix line. [11] Non-transitory storage medium on which instructions are stored that are executable by one or more hardware processors to perform operations that include: Assigning a vector clock to each node in a distributed scaleout system; when an event occurs at a node, updating a vector clock associated with the node; Determine a fault in the distributed scaleout system; Performing a reset operation using the vector clocks, wherein the vector clocks include a fault recovery line for recovering from a fault of the distributed scaleout system, and Perform a remediation operation based on the troubleshooting line. [12] The non-transitory storage medium of claim 11, wherein each of the vector clocks comprises a logical home clock for a corresponding node and one or more logical away clocks for other nodes in the distributed scaleout system. [13] The non-transitory storage medium of claim 11, wherein the event is an internal event, further comprising updating the logical home clock in the vector clock of the node that experienced the internal event. [14] The non-transitory storage medium of claim 11, wherein the event is an external event, further comprising updating a logical home clock in the vector clock of the node and updating the vector clock of the node using a vector clock of a sending node that sent the event to the node. [15] The non-transitory storage medium of claim 11, further comprising performing a cascaded reset operation using the vector clocks. [16] The non-transitory storage medium of claim 15, wherein the remediation operation comprises restoring from a snapshot and replaying logs, the logs storing a chronological history of events in the distributed scaleout system. [17] The non-transitory storage medium of claim 16, further comprising updating the vector clocks such that the vector clocks include a recovery line that establishes the recovery operation. [18] A non-transitory storage medium according to claim 17, wherein some entries in the vector clocks are deleted and replaced by new entries. [19] The non-transitory storage medium of claim 11, further comprising scaling the distributed computing system, wherein a vector clock of a new node added to the distributed computing system is initially set to zero. [20] The non-transitory storage medium of claim 11, further comprising advancing a plurality of nodes from a recovery line after using the error recovery line to identify the recovery line.

Images