A fully consistent, high-performance off-local storage cluster file system

Abstract

Heatmaps are used to identify access patterns of a series of data blocks at specific times to optimize the size of the replication node set for the purpose of reducing access latency. In one embodiment, heatmaps are used to enforce minimal replication for each data block, such as when a replication node crashes and is replaced. In another embodiment, heatmaps are used to adjust the minimum replication of data blocks during and after peak demand periods. In one embodiment, each data block is replicated on the minimum number of corresponding replication nodes in the cluster. A first node requests access to a data block. Based on the requested access, the heatmap is modified, and the data block is replicated to the first node. Based on the heatmap, the minimum number of nodes in the corresponding replication nodes for at least one data block is adjusted.

Description

[0001] Related Cases

[0002] The relevant non-patented document (NPL) "Designing NFS With RDMA for Security, Performance and Scalability" by Ranjit Noronha et al. presented at the International Conference on Parallel Processing (ICPP 2007) in 2007 is incorporated into this paper. Technical Field

[0003] This invention relates to using heatmaps of access activity to manage data replication across multiple storage nodes. Background Technology

[0004] For file clusters, there are various ways to manage shared storage and cluster membership. Some methods use shared storage and fully connected interconnects to provide a fully connected cluster. Other cluster types involve local storage and loosely connected clusters. These allow the cluster to scale beyond what a fully connected cluster typically provides, but may have different semantics in data representation.

[0005] At the file system level, a distributed file system can be implemented in at least two ways:

[0006] • Inconsistency – Files are fully replicated to each machine to prevent the total storage of the cluster from exceeding the storage of a single machine. No machine knows about updates to a specific file made by another machine. If two machines modify a file simultaneously, the last modification wins.

[0007] • Limited consistency – Files are replicated to a subset of machines, allowing the cluster's storage to exceed the storage of any single machine. Machines are aware of modifications made by other machines, but no distributed locking or other mechanisms are provided to prevent two machines from modifying the same file simultaneously; therefore, the last modification wins.

[0008] Generally speaking, distributed file systems have two unfortunate characteristics:

[0009] Access to non-local files is slow because the files must be sent to the node accessing them, or the client must be redirected to the node hosting the files.

[0010] • Due to the loose cluster consistency of the cluster servers, access patterns cannot be traced across clusters.

[0011] The current solution has various corresponding drawbacks, such as:

[0012] • Lack of or limited compliance with Portable Operating System Interface (POSIX);

[0013] • Two-step copying – Data is copied only after it has been written to disk. Most copying implementations use remote synchronization (rsync). Rsync has three main drawbacks: a) it only supports the entire file, b) it requires additional network round trips to detect which parts of the file have changed, and c) the file timestamps have a one-second clock resolution, meaning that two machines cannot change different parts of the same file within the same clock second, because it is assumed that the same timestamp indicates the same change;

[0014] Inefficient copying – regardless of which data blocks are modified, the entire file is copied.

[0015] copy;

[0016] • Unable to respond to access on the local node – no response to access patterns

[0017] Adjustment of style;

[0018] In some cases, public protocols such as NFS are unavailable—clients must use special translator libraries to speak the language of the distributed file system; and

[0019] It is only useful in certain scenarios—it is not universal. For it to be useful in a specific scenario, the data and access patterns must be just right, or the benefits must outweigh the drawbacks.

[0020] The methods described in this section are traceable, but not necessarily methods that have been previously conceived or pursued. Therefore, unless otherwise stated, no method described in this section should be assumed to be prior art simply because it is included in this section. Attached Figure Description

[0021] In the attached diagram:

[0022] Figure 1 It is a block diagram depicting an example cluster that replicates data blocks to appropriate subsets of nodes via a communication network or interconnection network.

[0023] Figure 2 This is a flowchart depicting an example implementation of copying data blocks from one node to another.

[0024] Figure 3 It depicts a sequence of example steps that may occur as various scenarios and interactions unfold to create and discard copies of data blocks.

[0025] Figure 4 An example read lifecycle for reader nodes and data blocks is depicted.

[0026] Figure 5An example computer process in which replication and heatmaps can influence each other is described.

[0027] Figure 6 This is a block diagram illustrating a computer system on which embodiments of the present invention can be implemented;

[0028] Figure 7 This is a block diagram illustrating the basic software system that can be used to control the operation of a computing system. Detailed Implementation

[0029] In the following description, numerous specific details are set forth for purposes of explanation in order to provide a thorough understanding of the invention. However, it will be apparent that the invention can be practiced without these specific details. In other instances, well-known structures and devices are shown in block diagram form to avoid unnecessarily obscuring the invention.

[0030] General Overview

[0031] Combining the fully connected cluster described in this paper with local storage offers interesting opportunities, such as clusters whose storage can grow to exceed the storage of any single machine in the cluster without needing to directly share storage between nodes. The cluster described in this paper, when fully interconnected for peer-to-peer data transfers, maintains appropriate file update modifications and prevents conflicting modifications to files that would corrupt or lose data. The cluster described in this paper guarantees file system consistency between nodes without the overhead of full replication.

[0032] This paper presents a cluster file system that provides a logical file system that clients can access without needing to know where their files are stored. The techniques described here combine the storage of cluster servers into a larger logical file system, which clients can access via convenient means such as Network File System (NFS), local access, Server Message Blocks (SMB), or other methods.

[0033] This paper provides a technology for clustered file systems that aggregate local storage on nodes within a cluster to represent a POSIX-compliant storage solution with efficient replication of persistent data objects such as files, portions of files, and / or individual blocks or series of blocks. Replication offers significant benefits such as data locality to reduce network traffic and redundancy to prevent data loss. Redundancy can prevent single points of failure that could otherwise hinder complex systems. For example, the technology in this paper provides high availability (HA) even when actively involved nodes experience disk crashes, software failures, or exhaustion of resources such as disk space, temporary storage, or processors or input / output (I / O) bandwidth. Replication can provide reliability, availability, and serviceability (RAS) to high-capacity or complex systems such as those used for data warehouses, data grids, or transactional clusters (such as elastic clouds) in the same or different data centers.

[0034] In a clustered file system, a client interacts with some or all of multiple nodes. Each node in the cluster maintains local storage, which is logically shared with all nodes in the cluster, even if there is no physical sharing. For example, for replication, the techniques described in this paper do not require cross-mounted file systems such as NFS. Each node implements a cache that can persist a corresponding subset of data blocks in the cluster. Similarly, each data block is replicated to a corresponding subset of nodes in the cluster. The subset of nodes that persist a copy of the same data block is referred to in this paper as the replica nodes for that data block. Cache consistency is maintained among the replica nodes. That is, each replica node of a data block maintains a consistent copy of the data block.

[0035] In one embodiment, a first node in a node cluster requests access to a set of replica nodes that store copies of a series of data blocks. The set of replica nodes is configured to store copies of the data block series within their respective local storage devices. Upon receiving the access request, a replica node in the set of replica nodes may offer a copy of the data blocks to the first node. The first node may receive the copy of the data block series. Once received, the first node may persist the copy of the data block series to a local cache on the first node. In one embodiment, the local cache includes both volatile and non-volatile storage devices, both storing the same data blocks as discussed herein.

[0036] Each node in the node cluster is enabled using cluster management software configured to manage locks on the block series and the set of replica nodes for the block series. Upon receiving a replica of the block series, the cluster management software on the first node can record access activity for the block series in a heatmap of access activity. This heatmap tracks read and write accesses to the block series and the locks allocated. The cluster management software can then determine whether to adjust the number of nodes in the replica node set configured to store replicas of the block series. In response to determining whether to adjust the number of nodes in the replica node set for the block series, the cluster management software can adjust the number of nodes in the replica node set.

[0037] By leveraging a Distributed Lock Manager (DLM) layer and a fully connected cluster, a heatmap of access patterns can be created. If each node requesting access to a file or data block through the DLM layer is assigned a unique identifier (this can be a requirement of the DLM), then the graph of nodes that most frequently access the file or data block can be dynamically updated. Read and modification operations can be tracked individually, allowing the cluster to know where and how files are accessed most frequently.

[0038] These heatmaps ensure that data is replicated to the nodes that use it most frequently. Various heuristics can be used: for N-way replication, identify the top N most in-demand nodes for a specific series of data blocks in the file—and ensure that all of these top nodes contain copies of the data. This serves two purposes—creating data redundancy through replication and keeping data access local for those nodes with the greatest demand for the data. Other algorithms and / or heuristics can be appended to or used in conjunction with heatmaps. Figure 1 This is used to tune cluster performance. For example, artificial intelligence (AI) algorithms such as Bayesian networks, prediction systems, and other optimization techniques can determine replication and storage settings for various files or data blocks. A general goal is to achieve low-latency writes and low-latency reads, which are opposing concerns in other approaches.

[0039] In an embodiment, if the cluster management software determines that the number of read access requests for a data block series exceeds a configured threshold, the cluster management software may increase the number of nodes in the replication node set. For example, an increase in the number of reads may mean that having additional copies of the data block series on additional replication nodes is more efficient, thereby reducing the latency associated with replicating the data block series from one replication node to a node not in the replication node set.

[0040] However, write access requests require the data blocks to be modified to be replicated to each replica node in the replica node set. If the replica node set is large, replicating new writes to each replica node can take a significant amount of time and resources, increasing latency for other access requests to the data block family. If the cluster management software determines that the number of write access requests to the data block family exceeds a configured threshold, it can reduce the number of nodes in the replica node set to reduce the number of writes across the cluster and thus reduce latency.

[0041] Heatmaps can be used to identify access patterns in a series of data blocks at specific times to optimize the size of the replication node set for the purpose of reducing access latency. In embodiments, such as when a replication node crashes and is replaced, heatmaps are used to enforce minimal replication for each data block. In embodiments, such as during and after demand peaks, heatmaps are used to adjust the minimum replication for data blocks.

[0042] In this embodiment, each data block is replicated on a minimum number of corresponding replication nodes in the cluster. A first node requests access to a data block. Based on the access request, the heatmap is modified, and the data block is replicated to the first node. Based on the heatmap, the minimum number of nodes in the corresponding replication nodes for at least one data block is adjusted.

[0043] Cluster File System

[0044] Figure 1 This is a block diagram depicting an example cluster 100 that replicates data blocks 121-124 to corresponding subsets of nodes AC via a communication network or internetwork. These nodes can be the same or different types of network components, such as rack servers (e.g., blade servers), mainframes, storage devices, laptops, smartphones, or virtual machines (VMs). Each of the data blocks 121-124 has a fixed amount of data storage capacity. In this embodiment, data blocks 121-124 are contiguous or non-contiguous disk blocks of the same or different files. In this embodiment, data blocks 121-124 are database blocks and / or not part of a file.

[0045] Data blocks 121-124 can be replicated individually or in various subsets on the same or different corresponding subsets of node AC. For example, data block 121 is replicated on all nodes AC, while data block 124 is replicated only on nodes BC. Different nodes may store the same or different numbers of data blocks. For example, node C has more data blocks than node A.

[0046] The recent access frequency of the corresponding series or individuals of data blocks 121-124 is recorded in heatmap 110, which may reside in one, some, or all of the volatile or non-volatile storage devices of a central server or node AC. In the illustrated embodiment, heatmap 110 has columns for separately counting (e.g., during the current time period) the recent reads and writes of each data block. For example, when any node reads data block 122, the read counter for data block 122 is incremented by 1 in heatmap 110.

[0047] In this embodiment, the heatmap does not have one or more access counters for a single data block. For example, there may be one or more access counters shared by multiple data blocks, such as a series of data blocks, data blocks with conflicting hash codes, or data blocks with a series of hash codes. For example, the hash function may compute the hash code of a data block based on an identifier such as a logical block address (LBA).

[0048] In one embodiment, the read and / or write counters are reset to zero in heatmap 110 between the same or different corresponding time periods. In another embodiment, the read and / or write counters decrease more or less gradually according to the same or different linear or non-linear cooling schedules.

[0049] In embodiments, reading and writing affect the same counter. Embodiments of heatmap 110 may contain more or fewer columns than those shown. In embodiments, some of the shown columns are additionally or alternatively included in one or more software components other than heatmap 110.

[0050] As discussed throughout this document, heatmap 110 contains tracking data and metrics that can be used to automatically optimize the number and location of replicas to protect data and throughput. Which component(s) in cluster 100 requests or performs such automatic replica optimization may vary depending on the embodiment. In most embodiments discussed herein, the sole or primary component performing automatic replica optimization is the node currently accessing the data block, which requires checking heatmap 110.

[0051] In an embodiment, additionally or alternatively, the lock manager performs automatic replication optimization based on heatmap 110. In this embodiment, heatmap 110 is object-oriented to provide data and behaviors such as automatic replication optimization. The advantage of having nodes responsible for automatic replication optimization is that decentralization avoids single points of failure in automatic replication optimization, making cluster 100 more robust. In such an embodiment, or in which the lock manager provides automatic replication optimization, heatmap 110 may be a passive data structure.

[0052] In various embodiments, automatic replication optimization can occur at various times. All embodiments can perform automatic replication optimization at important times, such as when: a) data blocks are locked, accessed, and / or unlocked, or b) when a node crashes or otherwise leaves cluster 100, as discussed later herein. Some embodiments are autonomous and perform automatic replication optimization at additional times (such as even when cluster 100 is otherwise idle) based on various schedules, triggers, and / or conditions discussed later herein.

[0053] As described above, in various embodiments, the active node, lock manager, or heatmap 110 can perform automatic replica optimization based on heatmap 110. Such a component that autonomously performs this action within cluster 100 is referred to herein as an autonomous component. Cluster 100 may have one or more autonomous components of the same or different kinds. For example, in embodiments, some or all nodes and heatmap 110 may be autonomous components, which may have the same or different reasons and logic for autonomously performing automatic replica optimization.

[0054] In this embodiment, during a node's access to a data block, the node records more than an incrementing access count in heatmap 110. Heatmap 110 may also record the identifiers of the nodes making the access and the data blocks. In this embodiment, heatmap 110 contains more or less detailed logs of any node's most recent access to any data block.

[0055] The current column of heatmap 110 indicates a subset of nodes AC that contain the corresponding data blocks 121-124. For example, all nodes AC contain data block 121. If a node has a locally persisted copy of the data block, then that node is a replica node of the data block. As shown in the figure, all nodes AC are replica nodes of data block 121, but node A is not a replica node of data blocks 123-124.

[0056] As discussed later herein, the minimum column of heatmap 110 is used at different times. Cluster 100 ensures that each data block is always replicated on at least the minimum number of replication nodes. In an embodiment not shown, all data blocks 121-124 have the same minimum number of replicas. In the embodiment shown, different data blocks have different minimum numbers. For example, data block 122 has more replicas than required.

[0057] Various embodiments may have data access locks of various granularities (such as a single data block, a series of data blocks, and / or all data blocks in a file). A node should acquire one or more locks when accessing one or more data blocks. In embodiments, all node ACs can request locks from a lock manager. Various embodiments have a lock manager for all locks or a separate lock manager for each lock. In various embodiments, requests to lock or unlock one or more data blocks are submitted to lock managers hosted by a central server or one, some, or all of the node ACs. In embodiments, heatmap 110 and the lock manager are the same component.

[0058] Before accessing a data block, a node should lock the data block, regardless of whether the data block has already been stored locally on the node. A node should always access its local copy of the data block, even if this requires suddenly copying the data block from another node's volatile or non-volatile storage to that node's volatile or non-volatile storage. For example, as shown in the figure, node A copies data block 124 from node B, even though node C could alternatively provide the same data block 124.

[0059] To determine whether node B or C should provide data block 124 to node A, various embodiments may employ various criteria, such as network topology, workload, and / or performance metrics. In various embodiments, node A, the lock manager, or heatmap 110 has logic to determine which replica node should provide the data block.

[0060] The lock semantics are as follows. As shown, there are separate locks for different access types (such as read and write). In this embodiment, the deletion of one or more data blocks, such as file truncation, is an example of a write. A write lock provides exclusive access to a node. As shown in the lock column, "W:A" means that node A has written-locked data block 121. In this embodiment, the lock column is part of the lock manager and not part of heatmap 110.

[0061] Multiple nodes can simultaneously lock the same data block for read. As shown, "R:A,C" means that nodes A and C both lock data block 122 for read. The same data block should not be locked for both write and read simultaneously. As shown, data blocks 123-124 are unlocked. In this embodiment, a node notifies other nodes and / or records within heatmap 110 that it is requesting a read lock or a write lock.

[0062] A node that acquires a write lock should eventually and explicitly release the lock. Read locks can be explicitly or implicitly released or destroyed as follows. A write lock can be acquired even for data blocks that have already been read and locked by other nodes. When a write lock on a data block is acquired, several reactions may occur, such as the following.

[0063] In this embodiment, all nodes that have acquired read locks on the data block are notified that their read locks are now automatically broken. In this embodiment, nodes that need to recover read access to the data block should reacquire the read lock, which is not granted until the write lock is explicitly released, at which point the new version of the data block can be read. Therefore, different versions of the data block are observed before and after the read lock is reacquired. In this embodiment, and in some cases, reader nodes can decide to continue using content from an aging local copy without a lock, such as after a read lock has been broken. For example, the database session isolation level can be configured for repeatable reads.

[0064] In one embodiment, a node that has acquired a read lock can upgrade to a write lock, which occurs in the same way as acquiring a write lock without already having a read lock, and this implicitly breaks the read lock. In another embodiment, a node that has acquired a write lock can downgrade to a read lock, which occurs in the same way as unlocking the write lock and then acquiring a read lock.

[0065] Fairness in lock granting prevents starvation of nodes that need data blocks. In embodiments, this is achieved, for example, by queuing requests to grant locks in the same chronological order as requests for the same data block. Requests to release locks are never queued. In embodiments, lock degradation occurs without queuing.

[0066] Minimal Replication

[0067] The techniques described in this article combine two types of replication. As discussed later in this article, on-demand replication requires creating additional copies of data blocks on nodes that need them but lack them. If the same data block is eventually needed on many or all nodes in the cluster (whether simultaneously or not), the implementation can eventually replicate that data block to many or all nodes in the cluster. In other words, many or all nodes can simultaneously be replicating nodes for the same data block, which ensures full replication of the data block.

[0068] Another type of replication is minimal replication, which ensures minimal redundancy regardless of demand. For example, a data block may not be needed by any of the nodes at present, but it should still be persisted on a small number of nodes to avoid data loss if any node needs the data block in the future. Due to demand replication, rather than minimal replication, data blocks with high demand on many nodes will typically have sufficient replicas in the cluster.

[0069] Technical problems can arise when only one node needs and writes a data block. Local writing of a data block inevitably leads to the remote replica becoming stale. Since other nodes have no current need for that data block, demand replication does not occur to propagate the corrected data block. This problem of lack of replication is addressed by minimal replication, but this raises other issues, including deciding which nodes should be replicators when: a) in a low-demand scenario, no or only one node needs the data block, or b) in the opposite high-demand scenario, there are too many stale replicas of the data block to replace all of them with the corrected data block. The technique presented in this paper addresses these problems of replicator node selection for minimal replication by integrating minimal replication into various triggers, such as the release of write locks as described below.

[0070] When the write lock on a data block is released, additional activities occur, including the enforcement of minimum replication. As explained above: a) if a node lacks a local copy, the node that acquired the read or write lock also receives and persists the copy; and b) more than the minimum number of data block copies can be created. When the write lock on a data block is released, the minimum number of replica nodes, including the node that acquired the write lock, are selected as replica nodes. In various embodiments, heatmap 110 or the node that acquired and released the write lock selects the remaining replica nodes. In embodiments, the same synchronous call path (i.e., control flow) includes: write locking, modifying the data block, unlocking, local persistence, notifying heatmap 110, and / or replicating the correction to the remaining replica nodes.

[0071] The criteria for selecting remaining replication nodes may or may not include: network topology with hierarchical networks, workload, and / or performance metrics such as unused local disk space. In embodiments and as discussed later herein, the Bayesian network computes and compares various probabilities to select remaining replication nodes by predicting which replication nodes can maximize throughput and / or optimize other performance metrics.

[0072] The reaction of the remaining replication nodes will be discussed later in this article. As explained above, a data block may have more replicas than needed. Additional replication nodes not selected as remaining replication nodes will no longer be replication nodes for the data block when the write lock is released. Even without write locks involved, the autonomous components of cluster 100, as discussed earlier in this article, can autonomously decide to increase or decrease the minimum and / or current replication of data blocks, as discussed later in this article.

[0073] In the embodiments discussed later in this document, writing a data block can invalidate copies of the data block at other nodes, even before the write lock is released. The dirty column in heatmap 110 indicates such invalidation. As shown, node C has written to node 123 and marked it as dirty.

[0074] However, invalidation may or may not require breaking read locks. As discussed later in this document, various embodiments may invalidate data blocks eagerly or lazily. For example, as shown in the figure, node B has not yet noticed that its replica has been invalidated by a write from node C and / or has not yet reacted to the invalidation of its replica by a write from node C.

[0075] Cluster 100's consistency model can provide read-after-write semantics, ensuring that stale data is never read. For example, even if replicas are distributed and not necessarily centrally managed, Cluster 100 can provide POSIX-compliant I / O, such as read-after-write. As discussed later in this paper, writer nodes can modify data blocks in volatile memory, for example, using I / O buffers with delayed refresh, which can include persistence and replication. Through the locking behavior discussed in this paper, and by identifying specific data blocks or families of data blocks for access, multiple reader nodes and multiple writer nodes can simultaneously operate on different corresponding portions of a POSIX-compliant file, even with delayed persistence and replication, without compromising POSIX consistency.

[0076] Horizontally scaled parallel reads may occur. For example, node A may: a) copy data blocks 123-124 from node B, b) alternatively copy the same blocks 123-124 from node C, or c) simultaneously copy data block 123 from node B and data block 124 from node C. In embodiments, which copying node provides the data block depends on which node requests the data block, such as using a hierarchical network discussed earlier herein.

[0077] Copy method

[0078] Figure 2 This is a flowchart depicting an example implementation 200 of copying data blocks from one node to another. Figure 2 This paper introduces a copying mode that may be applicable to the graphs presented later in this paper. Figure 2 The components in can be with Figure 1 The implementation of components that are more or less similar. Figure 2 The emphasis is placed on transmission interactions without context. The occasions and scenarios for copying will be discussed later in this document. Various embodiments may employ various transmission mechanisms, as discussed later in this document.

[0079] Replication method 200 includes various mutually exclusive methods beginning with corresponding steps 202A-202D, some of which share subsequent steps 204 and 206, as shown in the figure. For illustrative purposes, Figure 2The system needs to include the following: a) data blocks and heatmaps, b) sender replicator nodes that already have valid local copies of the data blocks, c) receiver replicator nodes that are becoming replicator nodes or are refreshing their stale copies of the data blocks, and d) autonomous components of the cluster as explained earlier in this document.

[0080] Figure 2 The various steps are performed by the various components shown below. As explained earlier in this document, replication is primarily driven by the active node, and the heatmap can be a passive data structure in most embodiments. Also, as explained earlier in this document, replication may sometimes require initiation and / or participation from autonomous components of the cluster. As discussed below, steps 202C-202D and 204 require active participation from autonomous components. As discussed below, step 202D actually requires autonomous activity from an autonomous component.

[0081] In step 202A, the sending replica node directly pushes the data block to the receiving replica node. For example, the sending replica node may perform a Remote Direct Memory Access (RDMA) write to copy the data block from the sending replica node's volatile memory to the receiving replica node's volatile memory. RDMA is discussed later herein. In various embodiments and although not shown, the copy may be incomplete until: a) the heatmap is updated via transfer, and / or b) the receiving replica node locally persists the data block.

[0082] In step 202B, the sending replica node directly notifies the receiving replica node of replication opportunities, such as events that may conditionally or unconditionally require replication. If the receiving replica node decides to respond, step 206 occurs. In step 206, the receiving replica node pulls data blocks directly from the sending replica node, such as via RDMA read. RDMA operation is presented in the relevant non-patent document (NPL) "Designing NFS With RDMA for Security, Performance and Scalability".

[0083] In step 202C, the sending replica node explicitly or implicitly notifies the autonomous component of a replication opportunity, such as an event that may conditionally or unconditionally require replication. Implicit notification may occur if the autonomous component subscribes to, monitors, observes, or otherwise receives or detects indications of activity by the sending replica node (such as locking, modifying, persisting, or unlocking data blocks). For example, in various embodiments, the autonomous component may observe some or all of the initiation and / or completion and / or the transitions shown between the steps. The autonomous component responds conditionally or unconditionally by performing step 204.

[0084] In step 204, the autonomous component notifies the receiving replication node of replication opportunities, such as events indicating that replication may be conditionally or unconditionally required. If the receiving replication node decides to respond, step 206 occurs as described above.

[0085] In step 202D, the autonomous component autonomously activates itself, such as when: a) a periodic expiration of an interval timer used for cooling or resetting the access count, b) a certain performance metric threshold for a node, data block, or the entire system, and / or c) as guided by probabilistic predictions based on a Bayesian network, as discussed later herein. The autonomous component can autonomously determine whether replication is necessary or desired, in which case steps 204 and 206 can occur as described above.

[0086] Creating and discarding copies

[0087] Figure 3 Example sequence of steps 301-310 is described as possible as various scenarios and interactions progress, resulting in the creation and discarding of copies of data blocks. Figure 3 It is a macroscopic view of the system's behavior. As will be presented later in this article, Figure 4 It is a micro view of the behavior of nodes (such as the reader node below) that can receive and / or read data blocks for various reasons. Figures 3-4 The same embodiments may or may not be depicted.

[0088] Steps 301-310 occur in the order shown. However, under certain conditions or embodiments, sub-steps of a given step may be reordered or parallelized within the step. For example, sub-steps AB of step 309 may occur simultaneously, as discussed later herein.

[0089] Figure 3 The various components in can be Figure 1 The implementation of similar components, to varying degrees. For illustrative purposes, Figure 3 The system needs to include the following: a) data blocks and heatmaps, b) original replication nodes that already have valid local copies of the data blocks, c) reader nodes and writer nodes that need but lack data blocks, and d) autonomous components of the cluster as explained earlier in this paper.

[0090] Figure 3 The various steps are performed by the various components shown in the actor list. Sub-steps of the same step are performed by the same actor. In various embodiments, the autonomous component may observe the initiation and / or completion of some or all of the shown steps and / or sub-steps, even if the autonomous component is currently inactive and does not otherwise directly participate in the steps.

[0091] Initially in step 301, the original replica node locally persists the data block as shown in the Actions column. As shown in the Replica Nodes column, the heatmap is used to enforce the minimum copy of the data block residing on the original replica node. In other words, the original replica node has copies of the data block on volatile memory and local disk, and no other node contains the data block.

[0092] In step 302, the reader node autonomously copies and reads the data block. Sub-steps of step 302 (not shown) may include: a) reading the locked data block, b) such as via... Figure 2 The process may involve: c) replicating the data block from the original replication node; c) storing the data block in volatile and non-volatile storage; and / or d) using the contents of the data block, such as during database query execution by a remote client of the reader node, for example, for online analytical processing (OLAP). In any case, step 302 makes the reader node an additional replication node, as shown in the replication node column, which can be tracked in the heatmap.

[0093] Steps 303-304 relate to the cluster membership of nodes. In an embodiment, the cluster has a static list of nodes that fluctuates only during failures or maintenance. In an embodiment, additional nodes may be added to: a) increase cluster storage capacity by providing more unused disk space, b) increase throughput by storing fewer replicas on each of more nodes, and / or c) increase reliability by increasing the minimum number of replicas.

[0094] In this embodiment, the cluster has federations of nodes that can join and leave the cluster individually and autonomously. Example federations include: a) elastic horizontal scaling, such as in a computer cloud or data grid, or a loose federation of personal computers, workstations, and / or mobile devices, such as laptops and smartphones.

[0095] In step 303, the original replica node leaves the cluster detected by the autonomous component. For example, the original replica node may leave explicitly or implicitly due to timeout or lost heartbeat (such as when a disk drive crashes). If the leaving node is the only replica node of the data block, this could cause catastrophic data loss, which is why more practical examples would never set the minimum replication factor below 2.

[0096] The reaction of an autonomous component to a departing node may vary as follows. If the cluster no longer has a minimum replica of a data block, a node that is not a replicating node is selected as another replicating node to persist the replacement of the lost replica. When the departing node results in insufficient replicas for multiple data blocks, then one or more new replicating nodes may be needed. The heuristics for selecting the following are discussed later in this paper: a) how many additional replicating nodes to recruit, b) which of those additional replicating nodes should receive which data blocks, and c) from which surviving replicating nodes the new replicas should be copied. In one example, a new replicating node receives all replacement replicas.

[0097] As shown in the replica node column of step 303, the reader node is the only surviving replica node for the data block. Because the smallest replica node is still available, the autonomous component does not need to recruit additional replica nodes.

[0098] In step 304, the original replica node rejoins the cluster. In various embodiments or scenarios, after rejoining, some or all of the original replica node's local replica list may be lost, outdated, or ignored by default. In some embodiments, the cluster does not accept replicas from the rejoined node. In the illustrated embodiment, the autonomous component or the rejoined node detects that some replicas on the rejoined node are still valid, such as when data blocks were not modified in the cluster between leaving and rejoining. In this case, and as shown in the replica node column of step 304, the cluster again has more data block replicas than needed.

[0099] In step 305, the autonomous component autonomously decides to increase the minimum number of replicas of the data block to two, as shown in the replica node column. Since the data block already has two replicas, the autonomous component does not need to recruit another replica node. Otherwise, recruitment would occur as discussed above.

[0100] Steps 306-309 involve a writer node, such as one used for database transactions like Online Transaction Processing (OLTP). In step 306, the writer node writes a lock on the data block. In substep 306A, and because the writer node is missing and the data block is needed, replication of the data block from any other replication node to the writer node, such as via... Figure 2 This occurs in a manner presented in the document. The heuristic for selecting which replicating node to replicate from is presented later in this document and can be determined by the writer node. As shown in the replicating node column of step 306, the writer node becomes another replicating node.

[0101] In step 306B, writing to the locked data block causes all read locks on that block to be destroyed, to which other nodes can react in various ways and embodiments, as discussed later herein. However, the heatmap still tracks which other nodes are replica nodes and whose replicas have not yet failed. For example, data is not lost simply because one node writes to the locked data block and then immediately crashes.

[0102] In step 307, the writer node modifies the data block. Specifically, in sub-step 307A, the writer node modifies the data block in volatile memory. Buffering will be discussed later in this document.

[0103] In substep 307B, the writer node marks a data block as dirty, meaning it has been modified but not locally persisted and / or not replicated. In this embodiment, a heatmap records which data blocks are dirty. As discussed later herein, marking data blocks as dirty can ultimately lead to lazy failures, such as when a replication node eventually and autonomously checks to see if a data block has been marked as dirty.

[0104] In step 308, while a data block is being written-locked, any other node may attempt to acquire a read lock or write lock on the data block. The attempt is suspended until the current write lock is released, and the attempting node waits for the requested lock to be granted.

[0105] In step 309, the writer node commits and refreshes the corrected data block by unlocking it. In substep 309A, the writer node selects the minimum number of replica nodes, including the writer node, to remain as replica nodes. The corrected copy of the writer node is replicated to the remaining replica nodes, which in this case are the writer node and the original replica node, as shown in the replica node column of substep 309C. An embodiment may select the most recent writer node, with no other recent writer node and the most recent reader node as the remaining replica nodes.

[0106] Such replication can be achieved through Figure 2 This occurs in a certain way, as shown in the diagram. A replicating node that has exceeded the minimum replication threshold is no longer a replicating node for the data block, such as a reader node as illustrated. Sub-step 309B is also triggered by unlocking, where the writer node locally persists the corrected data block. Unlocking is completed in sub-step 309C, and the same or different nodes can immediately or eventually lock the data block for different uses.

[0107] Step 310 requires autonomous behavior from the autonomous component. In sub-step 310A, the autonomous component autonomously reduces the minimum replication of the data block, such as after the demand peak for the data block has subsided and / or during demand peaks for different data blocks. In sub-step 310B, the autonomous component autonomously cancels redundant replication nodes for the data block. However, cancelling redundant replication nodes can be optional and may not occur in some cases.

[0108] Because the writer node was the most recently write-locked data block, it remains a replica node. In this case, the original replica node is no longer a replica node of the data block. In another example, even when the minimum replication factor remains constant, the data block is never write-locked, and cluster membership remains unchanged, the autonomous component can autonomously cancel redundant replica nodes. For example, dynamic fluctuations in demand for data or local or system performance metrics may trigger autonomous intervention from the autonomous component, which can create or discard copies of the same or different data blocks.

[0109] Read lifecycle

[0110] Figure 4 An example read lifecycle 400 of a reader node and a data block is depicted. In various embodiments, the initiation and / or completion of some or all of the illustrated steps and / or some or all of the transitions between the illustrated steps can be observed and / or recorded in a heatmap by the autonomous component. The illustrated steps are performed by the reader node during various scenarios involving the same data block in various ways.

[0111] Transitions are shown as solid arrows when adjacent steps occur more or less in a rapid sequence or simultaneously. Transitions are shown as dashed arrows if an external or asynchronous stimulus is required to trigger a transition after an infinitely long duration.

[0112] In a scenario where steps 401-412 are executed in this order, the reader node is missing and needs to read a data block. In step 401, the reader node decides to read the data block, such as when a remote client of the reader node uses the reader node to retrieve data during query execution. As discussed earlier herein, step 402 requests a read lock for the data block.

[0113] The part requesting a read lock may require additional activities, such as steps 403-404 and 415, which may occur before the read lock is granted. Steps 403 and 415 are shown as a decision diamond because the behavior involving requesting a read lock may be conditioned on the locality and / or dirtyness of the data block. Step 403 checks whether the data block already resides on the reader node.

[0114] In other words, step 403 checks whether the reader node is already a replica node of the data block, in which case the read can be satisfied locally. If the data block is not locally available, step 404 potentially waits for the data block to be unlocked by the writer node. If the data block has already been unlocked, there is no need to wait, and step 404 completes directly.

[0115] If step 403 alternatively detects that the data block is already locally available, then step 415 checks whether the data block has been marked as dirty by the writer node. For example, when requesting a read lock, the reader node may check the global directory of currently marked dirty data blocks and / or families of data blocks.

[0116] If the writer node has unlocked the data block and persisted and replicated the correction to the data block, then the data block is not write-locked and is not listed as dirty. Therefore, a read lock for a local data block that is not dirty or not write-locked will be granted immediately in step 405. Thus, when the data block is not dirty, step 415 immediately follows step 405.

[0117] If a data block is locally available but remotely corrupted by a writer node, the writer node has not yet released the write lock. In this case, step 415 is followed by step 404 to await the release of the write lock, as explained above. In an embodiment, the transition from step 415 to 404 includes the reader node notifying and causing the writer node to flush its buffered and corrupted data blocks, including replication and local persistence performed by the writer node. However, such a remote flush does not necessarily involve releasing the write lock.

[0118] As shown in the figure, waiting for the write lock to be released in step 404 may occur regardless of whether the data block is local. Therefore, the wait in step 404 can occur after steps 403 or 415. The write lock is eventually released, and a read lock is granted in step 405. After acquiring the read lock, the subsequent behavior of the reader node depends on step 406, which checks whether the data block is already locally available.

[0119] The detection results of steps 403 and 406 should be the same. If the data block is locally available, direct use of the data block can occur in step 411. For example, the reader node can inspect and analyze the contents of the data block. Otherwise, the data block is only remotely available and should be immediately replicated to the reader node, which requires steps 407-410 as follows.

[0120] Step 407 uses RDMA for reading, as described earlier in this article. Figure 2The discussion concerns receiving a copy of a data block from another node into the volatile memory of the reader node. Some embodiments may or may not include step 408, which involves locally storing the data block at a specific location in the volatile memory. For example, eventually, another node can use RDMA to read and retrieve the data block from that specific address in the volatile memory.

[0121] In one embodiment, step 407 uses a buffer such as an operating system (OS) input / output (I / O) buffer, which is maintained at a specific address in memory. In another embodiment, step 407 uses a buffer in a buffer cache for database blocks, such as one managed by a database management system (DBMS). RDMA buffering and addressing are presented in the relevant non-patent document (NPL) "Designing NFS With RDMA for Security, Performance and Scalability".

[0122] The volatile storage in step 407 may not be sufficient for copying this document, and the additional step 409 of locally persisting the received data blocks may also be necessary. For example, if the data blocks are part of the original file, depending on the embodiment and scenario, the data blocks may be: a) locally persisted to the corresponding block location within a local copy of the entire or partial copy of the original file, b) locally persisted as a single-block file, or d) locally persisted to another file containing unordered data blocks cataloged for random access. For example, various data blocks may initially be persisted regardless of their order within the original file and later reassembled into consecutive portions or the entirety of the original file.

[0123] In step 410, the reader node explicitly records the replication in the heatmap after persisting the data block locally. By persisting the local copy in step 410, the reader node has become the replication node for that data block, and in step 411, the reader node can directly use the contents of the local copy. Therefore, step 411 can occur whether replication is needed or the data block is already locally available. The prerequisite for step 411 can be that the data block resides in volatile or non-volatile local storage.

[0124] After step 411, the reader node can complete reading and using the data block, but the reader node remains a copy node of the data block in step 412. In various embodiments, the reader node may or may not release its read lock on the data block when it has finished using it. Because the reader node remains a copy node of the data block, step 413 can receive read requests from another node, which also becomes a copy node when step 414, such as sending a copy of the data block to and from volatile memory via RDMA, is performed, after which the ready state of step 412 is accessed again. Therefore, the reader node can retain and provide a copy of the data block while remaining in a ready state for more or less indefinitely.

[0125] When in the ready state of step 412, and although not shown, the autonomous component or writer node can cancel the reader node as a copy node for data blocks. In this case, the reader node can discard data blocks from both volatile and non-volatile memory.

[0126] When in the ready state of step 412, the reader node may detect or be notified that the local copy of the data block is stale because the writer node has marked the data block as dirty or committed a correction to the data block. In this case, and in various embodiments, invalid step 417 occurs, which may discard the data block from both volatile and non-volatile memory, and in some cases, causes the data block to be copied to the reader node again.

[0127] Even when in the ready state of step 412, any attempt by the reader node to use the data block later should be restarted in step 401, including relocking. When steps 401-403, 415, 405-406, and 411 are repeated in this order, a local copy of the data block can be reused locally without copying. Otherwise, copying needs to be repeated as described above.

[0128] As discussed earlier in this document, an autonomous component can proactively and autonomously induce replication that has not been requested by a node. For example, a replicating node may crash, or the autonomous component may autonomously increase the minimum number of replicas of a data block. In such cases, the autonomous component can autonomously recruit nodes to become replicating nodes. For example, if a reader node is not yet a replicating node for a data block, the autonomous component can make the reader node a replicating node in step 416, where replication occurs as described above.

[0129] Example autonomous and / or heatmaps used for replication

[0130] Figure 5An example computer process in which replication and heatmaps can influence each other is depicted. The system involved may include data blocks, heatmaps, autonomous components, and nodes, including requesting nodes.

[0131] Although step 501 is preparatory and requires persisting copies of the data blocks across various nodes, step 501 can explicitly occur as the initial distribution and storage of persistent data or through earlier replication operations as discussed earlier herein. For example, various files may contain various data blocks, and full or partial copies of the files may be distributed and stored on local disks across various nodes.

[0132] In step 502, the requesting node requests read or write access to various data blocks (such as a series of data blocks). As discussed earlier, step 502 may require acquiring read or write locks on the data blocks in question.

[0133] Based on the requested access, the heatmap is modified in step 503. For example, a read counter or write counter may be associated with each accessed data block or series of data blocks, respectively. The access is notified to the heatmap according to techniques proposed earlier herein. For example, requesting or granting a lock may result in an incrementing heatmap.

[0134] Based on the requested access, step 504 copies some or all of the accessed data blocks, as well as any other data blocks that may be accessed. For example, sequential access to many consecutive data blocks of the same file may: a) reuse local copies of some data blocks, b) result in the copying of other data blocks required for the same access request, and / or c) proactively (i.e., eagerly and autonomously) copy unrequested data blocks of the same file that may soon be needed. For example, step 504 may read ahead to prefetch some data blocks, such as for table scans.

[0135] Based on the heatmap, in step 505, the minimum number of replica nodes is adjusted for at least one data block, data block series, or file. As discussed earlier herein, increasing or decreasing the minimum number of replicas can occur in response to autonomous and proactive decisions by the autonomous component. The autonomous component can increase and then decrease the minimum number of data blocks at the beginning and end of demand peaks for those data blocks, respectively. As described below, the autonomous component can delegate some decisions to a Bayesian network that predicts optimal configuration and adjustments based on the probability of occurrence of corresponding identifiable patterns, such as those learned from historical accesses to data blocks and nodes (e.g., observed and / or recorded in the heatmap).

[0136] Copy mode

[0137] Various implementations of the logic used for automatic replication optimization can impose various replication modes on the entire cluster or corresponding subsets of nodes, data blocks, and / or files, such as the following replication modes:

[0138] A. All N-way – This type of replication copies data across N nodes, waits for all writes to complete, and then returns to the initiating client. This is the slowest modification type.

[0139] B. Failover Groups – Similar to N-way, but nodes are organized into node groups, and data and corrections must be replicated to at least one node in each group. This helps define data redundancy areas according to user needs (potentially across racks or other characteristics).

[0140] C. Single – without replication.

[0141] D. Asynchronous – Data and corrections are replicated across nodes without waiting for confirmation requests from other nodes. This is the fastest replication method, but it does not provide a perfect guarantee that data is actually replicated.

[0142] E. Local Mirroring + N Remote Nodes – Local disks or file systems can be configured as local copies. This provides redundancy against disk failures rather than node failures. Secondary remote nodes (or N remote nodes) can be used to protect against node failures, ensuring data survival even if the primary node is lost.

[0143] F. Local Mirroring + N Remote Asynchronous Replication—Local disks or file systems can be configured as local replicas—this provides redundancy for disk but not node failures. A second remote node (or N remote nodes) can be used to protect against node failures, ensuring data persistence in the event of primary node loss. Remote replicas are asynchronous to provide fast local redundancy.

[0144] G. Machine Learning and / or Continuous Tuning – By using tracking mechanisms, file system access patterns can be learned and identified. Heatmaps can be consulted to distribute data across the cluster in a way that is suitable or unsuitable for other strict replication modes. Combined with at least one other replication mode, data can be placed in areas where it is frequently accessed, thus ensuring lower latency for future accesses.

[0145] Therefore, autonomous components can apply replication modes and switch between them. After a replication mode is applied, the same or different autonomous components can continuously adapt and optimize the replication configuration.

[0146] Network transmission

[0147] Various embodiments may use various network transports to facilitate replication, such as RDMA. In embodiments, transmissions with or without RDMA may require the use of connectionless or unacknowledged network transports (such as User Datagram Protocol (UDP)) for increased throughput. For example, unlimited bandwidth has various transport modes that support and / or accelerate UDP transmissions. UDP also invites application-specific packet reordering and retransmission, such as when lost packets may conditionally need to be retransmitted.

[0148] Although the storage cluster of nodes is logically fully connected at the transport layer, which is one layer in the layer stack of the Open Systems Interconnection (OSI) network communication model, the network topology of the cluster can introduce asymmetries at the network layer below the transport layer. These asymmetries include different communication link bandwidths (such as for interconnection networks or for store-and-forward multi-hop distances) and varying link utilization that may lead to bottlenecks. The communication architecture may include a hierarchy of network switches. Factors and concerns similar to those in network routing can be included in the decisions of the automatic replica optimization logic. Such decisions can integrate static metrics and facts (such as those from infinite bandwidth topologies) as well as dynamic metrics and facts (such as relative link and / or node utilization and node membership in loosely federated clusters).

[0149] Transmissions can be encrypted, for example, over untrusted interconnected networks. A network session begins when a node joins the cluster and ends when a node leaves the cluster, regardless of whether the transmission is connectionless or connectionless. A network session may require selecting specific data blocks for transmission, such as database blocks.

[0150] For example, only non-contiguous blocks of data may be sent for random access. This could be done when: a) a table scan, or b) when sending all or part of a file, such as when a file system is used for persistence, a sequence of more data blocks than requested may be sent.

[0151] Various embodiments can utilize a variety of communication link technologies optimized for low-latency transmission, such as Fibre Channel (FC), Internet Small Computer System Interface (iSCSI), and Fibre Channel over Ethernet (FCoE). Various embodiments can utilize various RDMA protocols to avoid mechanical drive latency such as that from disk rotation or track switching. For example, RDMA can deliver a copy to the volatile memory of the reader node, where the following may occur simultaneously: a) the reader node uses the volatile content for application-specific purposes, and b) the content is persisted locally. RDMA protocols include RDMA over Converged Ethernet (RoCE) and iWARP.

[0152] Bayesian networks

[0153] To some extent, a node cluster can function as a distributed cache, where latency, throughput, energy consumption, and disk drive lifetime may depend on the data block distribution pattern. For example, OLTP latency may depend on the temporal and spatial locality of replicas. Sequential access to numerous data blocks, such as during table scans performed by OLAP, can disrupt the placement of required replicas. Post-write selection of replica nodes can jeopardize existing replica placement.

[0154] Automatic replica optimization logic can include or be guided by a Bayesian network that predicts various optimal replica distributions based on past visits and learned probabilities of future visits. Various Bayesian networks can compute probabilities of corresponding events, such as the following possibilities:

[0155] The two data blocks will be accessed together.

[0156] A data block will be accessed because another data block has already been accessed.

[0157] A specific node will access a specific data block, or

[0158] The lock will be released during the duration.

[0159] For example, a Bayesian network might learn that some nodes or applications hold locks for longer than others, or have a more or less stable working set of data blocks.

[0160] Hardware Overview

[0161] According to one embodiment, the techniques described herein are implemented by one or more dedicated computing devices. The dedicated computing device may be hardwired to execute these techniques, or may include digital electronic devices such as one or more application-specific integrated circuits (ASICs) or field-programmable gate arrays (FPGAs) persistently programmed to execute these techniques, or may include one or more general-purpose hardware processors programmed to execute these techniques according to program instructions in firmware, memory, other storage devices, or a combination thereof. Such a dedicated computing device may also combine custom hardwired logic, ASICs, or FPGAs with custom programming to implement these techniques. The dedicated computing device may be a desktop computer system, a portable computer system, a handheld device, a networking device, or any other device that combines hardwired and / or program logic to implement these techniques.

[0162] For example, Figure 6 This is a block diagram illustrating a computer system 600 on which embodiments of the present invention can be implemented. The computer system 600 includes a bus 602 or other communication mechanism for transmitting information, and a hardware processor 604 coupled to the bus 602 to process information. The hardware processor 604 may be, for example, a general-purpose microprocessor.

[0163] Computer system 600 also includes main memory 606, such as random access memory (RAM) or other dynamic storage device, coupled to bus 602, for storing information and instructions to be executed by processor 604. Main memory 606 can also be used to store temporary variables or other intermediate information during the execution of instructions to be executed by processor 604. When stored in a non-transitory storage medium accessible to processor 604, these instructions make computer system 600 a dedicated machine customized to perform the operations specified in the instructions.

[0164] The computer system 600 also includes a read-only memory (ROM) 608 or other static storage device coupled to the bus 602 for storing static information and instructions for the processor 604. A storage device 610 (such as a disk, optical disk, or solid-state drive) is provided and coupled to the bus 602 for storing information and instructions.

[0165] Computer system 600 can be coupled to display 612 (such as a cathode ray tube (CRT)) via bus 602 for displaying information to the computer user. Input device 614, including alphanumeric keys and other keys, is coupled to bus 602 for transmitting information and command selections to processor 604. Another type of user input device is cursor control 616 (such as a mouse, trackball, or arrow keys) for transmitting directional information and command selections to processor 604 and for controlling cursor movement on display 612. Such input devices typically have two degrees of freedom on two axes (a first axis (e.g., x) and a second axis (e.g., y)), allowing the device to specify a position in a plane.

[0166] Computer system 600 may implement the techniques described herein using custom hardwired logic, one or more ASICs or FPGAs, firmware and / or program logic (which, in conjunction with the computer system, enable or program the computer system 600 as a special-purpose machine). According to one embodiment, computer system 600 performs the techniques described herein in response to processor 604 executing one or more sequences of one or more instructions contained in main memory 606. These instructions may be read into main memory 606 from another storage medium (such as storage device 610). Execution of the sequence of instructions contained in main memory 606 causes processor 604 to perform the processing steps described herein. In alternative embodiments, hardwired circuitry may be used instead of or in combination with software instructions.

[0167] As used herein, the term "storage medium" refers to any non-transitory medium that stores data and / or instructions that enable a machine to operate in a particular manner. Such storage media can include non-volatile media and / or volatile media. Non-volatile media include, for example, optical discs, magnetic disks, or solid-state drives, such as storage device 610. Volatile media include dynamic memory, such as main memory 606. Common forms of storage media include, for example, floppy disks, flexible disks, hard disks, solid-state drives, magnetic tape or any other magnetic data storage media, CD-ROMs, any other optical data storage media, any physical media with a perforated pattern, RAM, PROMs and EPROMs, FLASH-EPROMs, NVRAMs, any other memory chips, or magnetic tape cassettes.

[0168] Storage media differ from transmission media but can be used in conjunction with them. Transmission media participate in the transfer of information between storage media. For example, transmission media include coaxial cables, copper wires, and optical fibers, including conductors containing bus 602. Transmission media can also take the form of sound waves or light waves, such as those generated during radio wave and infrared data communication.

[0169] Carrying one or more sequences of instructions to processor 604 for execution can involve various forms of media. For example, the instructions may initially be carried on a disk or solid-state drive of a remote computer. The remote computer may load the instructions into its dynamic memory and transmit them over a telephone line using a modem. A modem local to computer system 600 may receive data over a telephone line and convert the data into an infrared signal using an infrared transmitter. An infrared detector may receive the data carried in the infrared signal, and appropriate circuitry may place the data on bus 602. Bus 602 carries the data to main memory 606, from which processor 604 retrieves and executes the instructions. The instructions received by main memory 606 may optionally be stored on storage device 610 before or after execution by processor 604.

[0170] Computer system 600 also includes a communication interface 618 coupled to bus 602. Communication interface 618 provides bidirectional data communication coupled to network link 620, which is connected to local network 622. For example, communication interface 618 may be an Integrated Services Digital Network (ISDN) card, a cable modem, a satellite modem, or a modem providing data communication connectivity with a corresponding type of telephone line. As another example, communication interface 618 may be a Local Area Network (LAN) card to provide data communication connectivity with a compatible LAN. Wireless links may also be implemented. In any such implementation, communication interface 618 transmits and receives electrical, electromagnetic, or optical signals carrying streams of digital data representing various types of information.

[0171] Network link 620 typically provides data communication to other data devices via one or more networks. For example, network link 620 can provide a connection via local network 622 to host computer 624 or to data devices operated by Internet Service Provider (ISP) 626. ISP 626 then provides data communication services via a global packet data communication network (now commonly referred to as the "Internet" 628). Both local network 622 and Internet 628 use electrical, electromagnetic, or optical signals carrying digital data streams. Signals through various networks, as well as signals on network link 620 and through communication interface 618 (which carries digital data to and from computer system 600), are example forms of transmission media.

[0172] Computer system 600 can send messages and receive data, including program code, through one or more networks, network links 620, and communication interfaces 618. In the Internet example, server 630 can send requested code to the application through the Internet 628, ISP 626, local network 622, and communication interface 618.

[0173] The received code can be executed by processor 604 when it is received, and / or stored in storage device 610 or other non-volatile storage device for later execution.

[0174] Software Overview

[0175] Figure 7 This is a block diagram of a basic software system 700 that can be used to control the operation of computing system 600. Software system 700 and its components, including their connections, relationships, and functions, are intended to be exemplary only and are not intended to limit the implementation of one or more example embodiments. Other software systems suitable for implementing one or more example embodiments may have different components, including components with different connections, relationships, and functions.

[0176] Software system 700 is provided to guide the operation of computing system 600. Software system 700, which may be stored on system memory (RAM) 606 and fixed storage devices (e.g., hard disk or flash memory) 610, includes a kernel or operating system (OS) 710.

[0177] OS 710 manages the low-level aspects of computer operations, including managing process execution, memory allocation, file input and output (I / O), and device I / O. One or more applications, designated 702A, 702B, 702C...702N, can be "loaded" (e.g., transferred from fixed storage device 610 to memory 606) for execution by system 700. Applications or other software intended for use on computer system 600 can also be stored as downloadable computer-executable instruction sets, for example, for downloading and installing from internet locations (e.g., web servers, app stores, or other online services).

[0178] Software system 700 includes a graphical user interface (GUI) 715 for receiving user commands and data graphically (e.g., "click" or "touch gestures"). Furthermore, system 700 can act upon these inputs according to instructions from operating system 710 and / or (one or more) applications 702. GUI 715 is also used to display the results of operations from OS 710 and (one or more) applications 702, to which the user can provide additional input or terminate the session (e.g., log off).

[0179] OS 710 can execute directly on the bare hardware 720 of computer system 600 (e.g., one or more processors 604). Alternatively, a hypervisor or virtual machine monitor (VMM) 730 can be inserted between the bare hardware 720 and OS 710. In this configuration, VMM 730 acts as a software “buffer” or virtualization layer between OS 710 and the bare hardware 720 of computer system 600.

[0180] VMM 730 instantiates and runs one or more virtual machine instances (“guest machines”). Each guest machine includes a “guest” operating system (such as OS 710) and one or more applications (such as Application(s) 702) designed to run on the guest operating system. VMM 730 presents a virtual operating platform to the guest operating system and manages the execution of the guest operating system.

[0181] In some cases, VMM 730 can allow a guest operating system to run as if it were running directly on the bare hardware 720 of computer system 700. In these instances, the same version of the guest operating system configured to run directly on the bare hardware 720 can also run on VMM 730 without modification or reconfiguration. In other words, VMM 730 can provide full hardware and CPU virtualization to a guest operating system in some situations.

[0182] In other cases, a guest operating system can be specifically designed or configured to run on the VMM730 for improved efficiency. In these instances, the guest operating system is "aware" that it is running on the virtual machine monitor. In other words, the VMM730 can provide paravirtualization to the guest operating system in some situations.

[0183] Computer system processes involve the allocation of hardware processor time and the allocation of (physical and / or virtual) memory. Memory allocation is used to store instructions executed by the hardware processor, to store data generated by the execution of those instructions, and / or to store hardware processor state (e.g., register contents) between hardware processor time allocations when the computer system process is not running. Computer system processes run under the control of the operating system and can also run under the control of other programs executing on the computer system.

[0184] cloud computing

[0185] This article generally uses the term "cloud computing" to describe a computing model that enables on-demand access to a shared pool of computing resources, such as computer networks, servers, software applications, and services, and allows for the rapid provisioning and release of resources with minimal management effort or service provider interaction.

[0186] Cloud computing environments (sometimes called cloud environments or the cloud itself) can be implemented in various ways to best suit different requirements. For example, in a public cloud environment, the underlying computing infrastructure is owned by an organization that makes its cloud services available to other organizations or the public. In contrast, private cloud environments are generally used only by a single organization or within a single organization. Community clouds are designed to be shared by several organizations within a community; while hybrid clouds include two or more types of clouds (e.g., private, community, or public) bound together by data and application portability.

[0187] Generally, cloud computing models enable some of the responsibilities that might have previously been provided by an organization's own IT departments to be delivered as service layers within the cloud environment for use by consumers (inside or outside the organization, depending on the public / private nature of the cloud). Depending on the specific implementation, the precise definition of the components or features provided by or within each cloud service layer can vary, but common examples include: Software as a Service (SaaS), where consumers use software applications running on cloud infrastructure while the SaaS provider manages or controls the underlying cloud infrastructure and applications; Platform as a Service (PaaS), where consumers can use software programming languages ​​and development tools supported by the PaaS provider to develop, deploy, and otherwise control their own applications while the PaaS provider manages or controls other aspects of the cloud environment (i.e., everything in the runtime execution environment); and Infrastructure as a Service (IaaS), where consumers can deploy and run arbitrary software applications and / or provide processing, storage, networking, and other basic computing resources while the IaaS provider manages or controls the underlying physical cloud infrastructure (i.e., everything below the operating system layer). Database as a Service (DBaaS) is a service where consumers use database servers or database management systems running on cloud infrastructure, while the DBaaS provider manages or controls the underlying cloud infrastructure and applications.

[0188] The basic computer hardware and software, as well as the cloud computing environment, described above are presented to illustrate the basic underlying computer components that can be used to implement one or more example embodiments. However, the one or more example embodiments are not necessarily limited to any particular computing environment or computing device configuration. Instead, according to this disclosure, the one or more example embodiments can be implemented in any type of system architecture or processing environment that will be understood by those skilled in the art to be capable of supporting the features and functionality of the one or more example embodiments presented herein.

[0189] In the foregoing description, embodiments of the invention have been described with reference to numerous specific details, which may vary depending on the implementation. Therefore, the description and drawings should be considered illustrative rather than restrictive. The unique and exclusive indication of the scope of the invention, and the content that the applicant intends to define as the scope of the invention, is the literal and equivalent scope of the set of claims published from this application in the specific form of such claims, including any subsequent corrections.

Claims

1. A computer-implemented method, comprising: Each data block in a plurality of data blocks is copied to at least a minimum number of replicated network elements in a plurality of network elements; A specific network element among the plurality of network elements requests access to at least one data block, wherein the requesting step includes: acquiring a write lock for at least one data block among the plurality of data blocks, the write lock providing exclusive access to the at least one data block by the specific network element among the plurality of network elements; Based on the requested access: Modify heatmap, and Copy the at least one data block to the specific network element; Release the write lock for at least one data block; In response to releasing the write lock, the minimum number of replication network elements for the at least one data block is reduced based on the heatmap; and After reducing the minimum number of replicated network elements, the specific network element selects a subset of the plurality of network elements as the minimum number of replicated network elements for the at least one data block.

2. The method of claim 1, wherein selecting a subset of the plurality of network elements includes selecting a subset of the plurality of network elements by the specific network element.

3. The method of claim 2, wherein reducing the minimum number of replicated network elements comprises reducing the minimum number of replicated network elements to two.

4. The method of claim 1, wherein: The acquisition of the write lock is performed in response to the release of the write lock by a second network element among the plurality of network elements.

5. The method of claim 1, wherein the modified heatmap is based on the type of access.

6. The method of claim 1, further comprising, in response to the request for access, recording an identifier for the specific network element and the at least one data block.

7. The method of claim 1, wherein modifying the heatmap includes performing hashing.

8. The method of claim 1, wherein reducing the minimum number of replication network elements for the at least one data block comprises at least one selected from the group consisting of: asynchronous, Predict using Bayesian networks The minimum number of one or more replica network elements is reduced based on the amount of write access requests, and The minimum number of one or more replica network elements is increased based on the amount of read access requests.

9. The method of claim 8, wherein the prediction via Bayesian network comprises calculating the following probabilities: The two data blocks will be accessed together. One data block will be accessed because another data block has already been accessed. The specific network element will access a specific data block, or The lock will be released during the duration.

10. One or more non-transitory computer-readable storage media storing instructions that, when executed by one or more processors, cause the steps of the method as claimed in any one of claims 1-9 to be performed.

11. A computing device, comprising: One or more processors; as well as One or more non-transitory computer-readable storage media storing instructions that, when executed by the one or more processors, cause the steps of the method as described in any one of claims 1-9 to be performed.

12. A computer program product comprising instructions that, when executed by a computer, cause the computer to perform the steps of the method as claimed in any one of claims 1-9.

Images