Method, apparatus, and system for migrating an active file system

Abstract

A computer-implemented method for migrating metadata from a provider file system D having a rooted tree structure to a beneficiary file system B while processing commands operating on the metadata may include replicating the provider file system D with the beneficiary file system B by sequentially copying metadata of nodes in the provider file system D to the beneficiary file system B while no commands operating on the metadata are received. Upon receiving a command operating on the metadata, the command may be executed in both the provider file system D and the beneficiary file system B when all of the arguments of the command are present in both the provider file system D and the beneficiary file system B. When none of the arguments are present in the beneficiary file system B, the command may be executed only in the provider file system D. When only some of the arguments are present in the beneficiary file system B, the command may be queued until at least all of the arguments of the command are present in the beneficiary file system B. When all of the arguments of the command are present in the beneficiary file system B, the queued command is dequeued and scheduled for execution.

Description

[Background technology]

[0001] Today, so-called Big Data has found its natural home in the cloud, where leading companies are capitalizing on the cheap, scalable storage and flexibility that comes with the cloud's powerful analytics platform. However, even though there are compelling advantages to moving Big Data to the cloud, there are also potential business risks.

[0002] One approach to migrating big data to the cloud is through a manual migration. Manual migration is a customized, ad-hoc approach to copying big data. When migrating data manually, administrators create, manage, schedule, and maintain customized or open source scripts to migrate large data sets. When adding data transfer equipment to a big data to cloud migration plan, there is additional customized scripting required to upload the data. Business risks associated with this manual approach to migrating big data to the cloud include data inconsistency, business disruption, and high IT resource requirements. Risk of data inconsistency

[0003] Large datasets take time to migrate to the cloud. At 1 gigabit per second (Gbps), it takes approximately 100 days to migrate 1 petabyte (PB). Even with data transfer equipment, vendor load times can take weeks. While data is available in the cloud, changes and ingestion are still required. Changing data during the long migration time adds risk to accurately bringing large datasets to the cloud. However, when manual migrations rely on customized or open-source scripts that intensively copy data, it may not be feasible for the migration team to validate the accuracy of the migration. In practice, extensive manual coordination does not guarantee completely consistent data output. This approach also does not support new updates that are likely to occur during the migration process. Manual approaches for large-scale data migrations typically do not support changes to data that are modified or generated during the migration. Risk of business disruption

[0004] Due to scale and relevant benefits, organizations have increasingly strategically deployed mission-critical workloads to Hadoop. Enterprise-critical workloads bring with them expectations of availability, integrity, confidentiality, and auditability. In the area of ​​complexity, moving static datasets is easy, while moving dynamically changing datasets with these expectations is extremely challenging. Manual migrations often require meaningful descriptions of how business applications behave during the big data migration. Administrators who choose an incremental migration strategy that brings datasets to the cloud over months face disruptive updates and risk not meeting their service level agreements (SLAs). High IT resource requirements and risk of increased costs

[0005] Companies with significant capital investments that built out data centers to host their Hadoop data and workloads are now just past the typical two- to four-year depreciation period and can amortize these costs. It's easy to shift capital hardware depreciation to operational expenses for the cloud. Companies also invest heavily in people, processes, and applications that support their operational data infrastructure. Adding a manual migration to these sunk costs poses a risk to IT budgets. The operational overhead of attempting a disruptive, downtime-free big data migration is significant. The amount of resources required to create, test, manage, schedule, and maintain customized migration scripts may not be fully recognized or even quantifiable at the start of a data migration project. Due to the customized nature of manual migrations, such migration programs are prone to expense and delays. [Brief explanation of the drawings]

[0006] [Figure 1A] 1 is an illustration of a rooted tree structure for a provider D and a beneficiary B.

[0007] [Figure 1B] 1 is an illustration showing a sliding window of content drawn from a group of nodes bounded by a leading edge and a trailing edge, according to one embodiment.

[0008] [Figure 1C] FIG. 1 illustrates a fence post F as content is being migrated from a provider file system D to a beneficiary file system B, according to one embodiment.

[0009] [Figure 2] 1 illustrates aspects of one embodiment of the present computer-implemented method and system.

[0010] [Figure 3] FIG. 1 is a block diagram further illustrating aspects of the present computer-implemented method and system.

[0011] [Figure 4] FIG. 1 illustrates how repeated renames between unmigrated and migrated areas can stall the migration.

[0012] [Figure 5] FIG. 10 illustrates the behavior of a rename command from an unmigrated portion of beneficiary file system B to a migrated portion of beneficiary file system B.

[0013] [Figure 6] 1 is a flow diagram of a computer-implemented method for migrating metadata from a provider file system D having a rooted tree structure to a beneficiary file system B while processing commands that operate on the metadata, according to one embodiment.

[0014] [Figure 7] 1 illustrates physical hardware that may be used to implement aspects of the present method for migrating metadata from a provider file system D having a rooted tree structure to a beneficiary file system B while processing commands that operate on the metadata, according to one embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0015] Consider copying large amounts of data from a file system (called the provider), such as the Hadoop Filesystem (HDFS), to a destination (the beneficiary), such as cloud storage. Once the data is migrated, you can accomplish a "cut over" to the destination by simply abandoning the provider, or you can maintain replicas of both the provider and the destination by keeping them synchronized using an active-active replication solution. Tools for such active-active replication can be built using distributed consensus protocols, such as Paxos Fusion from WANdisco, Inc.

[0016] As noted above, transferring large data sets (e.g., petabytes or more) takes a long time, even over high-speed communication channels. Overall, therefore, it is conceivable that the activity of migrating large amounts of data could take significantly longer than the owner of that data can avoid needing to modify that data. For example, such a migration could take days or months, depending on the amount of data and available bandwidth, and the owner of the data may not have the option of not modifying the data during that period. Therefore, a key business requirement is the ability to allow client operations to occur on the provider's file system while the migration is in progress. Herein, a migration that allows client operations during the migration process is referred to as a non-disruptive migration. Such a non-disruptive migration is, by its nature, long-lasting, meaning that the migration may take days, weeks, or months to complete. Given the long-lasting nature of such a non-disruptive migration, another business requirement is that any such long-lasting operation must also be made robust enough to withstand failures, such as node and link failures.

[0017] Currently, petabyte-scale migrations are accomplished by first transferring the data to a storage transfer appliance, such as an AWS Import / Export Snowball, physically shipping the data to the cloud vendor, and waiting for the data to be uploaded to the cloud storage, a process that can take more than a week. Importantly, the data is inaccessible until it is uploaded. However, the data loaded into the storage appliance undoubtedly changes during this period, except while it is being migrated to the cloud storage. These changes (deltas) can be captured by taking snapshots and transmitting them over the wire. This process is repeated until both sites have identical content. When the final delta required to correct the content is small enough, it may be necessary to lock the file system and shut down clients, and disrupting customers by such a shutdown is deemed insignificant. Note that there are upper limits on data transfer rates to storage appliances, depending on the vendor. If transfer rate becomes a limiting factor, system administrators are expected to partition file systems and use multiple appliances in parallel. Furthermore, there is no guarantee how long this method will take before the data on both sides is identical. The manual steps involved in this style of migration are limited. Without strong integrity checking, this method is prone to data loss / corruption. Summary of the Invention [Problem to be solved by the invention]

[0018] Therefore, what is needed is a robust method and system for eliminating the risks or high costs of manual big data migration. What is also needed is a method and system for automated data migration at scale across all major commercial Hadoop distributions to the cloud with a single scan of the source storage area, even while the data continues to change. Such a solution should require no scripts, code maintenance, transfer equipment, scheduling, or inspections. [Means for solving the problem]

[0019] Thus, one embodiment enables the automation of large-scale data migration from continuously running production systems to the cloud. When changes are made anywhere in the provider system (where the data is being migrated), the present method and system for live migration ensures that the beneficiary system (where the data is being migrated) has consistent data upon completion. Additionally, the present computer-implemented method and system operates to minimize IT resources using one-click replication with a single scan of the source storage area across all major commercial (e.g., Hadoop) distribution and cloud storage and analytics services. The described computer-implemented live migration method and system includes an automated approach for big data migration that provides validation of data consistency between shared systems. Because changes can be made anywhere in the provider system, the beneficiary has consistent data upon completion, with no data loss and no uncertainty regarding the correctness of the migration. In doing so, the described live migration method and system provides 100% business continuity for hybrid, multi-region, and cloud environments, with production clusters operating continuously. With no impact to the provider's cluster and operations during the migration, the described live migration method and system enables companies to meet their critical SLAs in a timely and cost-effective manner.

[0020] In this specification, the data to be migrated is assumed to be structured in the form of a rooted tree, where the internal nodes of the tree are directories and the leaves represent files or empty directories. Those skilled in the art may recognize that this is an abstraction of the actual file system that ignores symbolic links, constraints on directory and file sizes, ownership permissions, and so on, and that different implementation details address these issues. Nevertheless, such an abstraction is useful in this case. During the migration, the data to be migrated is assumed to change. That is, throughout this disclosure, it is assumed that a file system client continues to issue commands c to provider D that modify data to be migrated, data being migrated, and data that has already been migrated. A migrator process M according to this embodiment is configured with the ability to block command c from being applied to provider D, and such blocking should be done only when unavoidable and in a manner that minimizes disruption. The migrator process M may selectively apply changes to beneficiary B requested by command c, with the goal being for beneficiary B to be identical to provider D after all changes by c have been applied to provider D and provider D has been quiesced. Once the metadata (including but not limited to bidirectional links between the node being migrated and its parent, permissions, last access time, and other last access times and attributes of the file or folder) has been migrated, the task of replicating the corresponding actual data blocks is delegated to another tool, e.g., Fusion. We assume that Beneficiary B is initially empty. If not, Beneficiary B may determine whether it needs to extract content after examining the metadata received from Provider D.

[0021] Thus, one embodiment is an asynchronous migration computer-implemented method M where the last action or command c is applied to provider D, and after D goes to sleep, beneficiary B receives all changes and becomes identical to D. Thus, a simple implementation would be: 1. Block all command c, 2. Copy all data from Provider D to Beneficiary B, 3. Unblock a blocked command queue This may include:

[0022] While this is correct, such an implementation is not optimal because it does not minimize disruption while migrating the file system. Consider a depth-first scan (DFS) traversal of a rooted tree 1002 such as that shown in FIG. 1A. The DFS starts at the root, shown as node 1 in FIG. 1A. After visiting a node, the DFS recursively scans all of the node's descendants starting with its first lexicographical child. After scanning the subtree rooted at the first child, the DFS proceeds to the second child, and so on, until all nodes have been visited. In FIG. 1A, the DFS starts with the first lexicographical child, in this case node 1, and scans from left to right. After node 1, node 2 is scanned, followed by node 3. Because node 3 has two descendants (nodes 4 and 5), the two descendants are scanned before node 2's other descendant, namely node 6. After scanning node 3 and its descendants 4 and 5, node 6, the next lexicographical child of node 2, is scanned. At this stage, node 2 and all of its descendants have been scanned. Proceeding again from left to right, node 7, which has no descendants, is then scanned. Node 8 and its descendants are then scanned. Thus, node 9 is scanned after node 8. Because node 9 has two descendants (nodes 10 and 11), these two descendants are scanned before scanning node 8's other descendant (node ​​12). Because nodes 10 and 11 still need to be scanned, node 9 is the leading edge and the last node scanned. Because nodes 2 and 7 and all of their descendants on provider D's side have already been scanned, these nodes and all of their corresponding descendants have been replicated (at least their metadata), as suggested by the filled vertical lines on beneficiary B's side. Because we have not yet traversed all of node 8's descendants, and because node 9 is the current leading edge of the DFS traversal, node 8 and its branches and descendants are not shown as replicated (at least their metadata) on beneficiary B's side. Thus, the node numbering scheme in this DFS traversal indicates the direction and order of movement.When we say a node is available or present at beneficiary B, we mean that the metadata corresponding to that node, and all of that node's descendants, has been successfully written to the file system at B. In Figure 1A, only the nodes shaded with vertical lines are available at node B; the nodes numbered 1 and 8 through 12 are not. Note that a DFS, in which children are recursively explored alphabetically, results in visiting each path in the file system in lexicographical order. Therefore, nodes can be ordered using the "<" (less than) operation. Accordingly, the concepts of less than and greater than can be used without any confusion with respect to the nodes being scanned. Also, the concepts of "to the left" and "to the right" with respect to the nodes being scanned can also be used without any confusion.

[0023] Also, while DFS is illustrated herein, the embodiments are not limited thereby. In practice, any scan that visits each node at least once (preferably exactly once), for example, breadth-first, can be used. Whatever the order chosen by the scanner at the provider, the beneficiary must be aware of that order so that it can correctly handle the client operation. According to one embodiment, upon receiving a request to retrieve the contents of a node, the beneficiary acknowledges the request after successfully writing the metadata in the target file system. In this approach, the provider and beneficiary work cooperatively in migrating the metadata. In one embodiment, the actual task of transferring the content is delegated to a separate module that may execute after migrating the metadata. The leading edge refers to the last node retrieved by the iterator 2012 (FIG. 3) from the provider file system. Similarly, the trailing edge refers to the smallest (or leftmost) node for which the scanner issued a request but the beneficiary has not yet acknowledged successful writing of the metadata corresponding to that node. Thus, from the provider's perspective, all nodes smaller than the trailing edge are in the migrated portion. The nodes between the leading and trailing edges constitute the in-migration portion of the file system. This set of nodes can be envisioned, according to one embodiment, as a sliding window that grows and shrinks as the provider issues requests and receives acknowledgments from the recipient. In a scenario where the iterator retrieves a single object from the provider file system using getNext() 2016, issues a request to the remote agent, and waits for confirmation before issuing the next request, the size of the sliding window will never be larger than 1. This approach may be too restrictive in many cases. Instead of reaching out-of-disk for every object, one implementation assumes that whenever the sliding window is empty, the iterator retrieves several objects at a time and keeps them in main memory. However, it may be preferable to never allow the sliding window to grow too large.Otherwise, limited resources at the provider or recipient may become overwhelmed, which can lead to a potential crash. Therefore, in most practical implementations, it is wise to have a large window for migration that is limited according to the resources available. Understood in this way, getNext() 2016 represents retrieving a collection of objects to be migrated, although for clarity we can assume that this description is constructed by getting one object at a time from disk.

[0024] FIG. 1B is an illustration showing a sliding window of content retrieved from a node, bounded by a leading edge and a trailing edge, according to one embodiment. As shown, sliding window 102 appears bounded by leading edge 104, which is the last node retrieved by iterator 2012 from provider file system D. Similarly, reference numeral 106 indicates trailing edge, which is the smallest (or leftmost) node for which the scanner has issued a request but for which beneficiary file system B has not yet acknowledged successful writing of the metadata corresponding to that node. Thus, from provider D's perspective, all nodes smaller than the trailing edge (those marked with a checkmark) are in migrated portion 108. As shown in FIG. 1B, the nodes between leading edge 104 and trailing edge 106 constitute in-transition portion 110 of the file system. In the example of FIG. 1B, the iterator 2012 is retrieving more than one node at a time, so the leading edge is not the same as the trailing edge. Nodes that have not yet been retrieved by the iterator 202 from the provider file system D are shown at 112 as to-be-migrated and currently unmigrated nodes 112. As shown in Figure 1B, the in-migration nodes are shown as being captured by a sliding window 102 that grows (to encompass a larger number of nodes) and shrinks (to encompass a smaller number of nodes) as the provider file system D issues requests and receives acknowledgments from the beneficiary file system B.

[0025] Figure 2 illustrates an aspect of a computer-implemented method for migrating an active file system, according to one embodiment. As illustrated herein, the computer-implemented process receives as input a provider file system D and an empty beneficiary file system B, and returns a beneficiary file system B that is identical to provider file system D, at least with respect to the metadata of the data stored in provider file system D. As illustrated in Figure 2, the root of provider file system D is shown as node "n." As further illustrated, commands issued by clients that operate on data in provider file system D are shown as "c," and args[] is the argument array for such command c. In Figure 2, Q is a min heap configured to hold intercepted commands c, with the right-most argument of each command c as the key to the min heap.

[0026] 2, one embodiment includes a scanner 2002 and a client proxy or migrator 2004 disposed between a client issuing a command c and the provider file system D. In this approach, every command c issued by a client is intercepted by the client proxy or migrator 2004, which determines when and how to implement the requested change (e.g., write, append, truncate, move, delete, rename, etc.) encoded in the command c received from the client. At the beneficiary file system B side, a migrator remote agent receives data objects O to be copied from the scanner 2002, writes metadata for such data objects O to the beneficiary file system B, and reports back to the scanner 2002 the success or failure of writing the metadata for the data objects O to the beneficiary file system B.

[0027] According to one embodiment, the scanner 2002 may be configured to scan through each node of the provider file system D, copy the metadata for that node to the beneficiary file system B, notify the min heap Q that the metadata for the current node has been copied to the beneficiary file system B, and repeat the process with the next node until the entire file system tree of the provider file system D has been traversed (in one embodiment, in a single traversal), whereupon the scanner 2002 may signal the completion of the migration to the client proxy or migrator 2004.

[0028] The migrator 2004, in conjunction with the scanner 2002, may be configured to migrate metadata from the provider file system D to the beneficiary file system B when a file system change occurs in the provider file system D, i.e., when a command operates on data (and therefore metadata) in the provider file system D to modify the data. Thus, the provider file system D is an active file system even when it is being migrated to the beneficiary file system B. The goal is to reach a point where the beneficiary file system B is identical to the provider file system D. Note again that this disclosure concerns moving metadata corresponding to data blocks stored by the provider file system D, and not moving the data blocks themselves. One possible method for moving the actual data blocks is described and shown in commonly assigned U.S. Patent Application Publication No. 15 / 457,837, filed March 13, 2017, entitled "Methods, Devices and Systems for Maintaining Consistency of Metadata and Data across Data Centers," which is incorporated herein in its entirety. Other methods may be used.

[0029] According to one embodiment, the client proxy or migrator 2004 may be configured to receive commands c and selectively enable or block execution of such received commands c depending on the current state of the migration. For example, if the migrator 2004 receives a command c that operates on both a data block X that exists only in the provider file system D and a data block Y that exists in both the provider file system D and the beneficiary file system B, one embodiment may be configured to block execution of such command until at least metadata for both data blocks X and Y exists in the beneficiary file system B. This is because the migrator 2004 may have already reached a node in the beneficiary file system B that references data block Y and migrated the metadata for data block Y (otherwise, the metadata for data block Y would not exist in the beneficiary file system B), while not yet reaching the node that contains or references data block X. If a change to the metadata of data block Y is enabled by execution of such command c, such a change may be the result of different changes being made in the provider file system D and the beneficiary file system B, which is not allowed.

[0030] As shown in FIG. 2, the migrator or client proxy may be configured to selectively execute an incoming command c depending on whether all of the arguments to command c are present in beneficiary file system B, some of the arguments to command c are present in beneficiary file system B, or none of the arguments to command c are present in beneficiary file system B. Thus, as shown in FIG. 1, the migrator 2004 may execute command c in both the donor file system D and the beneficiary file system B if all of the arguments to command c are present in both the donor file system D and the beneficiary file system B, with confidence that the metadata (data blocks) of the arguments to command c will remain identical in both the donor file system D and the beneficiary file system B. As shown in the first "else" statement, if none of the arguments to command c are present in B, the command may safely be executed, but only in the donor file system D. This is because the scanner has not reached one or more of the nodes that reference one or more of the arguments to command c. After executing command c, the scanner eventually reaches one or more nodes that reference data blocks in the arguments of command c, and migrates the modified metadata of the involved node or nodes to beneficiary file system B. As noted in FIG. 2 by the second "else" statement, if only some of the arguments of command c exist in beneficiary file system B (meaning some of the arguments of command c do not exist in beneficiary file system B), command c may be queued in min heap Q until at least all of the arguments of command c exist in beneficiary file system B, and as a result, the command may be dequeued and scheduled for execution in both provider file system D and beneficiary file system B, as shown in 3. When scanner 2002 reports that it has traversed the entire provider file system D, any remaining commands c that exist in min heap Q may be dequeued and executed or scheduled for execution.

[0031] FIG. 3 is a block diagram further illustrating aspects of a computer-implemented method for migrating an active file system, according to one embodiment. FIG. 3 shows aspects of this computer-implemented method for migrating an active file system occurring not only in provider file system D toward the top of the diagram, but also in beneficiary file system B toward the bottom of the diagram. The data repository at provider D is indicated by reference numeral 2006, and the data repository at beneficiary B is indicated by reference numeral 2008. According to one embodiment, data repositories 2006 and 2008 are created to be identical at the end of one pass by scanner 2010 while allowing execution of command c. After the single pass, the two repositories remain identical because the arguments to any command c are all present in beneficiary file system B; therefore, command c may be executed in both provider file system D and beneficiary file system B without compromising metadata and data synchronization. 3 is shown with the metadata migration in progress, with both data repository 2006 and data repository 2008 shown to contain not only the data and metadata currently being migrated from data repository 2006 to data repository 2008, but also the data and metadata that has already been migrated from data repository 2006 to data repository 2008. Since beneficiary file system B is assumed to be empty at the start of the migration, only data repository 2006, which serves as provider file system D, contains unmigrated data and metadata.

[0032] In FIG. 3 , the scanner 2010 may include an iterator module 2012 and an update progress module 2014. The iterator module 2012 may be configured to walk through a file system tree, such as a DFS, scanning each node of the provider file system in turn. As shown, the iterator module 2012 may issue a getNext() command 2016 to access the next or first node of the provider file system D, represented by the data repository 2006 in FIG. 3 , as is the case. If that node is the first node accessed, the entire data repository 2006 may be tagged as “unmigrated.” In response to the getNext() command, the first or next data object O may be retrieved in the data repository 2006, as indicated at 2018. The object O (at least its metadata) may then be sent to a migrator remote agent 2020 at the beneficiary side. Upon receiving the metadata for the data object O from the scanner 2010, the migrator remote agent 2020 may copy the metadata for the data object O to the data repository 2008 at the beneficiary side, with the metadata being in the "in transit" portion of the data repository 2008 during the write process and in the "migrated" portion of the data repository 2008 after the copy and successful transfer acknowledgement are complete. Of course, the "in transit" and "migrated" portions of the data repository 2008 need not be physical or logical parts of the data repository 2008. In fact, the metadata may be associated with an "in transit" identifier or tag until the copying is finished and the provider is aware of this fact, and then the metadata may be associated with a "migrated" identifier or tag. Other methods may be used.

[0033] Once the metadata of the data object O has been copied to the data repository 2008 at the beneficiary side, the migrator remote agent 2020 may notify the scanner 2010 of that fact, which may cause the update progress module 2014 to update the scanner 2010's location in the provider file system node traversal and the iterator to issue another getNext() command to traverse the DFS and copy at least the metadata of the data object stored in the data repository 2006 at the provider side to the data repository 2008 at the beneficiary side.

[0034] The above works for the static case where no command c is issued during the migration from the provider to the beneficiary. However, as shown at 2022 in FIG. 3 , when a client issues a command c that modifies a data object and / or its metadata during the migration process, other functionality comes into play to ensure an orderly migration of metadata from the provider file system to the beneficiary file system, according to one embodiment. As shown in FIG. 3 , when command c 2022 is issued by a client through data copy layer 2024 (in one implementation, Fusion, of the present assignee, as noted above), the migrator's scanner 2002 may first determine whether command c is intercepted, as shown at 2025. Stated differently, it may determine whether the leading edge of the scanner has reached the node associated with the command's argument. In an illustrative example, if the scanner has not yet reached the node corresponding to the command's argument, all of the arguments (metadata of) command c exist only in provider file system D, and none exist in beneficiary file system B, because scanner 2002 has not yet reached the arguments. Thus, command c may be executed only in provider file system D because the arguments to command c do not yet exist in beneficiary file system B. Similarly, if the trailing edge of the scanner determines that it has already passed all of the arguments to command c 2022 (NO branch of 2025), command c may also be executed because the arguments to command c already exist in both the provider file system and the beneficiary file system, as implied at 2026. Command c is then applied to both the provider file system and the beneficiary file system.

[0035] However, when scanner 2002 intercepts command c, meaning that some but not all of the arguments to command c exist only in the donor file system, and one or more of the arguments to command c exist in both the donor file system and the beneficiary file system, command c 2022 must not be executed immediately because it would cause the migrated portions of the donor and beneficiary file systems to differ. Moreover, because this migration is intended to be done in one pass, it is not optimal to simply save such commands for later execution after all nodes not involved in the command have been migrated, and then return and execute such commands.

[0036] Thus, one embodiment invokes temporarily blocking execution of the selected command c and inserting it into a command queue 2030, as shown at 2028. In one embodiment, the command queue 2030 is or includes a priority queue, also known as a min heap. In this approach, a command c whose arguments are more likely to be migrated sooner is closer to the head of the queue than a command whose arguments are more likely to be migrated later. Once a command is blocked and inserted into the min heap 2030 at 2028, its execution is postponed and migration proceeds in the same order as before, traversing the node tree of the provider file system D (for example) in the manner of the DFS described above.

[0037] As noted above, once the metadata for data object O has been copied to data repository 2008 at the beneficiary side, migrator remote agent 2020 so notifies scanner 2010, which causes update progress module 2014 to update scanner 2010's position in the provider file system D node traversal and iterator 2012 to issue another getNext() command to traverse the DFS and copy at least the metadata for the data object stored in data repository 2006 at the provider side to data repository 2008 at the beneficiary side. As scanner 2002 processes the node and sends the data object to migrator remote agent 2020 for copying to beneficiary file system B, it is update progress module 2014 that triggers a determination at 2032 whether scanner 2002 is blocking ahead of min heap 2030.

[0038] Once the scanner 2002 has processed all nodes that reference data in the arguments of any blocked command, meaning that the arguments of that command exist in both the provider file system D and the beneficiary file system B (meaning the migrator is no longer blocking the front of the priority queue 2030, see the NO branch at 2032), the command at the front of the min heap 2030 may not be blocked at 2034 and may be dequeued and executed in both the provider file system D and the beneficiary file system B, as associated at 2026. Copying in Adversarial Scenarios

[0039] It would be desirable to claim to have copied all nodes up to their trailing edges throughout the entire scan. However, this claim is impossible to maintain under the following typical adversarial scenarios: 1. If new content is introduced into existing files faster than the introduced content, whether new files or new content, can be transferred, this claim cannot be maintained even if it is temporarily true during a scan. 2. The migration effort can also be prevented without even introducing any new content into the provider file system D. As an illustrative example, consider this snippet of code that performs repeated rename operations: while ( true ) rename a → x / / migrated a, unmigrated x rename x → a

[0040] Figure 4 illustrates how such repeated renaming between unmigrated and migrated regions can stall migration. In Figure 4, a rooted node tree is shown generally at 4002. The shaded portions of the rooted node tree indicate portions of the node tree that have already been migrated 4004, while the unshaded, white portions of the rooted node tree indicate portions that have not yet been migrated.

[0041] The difficulties associated with handling renames and repeated renames are significantly ameliorated by tracking the history of changes a path has undergone. One possible method for tracking rename history, referred to as the Application Program Interface MapFilename by a name mapping service, is described and shown in commonly assigned U.S. Patent Application Publication No. 15 / 457,837, filed March 13, 2017, entitled "Methods, Devices and Systems for Maintaining Consistency of Metadata and Data across Data Centers," which is incorporated herein in its entirety. Other methods may be used.

[0042] For example, different copy mechanisms exist for the particular difficulties associated with repeated directory renames of the type described above. Recall that the leading edge refers to the last node for which the scanner 2002 issued a request for content to be migrated, and the trailing edge refers to the smallest (or leftmost) node for which the scanner issued a request but for which the beneficiary has not yet acknowledged successful writing of the metadata corresponding to that node. If one of the arguments to the rename is the leading edge or the trailing edge, or a node between these two edges, the rename command, according to one embodiment, is temporarily blocked until the scanner exits the leading edge or the trailing edge, or a node between these two edges. Also, the arguments to the rename command are handled as described above, except in the case where some arguments to the rename command are in the migrated area and other arguments are in the unmigrated area. That is, if all of the arguments to the rename command are already available in the beneficiary file system B, the rename may be safely performed in both the provider file system D and the beneficiary file system B. Similarly, if none of the arguments to the rename command are available in the beneficiary file system B, the rename may be safely performed only in the provider file system D. This leads us to consider the following two cases: 1. Rename Migrated to Unmigrated: One embodiment is to perform the rename locally, followed by deleting the node at the remote end, although other implementations are possible. In practice, performing the rename locally, followed by deleting the node at the remote end (i.e., performing the rename command locally and deleting the corresponding object at the remote server), while correct, is not optimally efficient. Suppose the operation is rename s → t, where s and t are migrated and unmigrated, respectively. In the provider file system D, it makes no difference, but in the remote beneficiary file system B, it is wasteful to delete only the subtree rooted at s and later recreate the entire subtree under the name t. One option is to perform the operation immediately in the provider file system D and schedule the remote operation to be handled by the beneficiary file system B for later, for example, after the scanner has had a chance to create the parent of t. 2. Rename from Unmigrated to Migrated: This case requires a more detailed explanation and is handled as follows with reference to Figure 5. When a rename request arrives, it is immediately fulfilled at the provider and the root of the request is pushed onto a size-limited stack 5002, which schedules migrations to treat the specified area as an additional "unmigrated area". In other words, these renames create "holes" in the contiguous migrated area, but since the size of the stack is upper bounded by a predetermined constant, the number of holes is limited at any given time. If the stack containing these additional migrations becomes full, new rename requests of this kind may be rejected. Tracking open files

[0043] Commands issued to a file system have one or more arguments. These arguments are paths that lead to directories or files within the file system. As noted above in

[0022] , these arguments can be compared using the < operator, which defines a total ordering of all possible arguments in a set.

[0044] In one embodiment, efficient migration of a file system can be facilitated by tracking the set S of arguments of commands that are permitted to execute and that operate on unmigrated portions of the provider file system D but are not known to have terminated. Therefore, the set S can be stored in any data structure that supports efficient insertion and deletion of elements and an efficient method for finding the lowest element, called a fencepost F, shown in FIG. 1C. In FIG. 1A, the unmigrated nodes of provider file system D are nodes 10, 11, and 12. As shown, a client is currently interacting with node 11, which is the argument of a command that is permitted to execute and that operates on the unmigrated portion of provider file system D but is not known to have terminated. Thus, S = {11}, and node 11 is also the fencepost (the lowest or smallest element of S). To use another example, assume there is client activity at node 12. Then, the set S, as defined herein, is {12}. Since S has only one element, the lowest (or smallest) element of set S is node 12. Suppose another command is issued to open node 10 while node 12 is still open. Then the set S becomes {10, 12}. Since 10 is the lowest (smallest) element of S, the fence post is now node 10. Then, if node 10 finishes the transition before node 12, S reverts to the one-element set {12}, and fence post F accordingly reverts to node 12. Note that the insertion and deletion of elements in S may be in a random order. The elements of S can be structured as an automatically balanced binary search tree that supports these operations efficiently, for example, a min-heap or a red-black tree (Reference: Cormen, Thomas H., Leiserson, Charles E., Rivest, Ronald L.; Stein, Clifford (2009)

[1990] , Introduction to Algorithms (3rd ed.), MIT Press and McGraw-Hill, ISBN 0-262-03384-4), which is incorporated herein by reference in its entirety.

[0045] A sufficient condition to ensure the correctness of the migration is to synchronize the updates to the scanner 2002 and S so that the scanner never overtakes F. Stated otherwise, the leading edge must not cross the fence post. This can be achieved by not allowing the fence post to move when the scanner advances, i.e., by not allowing any command with an argument <F and ≧ (greater than or equal to) the leading edge, as shown at 114 in FIG. 1C. In other words, any client activity occurring between the leading edge and the fence post at the start of the scan is delayed. Client requests that would result in a lower fence post value are delayed until the scan is complete. According to one embodiment, when the scan is complete, the delayed client requests are permitted to proceed, and some of the delayed client requests now operate on the migrated portion of the file system. The remaining portion that still needs to operate on the non-migrated portion is added to S. Other ways to achieve this may occur to those skilled in the art. Note that F is not defined when S is empty. One way to ensure the correctness of the migration in this situation is to not allow any changes to the non-migrated portion of the file system while the scanner 2002 is advancing. Other ways to handle this situation may also become apparent to those skilled in the art. FIG. 1C illustrates the usefulness of the fence post concept.

[0046] In FIGS. 1A and 1C, the leading edge (which happens to also be the trailing edge) is at node 9. For the file system's getNext(), the scanner only needs to block nodes 9, 10, and 11 (the actual leading edge and its children), i.e., delay changes to these nodes, and node 12 can be freely operated on by the client while the scanner moves on to visit the next node. When the scanner 2002 reaches node 11, the fence post F may be moved to a higher-ranked (right-side) node accordingly. Parentless iterator

[0047] Some client requests can interfere with the natural progression of the scanner 2002, causing them to lose their location within the file system and become orphaned if precautions are not taken. These include: 1. A request to delete a directory that is an ancestor of the previous edge 2. A request to move a directory that is an ancestor of the previous edge

[0048] For example, referring to Figure 1, if you execute the rename 8 → 2 command, getNext() will return an incorrect value when the iterator reaches the last descendant of 8, i.e., 12. Similarly, if you execute a delete x and x is an ancestor of the scanner's current position, it is pointless to migrate the nodes below the subtree rooted at x.

[0049] These types of problems can be addressed by recognizing the offending command, meaning a command with one or more arguments that are leading ancestors. Consider a delete operation. For example, according to one embodiment, a delete x command c may be executed by first running getNext() on x using that command as the node where the scanner should resume its operation, and then deleting x in both the provider file system D and the beneficiary file system B. Rename commands may also be reduced to some combination of renames, such as from unmigrated to migrated, or vice versa, depending on the target, according to one embodiment. For example, if the target of the rename is in a migrated area of ​​the provider file system D, the target may be treated as if it were a rename from an unmigrated area to a migrated area, as detailed above. Similarly, if the target is in an unmigrated area of ​​the provider file system D, the rename may be performed locally, followed by deleting the node on a remote node, or optimizing the node. Launching the Migrator while the donor file system is active

[0050] The migrator may start at any time; that is, the donor file system D may be dormant or may currently be active. If the donor file system D is actively processing commands, one embodiment invokes a step of setting a configurable time t during which the migrator implicitly collects data on open files without starting a scanner. During this time, the migrator learns about activity on the file system so that it can use this information during scanning. The assumption here is that files that need to be migrated remain open for less than time units t.

[0051] A more intrusive approach is to use suggestions so that all further client activity is blocked until a suggestion with the appropriate path is addressed. Eventually the file system reaches a quiescent state, at which point the migrator starts the scanner and unblocks any client activity that may have been blocked. One downside to using suggestions is that if there are long-running operations, client activity may become blocked for an excessive amount of time. Therefore, this approach should be used with caution.

[0052] 6 is a flow diagram of a computer-implemented method for migrating an active file system from a provider file system D to a beneficiary file system B, according to one embodiment. As shown herein, while processing commands that operate on metadata, a computer-implemented method for migrating metadata from a provider file system D having a rooted tree structure to a beneficiary file system B may comprise determining whether a command that operates on metadata of the provider file system D is received, as shown in block B602. If no such command is received, block B604 may be performed, in which the rooted tree structure of the provider file system D may be replicated in the beneficiary file system B by sequentially copying metadata of nodes of the provider file system D to the beneficiary file system B.

[0053] When a command is received that operates on metadata in provider file system D, the received command may be executed in both provider file system D and beneficiary file system B if all of the arguments of the received command exist in both provider file system D and beneficiary file system B, as shown in B606. When none of the arguments of the received command exist in beneficiary file system B, the received command may be executed only in provider file system D, as shown in B608.

[0054] As shown in block B610, when only some of the arguments of the received command are present in beneficiary file system B, the received command may be queued until at least all of the arguments of the received command are present in beneficiary file system B. When all of the arguments of the received command are present in beneficiary file system B, as shown in block B612, the received command may be dequeued and scheduled for execution.

[0055] According to one embodiment, the computer-implemented method may further include dequeuing any remaining commands from the queue and scheduling the dequeued commands for execution when the rooted tree structure of the provider file system D is replicated to the beneficiary file system B. The queue may be a priority queue, such as a minimum heap. Replicating the rooted tree structure of the provider file system D may include traversing and copying the rooted tree structure, for example, in a Depth First Scan (DFS) manner. When the command operating on the metadata includes a delete operation on a node that is an ancestor of a leading edge of the replicated rooted tree structure, the computer-implemented method may further include identifying a next node after the node to be deleted as a node at which sequential copying should be resumed, performing the delete operation on both the provider file system D and the beneficiary file system B, and resuming sequential copying at the identified next node. When the command operating on the metadata includes a rename operation on metadata that has already been copied from the provider file system D to the beneficiary file system B, the computer-implemented method may further include performing the rename operation as if the rename operation were on metadata that has not previously been copied from the provider file system D to the beneficiary file system B. Alternatively, when the command operating on the metadata includes a rename operation on metadata that has not previously been copied from the provider file system D to the beneficiary file system B, the computer-implemented method may further include performing the rename operation in the provider file system D and deleting any nodes affected by the rename operation in the beneficiary file system B. The computer-implemented method may further include independently copying data blocks associated with the metadata from the provider file system D to the beneficiary file system B.The computer-implemented method may further comprise stepping through the rooted tree structure in stages using a scanner process at the beneficiary file system B to retrieve metadata for the next node in the unmigrated portion of the provider file system D and providing the retrieved metadata to a remote agent configured to copy the provided metadata to the beneficiary file system B. Receiving a command to operate on the metadata may comprise intercepting the command from a sender of the command before executing the command to operate to modify the metadata stored in the provider file system D. The computer-implemented method may also include dequeuing and scheduling for execution any remaining commands that are in the queue after the rooted tree structure of the beneficiary file system B has been migrated to the beneficiary file system B. Migrating metadata from the provider file system D to the beneficiary file system B while processing commands that operate on the metadata may, according to one embodiment, be performed through one pass through the rooted tree structure of the beneficiary file system B. Queuing the received command comprises temporarily blocking execution of the received command. The computer-implemented method may further comprise receiving repeated rename commands and temporarily blocking execution of the repeated rename commands by pushing the repeated rename commands onto a size-limited stack for later execution. When the step of copying metadata of nodes of the provider file system D to the beneficiary file system B encounters an open file, the method may further comprise temporarily blocking a portion of the rooted tree structure extending from the open file to a hierarchical subportion of the rooted tree structure.

[0056] Another embodiment is a computer-implemented system comprising a donor file system D having a rooted tree structure, a beneficiary file system B, a migrator remote agent coupled to the beneficiary file system B, and a migrator process configured to intercept commands issued by a sender and modify metadata stored in the donor file system D, the migrator process comprising a scanner process configured to traverse the rooted tree structure of the donor file system D and sequentially transmit metadata retrieved in the donor file system D to the migrator remote agent over a computer network for copying into the beneficiary file system B to replicate the donor file system D to the beneficiary file system B. The migrator process may be further configured to process the intercepted command by executing the received command in both the provider file system D and the beneficiary file system B when all of the intercepted command's arguments are present in both the provider file system D and the beneficiary file system B, executing the received command only in the provider file system D when none of the intercepted command's arguments are present in the beneficiary file system B, queuing the received command until at least all of the intercepted command's arguments are present in the beneficiary file system B when only some of the intercepted command's arguments are present in the beneficiary file system B, and dequeuing and scheduling the intercepted command for execution when all of the intercepted command's arguments are present in the beneficiary file system B. Physical Hardware

[0057] FIG. 7 illustrates a block diagram of a computing device in which embodiments may be implemented. The computing device of FIG. 7 may include a bus 701 or other communication mechanism for communicating information and one or more processors 702 coupled with the bus 701 for processing information. The computing device may further include random-access memory (RAM) or other dynamic storage device 704 (referred to as main memory) coupled to the bus 701 for storing information and instructions to be executed by the one or more processors 702. The main memory (which is tangible and non-transitory; this term excludes signals per se and waveforms herein) 704 may also be used to store temporary variables or other intermediate information during execution of instructions by the processor(s) 702. The computing device of FIG. 7 may also include read-only memory (ROM) and / or other static storage device 706 coupled to the bus 701 for storing static information and instructions for the one or more processors 702. A data storage device 707, such as a magnetic disk and / or solid data storage device, may be coupled to bus 701 for storing information and instructions, such as required to perform the functionality shown and disclosed in connection with FIGS. 1 through 6. The computing device may also be coupled to a display device 721 via bus 701 for displaying information to a computer user. An alphanumeric input device 722, including alphanumeric and other keys, may be coupled to bus 701 for communicating information and command selections to the one or more processors 702. Another type of user input device is a cursor control 723, such as a mouse, trackball, or cursor direction keys, for communicating directional information and command selections to the one or more processors 702 and for controlling cursor movement on the display device 721. The computing device of FIG. 7 may be coupled to a network 726 via a communications interface (e.g., a modem, network interface card, or NIC) 708.

[0058] As shown, storage device 707 may include a direct-access data storage device such as a magnetic disk 730, non-volatile semiconductor memory (EEPROM, flash, etc.) 732, or a hybrid data storage device comprising both magnetic disk and non-volatile semiconductor memory, as suggested at 731. Reference numerals 704, 706, and 707 are examples of tangible, non-transitory computer-readable media having stored thereon data that, when executed by one or more computing devices, implement aspects of the embodiments described and illustrated herein. Some of these instructions may be stored locally within the client computing device, while others may be stored (and / or executed) remotely and communicated to the client computing device via network 726. In other embodiments, all of these instructions may be stored locally on the client or other standalone computing device, while in still other embodiments, all of these instructions are stored and executed remotely (e.g., on one or more remote servers), with the results communicated to the client computing device. In yet another embodiment, the instructions (processing logic) may be stored on another form of tangible, non-transitory computer-readable medium, such as that shown at 728. For example, reference numeral 728 may be implemented as an optical (or some other storage technology) disk that may constitute a data carrier suitable for loading the stored instructions onto one or more computing devices, thereby reconfiguring the one or more computing devices into one or more of the embodiments described and shown herein. In other implementations, reference numeral 728 may be embodied as an encrypted solid state drive. Other implementations are possible.

[0059] Embodiments of the present invention relate to using a computing device to replicate an active file system. According to one embodiment, the methods, devices, and systems described herein may be provided by one or more computing devices in response to one or more processors 702 executing sequences of instructions contained in memory 704 that embody aspects of the computer-implemented methods shown and described herein. Such instructions may be read into memory 704 from another computer-readable medium, such as data storage device 707 or another (optical, magnetic, etc.) data carrier, such as shown at 728. Execution of the sequences of instructions contained in memory 704 causes one or more processors 702 to perform the steps and functions described herein. In alternative embodiments, hardwired circuitry may be used in place of, or in combination with, software instructions to implement the described embodiments. Consequently, the embodiments are not limited to any specific combination of hardware circuitry and software. Indeed, those skilled in the art will recognize that any suitable computer system may implement the functionality described herein. The computing device may include one or more microprocessors operative to perform the desired functions. In one embodiment, instructions executed by one or more microprocessors are operable to cause the one or more microprocessors to perform the steps described herein. The instructions may be stored on any computer-readable medium. In one embodiment, the instructions may be stored in non-volatile semiconductor memory external to the microprocessor or integrated with the microprocessor. In another embodiment, the instructions may be stored on disk and loaded into volatile semiconductor memory before execution by the microprocessor.

[0060] Portions of the above detailed description describe processes and symbolic representations of operations by a computing device, which may include computer components including a local processing unit, memory storage for the local processing unit, display devices, and input devices. Moreover, such processes and operations may utilize computer components in a heterogeneous, distributed computing environment, including, for example, remote file servers, computer servers, and memory storage devices. These distributed computing components may be accessible to the local processing unit over a communications network.

[0061] The processes and operations performed by the computer include the manipulation of data bits by a local processing unit and / or a remote server, and the maintenance of these bits within data structures residing in one or more local or remote memory storage devices. These data structures impose a physical organization on the collection of data bits stored within the memory storage devices and represent electromagnetic spectrum elements. Moreover, the computer-implemented methods disclosed herein improve computer functionality by enabling file system migration from a donor file system to a beneficiary file system while issuing and executing commands that modify metadata and file system data. Such computer-implemented methods cannot be performed as effectively by human mental processes.

[0062] Processes, such as computer-implemented methods described and illustrated herein, may generally be defined as a sequence of computer-executed steps leading to a desired result. These steps require physical manipulations of physical quantities. Usually, though not necessarily, these quantities may take the form of electrical, magnetic, or optical signals capable of being stored, transferred, combined, compared, and otherwise manipulated. It is common for those skilled in the art to refer to these signals as bits or bytes (when these signals have binary logic levels), pixel values, operations, values, elements, symbols, characters, terms, numbers, points, records, objects, images, files, directories, subdirectories, or the like. However, it should be borne in mind that these and similar terms are to be associated with the appropriate physical quantities of computer operations and that these terms are merely general labels applied to physical quantities present within and during the operations of a computer.

[0063] Furthermore, it should be understood that operations within a computer are often referred to in terms such as adding, comparing, moving, positioning, arranging, illuminating, removing, modifying, etc. The operations described herein are machine operations performed in conjunction with various inputs provided by a human operator or user, either a human or an artificial intelligence agent interacting with the computer. Machines used to perform the operations described herein include local or remote general-purpose digital computers or other similar computing devices.

[0064] Additionally, the programs, processes, methods, etc. described herein are not related to or limited to any particular computer or apparatus, or to any particular communications network architecture. Rather, various types of general-purpose hardware machines may be used with program modules constructed in accordance with the teachings described herein. Similarly, it may prove advantageous to construct specialized apparatuses to perform the method steps described herein using specialized computer systems in particular network architectures, with programs stored in hard-wired logic or non-volatile memory, such as read-only memory.

[0065] While certain example embodiments have been described, these embodiments are presented by way of example only and are not intended to limit the scope of the embodiments disclosed herein. Consequently, nothing in the foregoing description is intended to imply that any particular feature, characteristic, step, module, or block is necessary or essential. Indeed, the novel methods and systems described herein may be embodied in a variety of other forms, and various omissions, substitutions, and modifications may be made to the methods and systems described herein without departing from the spirit of the embodiments disclosed herein.

[0066] The accompanying claims and their equivalents are intended to cover such forms or modifications as fall within the scope and spirit of the present disclosure. For example, those skilled in the art will recognize that the actual physical and logical structures of various embodiments may differ from those depicted in the figures. Depending on the embodiment, certain steps described in the above examples may be omitted, and other steps may be added. Also, the features and attributes of the specific embodiments disclosed above may be combined in different ways to form additional embodiments, all of which fall within the scope of the present disclosure. While the present disclosure provides certain preferred embodiments and applications, other embodiments apparent to those skilled in the art, including embodiments that do not provide all of the features and advantages set forth herein, are also within the scope of the present disclosure. Accordingly, the scope of the present disclosure is intended to be defined solely by reference to the appended claims.

Claims

1. 1. A computer-implemented method for migrating metadata from a provider file system D having a rooted tree structure to a beneficiary file system B while processing commands operating on the metadata, comprising: replicating the rooted tree structure of the provider file system D in the beneficiary file system B by sequentially copying metadata of nodes of the provider file system D to the beneficiary file system B while not receiving commands to operate on the metadata; receiving a command to operate on the metadata; executing the received command on both the provider file system D and the beneficiary file system B when all of the metadata specified by the arguments of the received command exists in both the provider file system D and the beneficiary file system B; executing the received command only in the provider file system D when none of the metadata specified by the arguments of the received command exists in the beneficiary file system B; when only a portion of the metadata specified by the arguments of the received command exists in the beneficiary file system B, queuing the received command until at least all of the metadata relating to the arguments of the received command exists in the beneficiary file system B; when all of the metadata specified by the arguments of the received command is present in the beneficiary file system B, removing the received command from the queue and scheduling it for execution; 1. A computer-implemented method comprising:

2. 2. The computer-implemented method of claim 1, further comprising removing any remaining commands from the queue and scheduling the commands removed from the queue for execution when the rooted tree structure of the provider file system D is replicated to the beneficiary file system B.

3. The computer-implemented method of claim 1 , wherein the queue is a min-heap.

4. 2. The computer-implemented method of claim 1, wherein replicating the rooted tree structure of a provider file system D comprises traversing and copying the rooted tree structure in a depth-first scan (DFS) manner.

5. 2. The computer-implemented method of claim 1, further comprising: when the command operating on the metadata includes a delete operation of a node that is an ancestor of the leading edge of the replica of the rooted tree structure, identifying a next node after the node to be deleted as the node at which to resume the sequential copy, and performing the delete operation on both the provider file system D and the beneficiary file system B to resume the sequential copy at the identified next node.

6. 2. The computer-implemented method of claim 1, further comprising: when the command operating on the metadata includes a rename operation of metadata that has already been copied from the provider file system D to the beneficiary file system B, performing the rename operation as if the rename operation were a rename operation of metadata that has not previously been copied from the provider file system D to the beneficiary file system B.

7. 2. The computer-implemented method of claim 1, further comprising: when the command operating on the metadata includes a rename operation for metadata that has not previously been copied from the provider file system D to the beneficiary file system B, performing the rename operation in the provider file system D and deleting any nodes in the beneficiary file system B affected by the rename operation.

8. The computer-implemented method of claim 1 , further comprising independently copying data blocks associated with the metadata from the provider file system D to the beneficiary file system B.

9. traversing the rooted tree structure stepwise using a scanner process to retrieve metadata for the next node in the unmigrated portion of the provider file system D and provide the retrieved metadata to a remote agent in the beneficiary file system B.

2. The computer-implemented method of claim 1, further comprising: wherein the remote agent is configured to copy the provided metadata to the beneficiary file system B.

10. 10. The computer-implemented method of claim 9, further comprising: preventing the scanner process from retrieving the metadata of the next node in the unmigrated portion of the provider file system D if retrieving the metadata of the next node in the unmigrated portion of the provider file system D would cause the scanner process to overtake a lowest element of the set S of nodes in the rooted tree structure that is an argument of a command that is authorized to be executed and that operates on the unmigrated portion of the provider file system D and is not known to have terminated.

11. 2. The computer-implemented method of claim 1, wherein receiving a command to operate on the metadata comprises intercepting the command from a sender of the command before the command is executed and operates to change metadata stored in the provider file system D.

12. 2. The computer-implemented method of claim 1, further comprising removing from the queue and scheduling for execution any remaining commands that are in the queue after the rooted tree structure of the provider file system D is migrated to the beneficiary file system B.

13. 2. The computer-implemented method of claim 1, wherein the step of migrating the metadata from the provider file system D to the beneficiary file system B while processing the command that operates on the metadata is performed through one pass of the rooted tree structure of the provider file system D.

14. The computer-implemented method of claim 1 , wherein queuing the received command comprises temporarily blocking execution of the received command.

15. 2. The computer-implemented method of claim 1, further comprising receiving a repeated rename command and temporarily blocking execution of the repeated rename command by pushing the repeated rename command onto a size-limited stack for later execution.

16. 2. The computer-implemented method of claim 1, further comprising the step of temporarily blocking a portion of the rooted tree structure extending from the open file to a hierarchical lower portion of the rooted tree structure when the step of copying the metadata of the node of the provider file system D to the beneficiary file system B encounters an open file.

17. 2. The computer-implemented method of claim 1, wherein the step of sequentially copying comprises copying metadata of a sliding window encompassing at least one node bounded by a leading node that is the last node from which metadata was retrieved for copying, and a trailing node that is the lowest node from which a request for metadata was issued and for which the beneficiary file system B has not yet acknowledged successful writing of the request for the metadata.

18. 18. The computer-implemented method of claim 17, further comprising: moving the sliding window to retrieve additional metadata from the provider file system D when an acknowledgment of a successful write is received from the beneficiary file system B and when the last node of the provider file system D from which metadata was retrieved has been updated.

19. 20. The computer-implemented method of claim 18, wherein the number of nodes of the provider file system D within the sliding window increases when a request for metadata is issued to the provider file system D and decreases when an acknowledgment of successful writing of the metadata is received from the beneficiary file system B.

20. 2. The computer-implemented method of claim 1, further comprising: not allowing a leading edge node, which is the last node in the rooted tree structure from which metadata has been retrieved, to overtake a lowest element of a set S of nodes in the rooted tree structure that is an argument of a command that is authorized to execute and that operates on an unmigrated portion of the provider file system D and is not known to have terminated.

21. 1. A computer-implemented system comprising: a provider file system D having a rooted tree structure; a beneficiary file system B; a migrator remote agent coupled to said beneficiary file system B; a migrator process configured to intercept commands issued by a sender to modify metadata stored in the provider file system D, the migrator process comprising a scanner process configured to traverse a rooted tree structure of the provider file system D and sequentially transmit metadata obtained from the rooted tree structure to the migrator remote agent over a computer network for sequential copying into the beneficiary file system B to replicate the provider file system D to the beneficiary file system B; Equipped with executing the intercepted command on both the provider file system D and the beneficiary file system B when all of the metadata specified by the arguments of the intercepted command exists in both the provider file system D and the beneficiary file system B; executing the intercepted command only in the provider file system D when none of the metadata specified by the arguments of the intercepted command exists in the beneficiary file system B; when only a portion of the metadata specified by the arguments of the intercepted command is present in the beneficiary file system B, queuing the intercepted command at least until all of the arguments of the intercepted command are present in the beneficiary file system B; and removing the intercepted command from the queue and scheduling it for execution when all of the metadata specified by the arguments of the intercepted command is present in the beneficiary file system B. wherein the migrator process is further configured to process the intercepted command.

22. 22. The computer-implemented system of claim 21, wherein the migrator process is further configured to remove any remaining commands from the queue and schedule the commands removed from the queue for execution when the rooted tree structure of the provider file system D is replicated to the beneficiary file system B.

23. 22. The computer-implemented system of claim 21, wherein the queue is a min-heap.

24. 22. The computer-implemented system of claim 21, wherein replicating the rooted tree structure of a provider file system D comprises traversing and copying the rooted tree structure in a depth-first scan (DFS) manner.

25. 22. The computer-implemented system of claim 21, wherein when the command operating on the metadata includes a delete operation of a node that is an ancestor of a leading edge of the replica in the rooted tree structure, the migrator process is further configured to identify a next node after the node to be deleted as the node at which to resume the sequential copy, perform the delete operation on both the provider file system D and the beneficiary file system B, and resume the sequential copy at the identified next node.

26. 22. The computer-implemented system of claim 21, wherein when the command operating on the metadata includes a rename operation of metadata that has already been copied from the provider file system D to the beneficiary file system B, the migrator process is further configured to perform the rename operation as if the rename operation were a rename operation of metadata that has not previously been copied from the provider file system D to the beneficiary file system B.

27. 22. The computer-implemented system of claim 21, wherein when the command operating on the metadata includes a rename operation for metadata that has not previously been copied from the provider file system D to the beneficiary file system B, the migrator process is further configured to perform the rename operation in the provider file system D and delete any nodes in the beneficiary file system B affected by the rename operation.

28. 22. The computer-implemented system of claim 21, further comprising a process for independently copying data blocks associated with the metadata from the provider file system D to the beneficiary file system B.

29. 22. The computer-implemented system of claim 21, further configured to precede the step of the scanner process retrieving the metadata of the next node in the unmigrated portion of the provider file system D if retrieving the metadata of the next node in the unmigrated portion of the provider file system D would overtake a lowest element of the set of nodes S of the rooted tree structure that is an argument of a command that has permission to execute, operates on the unmigrated portion of the provider file system D, and is not known to have terminated.

30. 22. The computer-implemented system of claim 21, wherein the intercepted command is received by the migrator process before the command is executed and operates to modify metadata stored in the provider file system D.

31. 22. The computer-implemented system of claim 21, wherein the migrator process is further configured to remove from the queue and schedule for execution any remaining commands present in the queue after the rooted tree structure of the provider file system D is migrated to the beneficiary file system B.

32. 22. The computer-implemented system of claim 21, wherein the migrator process is further configured to migrate metadata from the provider file system D to the beneficiary file system B while processing commands that operate on the metadata executed through a single pass of the rooted tree structure of the provider file system D.

33. 22. The computer-implemented system of claim 21, wherein the migrator process is further configured to temporarily block execution of the intercepted command when queuing the intercepted command.

34. 22. The computer-implemented system of claim 21, wherein the migrator process is further configured to intercept repeated rename commands and temporarily block execution of the repeated rename commands by pushing the repeated rename commands onto a size-limited stack for later execution.

35. 22. The computer-implemented system of claim 21, wherein the migrator process is further configured to, when a scanner process encounters an open file in the provider file system D, temporarily block replicating a portion of the rooted tree structure extending from the open file to hierarchical subportions of the rooted tree structure.

36. 22. The computer-implemented system of claim 21, wherein the Migrater process of the step of sequentially copying comprises copying metadata of a sliding window encompassing at least one node bounded by a leading node that is the last node from which metadata was retrieved and a trailing node that is the lowest node from which a request for metadata was issued and for which the beneficiary file system B has not yet acknowledged successful writing of the request for the metadata.

37. 37. The computer-implemented system of claim 36, wherein the migrator process further comprises moving the sliding window to retrieve additional metadata from the provider file system D upon receiving an acknowledgment of a successful write from the beneficiary file system B and upon updating the last node of the provider file system D from which metadata was retrieved.

38. 38. The computer-implemented system of claim 37, wherein the number of nodes of the provider file system D within the sliding window increases when a request for metadata is issued to the provider file system D and decreases when an acknowledgment of a successful write is received from the beneficiary file system B.

39. 2. The computer-implemented method of claim 1, further comprising: not allowing a leading edge node, which is the last node in the rooted tree structure from which metadata has been retrieved, to overtake a lowest element of a set S of nodes in the rooted tree structure that is an argument of a command that is authorized to execute and that operates on an unmigrated portion of the provider file system D and is not known to have terminated.

Images