Metadata-based data replication

Abstract

A plurality of computing nodes are monitored for updates to one or more files located on the plurality of computing nodes. A first file operation related to a first file located on a first computing node is detected based on the monitoring. The first file is associated with a first copy located on a second computing node. The first file operation is determined to be a copy operation of the first file to a second file located on the first computing node. In response to the copy operation, a copy command related to the first file is forwarded to the second computing node. In response to the copy operation, a second copy file is created on a second computing node of the plurality of computing nodes. The second copy file is associated with the second file located on a first computing node of the plurality of computing nodes.

Description

[Technical Field]

[0001] The present disclosure relates to file replication, and more particularly to leveraging metadata to perform data copy operations. [Background technology]

[0002] Data replication may operate to replicate file operations performed on a first computer to other computers. Data replication may enable computers located in different locations to maintain a consistent logical file system for user access. Data replication may be performed using processing cycles and network bandwidth. Summary of the Invention [Means for solving the problem]

[0003] According to the present embodiments, a method, a system, and a computer program product are disclosed.

[0004] A data replication system monitors updates to one or more files located on a plurality of computing nodes. A first file operation related to a first file is detected based on the monitoring. The first file is located on a first computing node of the plurality of computing nodes. The first file is associated with a first copy file. The first copy file is located on a second computing node of the plurality of computing nodes. Based on the first file operation, it is determined that the first file operation is a copy operation of the first file to a second file. The second file is located on the first computing node of the plurality of computing nodes. In response to the copy operation, a copy command related to the first file is forwarded to the second computing node of the plurality of computing nodes. In response to the copy operation, a second copy file is created on the second computing node of the plurality of computing nodes. The second copy file is associated with the second file located on the first computing node of the plurality of computing nodes.

[0005] According to an embodiment, forwarding the copy command further includes, in response to the copy operation, updating metadata associated with the first file with the copy command, and, in response to the metadata, forwarding the updated metadata of the first file to a second computing node of the plurality of computing nodes.

[0006] According to an embodiment, the first file including the data of the first file is not transferred to a second computing node of the plurality of computing nodes.

[0007] The above summary is not intended to describe each illustrated embodiment or every implementation of the present disclosure.

[0008] The drawings included in this application are incorporated into and form a part of this specification. The drawings illustrate embodiments of the present disclosure and, together with the description, serve to explain the principles of the present disclosure. The drawings are merely illustrative of certain embodiments and are not intended to limit the disclosure. [Brief explanation of the drawings]

[0009] [Figure 1] FIG. 1 illustrates representative major components of an exemplary computer system that may be used in accordance with some embodiments of the present disclosure. [Figure 2] FIG. 2 illustrates a cloud computing environment in accordance with an embodiment of the present invention. [Figure 3] FIG. 3 illustrates abstraction model layers according to an embodiment of the present invention. [Figure 4] FIG. 4 illustrates an exemplary system for performing data replication consistent with some embodiments of the present disclosure. [Figure 5] FIG. 5 illustrates an exemplary method for performing data replication consistent with some embodiments of the present disclosure. [Figure 6] FIG. 6 illustrates an exemplary method for performing data replication consistent with some embodiments of the present disclosure.

[0010] While the invention is susceptible to various modifications and alternative forms, specific features thereof have been shown by way of example in the drawings and will be described in detail. It should be understood, however, that the intention is not to limit the invention to the particular embodiments described. On the contrary, the intention is to cover all modifications, equivalents, and alternatives falling within the scope of the invention. DETAILED DESCRIPTION OF THE INVENTION

[0011] Aspects of the present disclosure relate to data replication, and more particularly to leveraging metadata to perform data copy operations. While the present disclosure is not necessarily limited to such applications, various aspects of the present disclosure will be understood through the discussion of various examples using this context.

[0012] Data replication systems can be used in an increasingly diverse range of information and computer systems. Data replication systems can be part of a distributed file system. Distributed file systems support local disks on cluster nodes and storage area networks (SANs). Logical and physical separation is supported, so that file sets can be separated within the file system (logical separation) or can be part of separate storage pools (physical separation). An example of a data replication system is the IBM Global Parallel File System. 商標 (GPFS: Global Parallel File System 商標 ) Active File Management for Distributed File Systems 商標 (AFM: Active File Management)

[0013] The data replication system may be a scalable, high-performance file system caching layer integrated into a distributed file system. The data replication system may enable the creation of an association from a local cluster of systems to a remote cluster of systems. Data replication may enable the creation of an association from a local cluster of systems to another storage device (e.g., an auxiliary storage device, a cold storage device, a backup device). The data replication system may also enable the implementation of a single namespace view across multiple clusters or multiple geographic locations. The data replication system may use a home and cache model. For example, a single home provides primary storage of data, and exported data is cached within the local file system.

[0014] The data replication system may further enable augmenting the recovery operation of an entire computer system through fileset-level replication disaster recovery capabilities. The disaster recovery operation may implement a strict one-to-one active-passive model. The data replication system may implement a two-site policy involving a primary site and a secondary site. The primary site may be a read-write fileset where an application is currently running, and the primary site has read and write access to the data. The secondary site may be recommended to be read-only. All data at the primary site may be asynchronously replicated to the secondary site. All file user data, metadata, hard links, renames, and clones at the primary site may be replicated to the secondary site. All file system and fileset-related attributes, such as user, group, or fileset quotas, replication factors, and dependent filesets, may not be replicated from the primary site to the secondary site. A consistent point-in-time view of the data in the primary fileset may be propagated inline to the secondary fileset using fileset-based snapshots.

[0015] A data replication system may have performance or other computing drawbacks. For example, a data replication system may perform the same file system operation on a local file system and on a remote file system. When the local file system updates or saves a local file, the corresponding remote copy of the file may also be updated. To maintain system integrity, the replication operation may occur frequently or in near real time. In some instances, the replication may be performed asynchronously in near real time. Examples of file operations may be create file, update file, move file, copy file, delete file, and other related file operations.

[0016] As more complex operations are performed on a distributed file system, data replication systems can encounter performance bottlenecks. For example, if many users perform updates to data stored in one portion of the distributed file system, all updates must be transmitted across the network between various computing nodes. In another example, a large, complex data set may occupy a large portion of the network bandwidth for transmission between computing nodes. The updates may be delayed or queued for later execution, but data integrity issues may arise because the delayed files become out of sync with the workloads running on each node. In many of these cases, the memory and processing power of each node may be tied up in processing operations to offset network performance. For example, a distributed file system may choose to have each node that updates a file compress the file before transferring it, and then have the receiving node decompress the file on the other side. The compression and decompression may consume memory and processor resources.

[0017] A metadata aware data replication system (MADRS) may provide improved data replication. The MADRS may operate by detecting a file operation on a local computing node and determining that the file operation is a copy operation. The MADRS may operate by determining that the copy operation is to be performed on an existing first file (e.g., a local source file) already located on the local computing node and directed to a new second file (e.g., a local destination file). The MADRS may further operate by determining that an existing first file on the local computing node has an existing replicated copy (alternatively, a first copy file) already located on a second computing node. The MADRS may forward a copy command directed to the respective replicated copy of the first file (alternatively, the first copy file) to a second computing node of the data replication system. The MADRS may update the metadata of the first file to indicate that the first file will be copied to the second file (e.g., to notify the copy command prior to the transfer). The MADRS may include a path, destination, uniform resource location (URL), or other file system location in the metadata. The data replication system may detect the command (or updated metadata) and perform a remote copy on a second computing node based only on the existing replicated copy, for example, by creating a new replicated second file corresponding to the new second file based on the existing replicated first copy file and based on information stored in the command (or metadata).

[0018] MADRS may transfer only metadata and not the file to any of the remote nodes in the distributed file system. For example, a first local file may be copied to a second local file on a first computing node. MADRS may detect the copy and generate metadata associated with the first local file. The generated metadata associated with the first local file may include the name and destination (e.g., path) of the second local file. The generated metadata may cause a portion of a data replication system to transfer the updated metadata to a second computing node (or any other computing node) in the distributed file system. The second computing node may create the second copy file on the second computing node based on the updated metadata. The creation may be performed based only on path information from the metadata and based on the first copy file, without any access to the original, uncopied version of the first local file.

[0019] In some embodiments, MADRS may forward only the copy command and not the file to any remote nodes in the distributed file system, meaning that the file data is not transferred to other computing nodes. For example, a first local file may be copied into a second local file on a first computing node. MADRS may detect that the first local file has been copied and set metadata associated with the first local file to include the copy operation and a path to the second local file. In response, the system may detect the metadata and generate a copy command to transfer to another computing node (e.g., a second computing node) in the distributed file system. In response to the received copy command, the second computing node may create a second copy of the first copy on the second computing node. The creation may be performed based solely on path information from the command, without any access to the original, uncopied version.

[0020] The copy operation may be a privileged operation, such as a privileged application, kernel, superuser, operating system, hypervisor, etc. For example, a copy operation from a first file represented by a first local node to a second file represented by a second local node. The copy operation may be performed using a byte-by-byte copy to another file, or may be performed using an in-kernel copy operation using a system call between two file descriptors, etc. The copy operation may be performed by a non-privileged operation, such as a user application or program running on a computing node. For example, the non-privileged operation may further utilize the byte-by-byte copy to avoid the additional memory and processing costs of transferring data from the kernel to user space and then back to the kernel. The non-privileged operation may be a "CP" command issued by a user with limited privileges. The non-privileged operation may be a "move" command issued by an application without elevated privileges.

[0021] Non-transferring files may save memory, processing, and network usage. Because the metadata and / or copy commands may be relatively small in size, the transfer cost relative to network performance may be low. Furthermore, because the metadata and / or commands do not need to be compressed, the processing and memory usage of MADRS may be significantly reduced. In practice, many bytes of bandwidth and many input / output operations may be avoided by simply transmitting the metadata and / or commands.

[0022] FIG. 1 illustrates representative major components of an exemplary computer system 100 (alternatively, a computer) that may be used in accordance with some embodiments of the present disclosure. It is understood that the individual components may vary in complexity, number, type, or configuration, or combinations thereof. The specific example disclosed is for illustrative purposes only and is not necessarily the only such variation. Computer system 100 may include a processor 110, memory 120, an input / output interface (herein, I / O or I / O interface) 130, and a main bus 140. Main bus 140 may provide a communication path for other components of computer system 100. In some embodiments, main bus 140 may connect to other components, such as a component-specific digital signal processor (not shown).

[0023] Processor 110 of computer system 100 may be comprised of one or more cores 112A, 112B, 112C, and 112D (collectively 112). Processor 110 may further comprise one or more memory buffers or caches (not shown) that provide temporary storage of instructions and data for cores 112. Core 112 may execute instructions on input provided from the cache or from memory 120 and output results to the cache or memory. Core 112 may be comprised of one or more circuits configured to execute one or more methods consistent with embodiments of the present disclosure. In some embodiments, computer system 100 may comprise multiple processors 110. In some embodiments, computer system 100 may have a single processor 110 with a single core 112.

[0024] The memory 120 of the computer system 100 may include a memory controller 122. In some embodiments, the memory 120 may include a random-access semiconductor memory, storage device, or storage medium (either volatile or non-volatile) for storing data and programs. In some embodiments, the memory may be in the form of a module (e.g., a dual in-line memory module). The memory controller 122 may communicate with the processor 110 to facilitate the storage and retrieval of information in the memory 120. The memory controller 122 may communicate with the I / O interface 130 to facilitate the storage and retrieval of inputs or outputs in the memory 120.

[0025] I / O interface 130 may include an I / O bus 150, a terminal interface 152, a storage interface 154, an I / O device interface 156, and a network interface 158. I / O interface 130 may connect main bus 140 to I / O bus 150. I / O interface 130 may direct instructions and data from processor 110 and memory 120 to various interfaces of I / O bus 150. I / O interface 130 may also direct instructions and data from various interfaces of I / O bus 150 to processor 110 and memory 120. The various interfaces may include terminal interface 152, storage interface 154, I / O device interface 156, and network interface 158. In some embodiments, the various interfaces may include a subset of the interfaces described above (e.g., an embedded computer system in an industrial application may not include terminal interface 152 and storage interface 154).

[0026] Logical modules throughout computer system 100 (including, but not limited to, memory 120, processor 110, and I / O interface 130) may communicate faults and changes to one or more components to a hypervisor or operating system (not depicted). The hypervisor or operating system may allocate the various resources available in computer system 100 and track the location of data within memory 120 and the locations of processes assigned to the various cores 112. In embodiments that combine or rearrange elements, aspects and capabilities of the logical modules may be combined or redistributed. These variations will be apparent to those skilled in the art.

[0027] Although this disclosure includes detailed descriptions related to cloud computing, it should be understood that implementation of the teachings recited herein is not limited to a cloud computing environment. Rather, embodiments of the present invention can be implemented in conjunction with any other type of computing environment now known or later developed.

[0028] Cloud computing is a service delivery model for enabling convenient, on-demand network access to a shared pool of configurable computing resources (e.g., networks, network bandwidth, servers, processing, memory, storage, applications, virtual machines, and services) that can be rapidly provisioned and released with minimal administrative effort or interaction with the service provider. The cloud model can include at least five characteristics, at least three service models, and at least four deployment models.

[0029] The features are as follows:

[0030] On-demand self-service: Cloud consumers can unilaterally provision computing capacity, such as server time and network storage, as needed, without requiring human interaction with the provider of the service.

[0031] Broad network access: Functionality is available over the network and accessed via standard mechanisms that facilitate use by heterogeneous thin or thick client platforms (e.g., mobile phones, laptops, and PDAs).

[0032] Resource Pooling: A provider's computing resources are pooled to serve multiple consumers using a multi-tenant model, and various physical and virtual resources are dynamically allocated and reallocated according to demand. Consumers generally have no control or knowledge of the exact location of the resources provided, but are said to be location-independent in that they may be able to specify a location at a higher level of abstraction (e.g., country, state, or data center).

[0033] Rapid Elasticity: Capabilities can be provisioned quickly and elastically, sometimes automatically, scaled out quickly, released quickly, and scaled in quickly. To the consumer, the capabilities available for provisioning are often unlimited and can be purchased in any quantity at any time.

[0034] Measured Services: Cloud systems automatically control and optimize resource usage by using metering capabilities at some level of abstraction appropriate to the type of service (e.g., storage, processing, bandwidth, and active user accounts). Resource usage can be monitored, controlled, and reported, providing transparency to both providers and consumers of the services being used.

[0035] The service model is as follows:

[0036] Software as a Service (SaaS): The ability to offer consumers the ability to use a provider's applications running on a cloud infrastructure. The applications are accessible from a variety of client devices through a thin-client interface, such as a web browser (e.g., web-based email). The consumer does not manage or control the underlying cloud infrastructure, including the network, servers, operating systems, storage, or even individual application functions, with the possible exception of limited user-specific application configuration settings.

[0037] Platform as a Service (PaaS): The capability offered to consumers to deploy consumer-created or acquired applications, created using programming languages and tools supported by the provider, onto a cloud infrastructure. The consumer does not manage or control the underlying cloud infrastructure, including networks, servers, operating systems, or storage, but does have control over the deployed applications and, in some cases, the application-hosting environment configuration.

[0038] Infrastructure as a Service (IaaS): The capability offered to consumers to provision processing, storage, network, and other basic computing resources on which they can deploy and run any software, which may include operating systems and applications. The consumer does not manage or control the underlying cloud infrastructure, but does have control over the operating systems, storage, deployed applications, and in some cases, limited control over selecting network components (e.g., host firewalls).

[0039] The deployment models are as follows:

[0040] Private Cloud: Cloud infrastructure is operated exclusively for an organization. The cloud infrastructure may be managed by the organization or a third party, and may reside on-premises or off-premises.

[0041] Community Cloud: Cloud infrastructure is shared by several organizations and supports a specific community with common interests (e.g., mission, security requirements, policies, and compliance considerations). The cloud infrastructure may be managed by the organizations or a third party and may reside on-premises or off-premises.

[0042] Public Cloud: Cloud infrastructure is available to the general public or large industry groups and is owned by organizations that sell cloud services.

[0043] Hybrid Cloud: A cloud infrastructure is a blend of two or more clouds (private, community, or public) that remain unique entities but are brought together by standardized or proprietary technologies (e.g., cloud bursting for load balancing between clouds) that enable data and application portability.

[0044] A cloud computing environment is service oriented with a focus on statelessness, low coupling, modularity, and semantic interoperability. At the heart of cloud computing is an infrastructure comprising a network of interconnected nodes.

[0045] Referring now to FIG. 2 , an exemplary cloud computing environment 50 is illustrated. As illustrated, the cloud computing environment 50 includes one or more cloud computing nodes 10, with which local computing devices used by cloud consumers, such as a personal digital assistant (PDA) or mobile phone 54A, a desktop computer 54B, a laptop computer 54C, or an automotive computer system 54N, or combinations thereof, may communicate. The nodes 10 may communicate with each other. The nodes 10 may be physically or virtually grouped into one or more networks (not shown), such as a private cloud, community cloud, public cloud, or hybrid cloud, or combinations thereof, as described herein above. This enables the cloud computing environment 50 to provide infrastructure, platform, or software, or combinations thereof, as a service without the cloud consumer having to maintain resources on their local computing device. It is understood that the types of computing devices 54A-54N shown in FIG. 1 are intended to be illustrative only, and that cloud computing node 10 and cloud computing environment 50 can communicate with any type of computerized device (e.g., using a web browser) over any type of network or network-addressable connection or combination thereof.

[0046] Referring now to Figure 3, there is shown a set of functional abstraction layers provided by cloud computing environment 50 (Figure 2). It should be understood that the components, layers, and functions shown in Figure 3 are intended to be merely exemplary, and that embodiments of the present disclosure are not limited thereto. As shown, the following layers and corresponding functions are provided:

[0047] The hardware and software layer 60 includes hardware and software components. Examples of hardware components include a mainframe 61, a RISC (Reduced Instruction Set Computer) architecture-based server 62, a server 63, a blade server 64, storage devices 65, and network and networking components 66. In some embodiments, the software components include network application server software 67 and database software 68.

[0048] The virtualization layer 70 provides an abstraction layer from which the following examples of virtual entities can be provided: virtual servers 71; virtual storage 72; virtual networks 73, including, for example, virtual private networks; virtual applications and operating systems 74; and virtual clients 75.

[0049] In one example, management layer 80 may provide several functions, as described below. Resource provisioning 81 provides dynamic procurement of computing and other resources utilized to execute tasks within the cloud computing environment. Metering and pricing 82 provides cost tracking as resources are utilized within the cloud computing environment and billing or invoicing for the consumption of these resources. In one example, these resources may include application software licenses. Security provides identity verification for cloud consumers and tasks and protection for data and other resources. User portal 83 provides access to the cloud computing environment for consumers and system administrators. Service level management 84 provides allocation and management of cloud computing resources so that required service levels are met. Service level agreement (SLA) planning and fulfillment 85 provides pre-provisioning and procurement of cloud computing resources where future requirements are predicted according to SLAs.

[0050] Workload tier 90 provides examples of functions for which a cloud computing environment may be utilized. Examples of workloads and functions that may be provided from this tier include mapping and navigation 91; software development and lifecycle management 92; virtual classroom instruction delivery 93; data analytics processing 94; transaction processing 95; and one of data replication 96.

[0051] FIG. 4 illustrates an exemplary system 400 for performing data replication consistent with some embodiments of the present disclosure. System 400 may be a MADRS and may operate as a collection of computing nodes as a distributed file system to cache files on remote computing nodes while reducing network, processing, and memory resources of one or more computing nodes of the system. System 400 may operate as part of a cloud computing environment, such as cloud computing environment 50. System 400 may include multiple computing nodes, including at least first computing node 410 and second computing node 460; and network 450 communicatively connecting the multiple computing nodes. For illustrative purposes, only first computing node 410 and second computing node 460 are depicted. It should be understood that other configurations of system 400 are contemplated, including many computing nodes (e.g., three or more, eight or more) and are depicted for ease of depiction of two computing nodes.

[0052] The first computing node 410 may be a computer configured to host data and provide user access to the data. The first computing node 410 may be a computer system, such as computer system 100. The first computing node 410 may be configured to store and retrieve files from a first data store 420. The first data store 420 may be a database, file system, storage device, etc., for data local to the first computing node 410. For example, the data store 420 may be a file system running on a cluster of communicatively connected disk drives directed to the first computing node 410.

[0053] Similarly, second computing node 460 may be a computer configured to host data and provide user access to the data. Second computing node 460 may be a computer system, such as computer system 100. Second computing node 460 may be configured to store and retrieve files from second data store 470. Second data store 470 may be a database, file system, storage device, etc., for data local to second computing node 460. For example, data store 470 may be a file system cache running on a communicatively connected drive directed to second computing node 460.

[0054] The system 400 may operate to replicate data between the first computing node 410 and the remaining computing nodes of the plurality of computing nodes. The system 400 may operate as a MADRS to facilitate replication without performing certain data transfers, such as copying files from one computing node to another computing node of the plurality of computing nodes.

[0055] More specifically, a first computing node 410 may have a first local file (first file) 430. The first local file 430 may be located at a first path 432 in a first data store 420. The first local file 430 may be copied to a second local file 440 based on a first file operation 402. The first file operation 402 may be performed by a non-privileged operation or a privileged operation, such as a user application move, a kernel-level copy, etc. The first file operation 402 may create a second local file 440 at a second path 442 in the first data store 420. For example, if a file named "example.txt" is located at location "C:\directory1\," the first file operation 402 may copy the file to a second filename "new_file.txt" at location "E:\temp" in the data store 420.

[0056] The system 400 may detect a first file operation 402, determine that the first file operation is a copy operation, and update one or more metadata 434 of the first local file 430 based on the determination. The metadata 434 may be data associated with or otherwise related to the first local file 430. For example, the metadata 434 may be in the form of an "xattr" value. In another example, the metadata 434 may be in the form of one or more extended attributes. The metadata 434 may be in a predetermined format, and updating the metadata may involve creating one or more values in the predetermined format. For example, the metadata may be in the form of key-value pairs, and the update may be in the form of creating three key pairs: a first key equal to "command" and having a value equal to "copy"; a second key equal to "name" and having a value equal to "new_file.txt"; and a third key equal to "path" and having a value equal to "E:\temp\".

[0057] At step 404, metadata 434 may be transferred to a second computing node 460 of the plurality of computing nodes. In some embodiments, at step 404, a command may be generated based on metadata 434, and the command may be transferred to a second computing node 460 of the plurality of computing nodes. The second computing node 460 may already have a copy of a first local file (first copy file) 480. The first copy file 480 may be located at a first copy path 482. The first copy path 482 may correspond to the first path 432. After transfer at step 404 to another computing node of the plurality of computing nodes, including the second computing node 460, the updated metadata may be deleted or otherwise removed from the first local file 430.

[0058] In step 406, a second local file copy (second copy file) 490 may be created. The second copy file 490 may be created without any transfer of the first local file 430. For example, the second copy file 490 may be created only from the data contained in the first copy file 480 and the command received in step 404. The second copy file 490 may be created in the second computing node 460 in a second pass 492.

[0059] 5 illustrates an example method 500 for performing data replication consistent with some embodiments of the present disclosure. Method 500 may be performed by a data replication system, such as data replication system 400 illustrated in FIG. 4. Method 500 may be performed by a single computing node among multiple computing nodes of a distributed storage system. For example, method 500 may be performed by first computing node 410 of data replication system 400.

[0060] Starting at step 505, one or more computing resources of the data replication system are monitored at step 510. The data replication system resources may be the memory footprint, processor utilization, network activity, or other related metrics of the data replication system 400. Monitoring the data replication system resources may be based on one or more predetermined performance thresholds. The monitoring may include determining that the predetermined performance threshold is being adhered to. For example, the predetermined performance threshold may be the network of the data replication system 400, and the monitoring may include determining whether the bandwidth of the network 450 exceeds a predetermined value. In another example, one or more file input / output queues of multiple computing nodes, including the first node 410 and the second node 460, may be monitored, and it may be determined that the queue size is not exceeded. In yet another example, multiple processor utilizations or other loads may be monitored, and the predetermined performance threshold is a processor utilization of less than 65% of maximum processor load. In yet another example, the file size of a file is monitored, and the predetermined performance threshold is a file size greater than 130 megabytes. If the predetermined performance threshold is not achieved at step 520, then proceed "No" and method 500 ends at step 595.

[0061] If a predetermined performance threshold is achieved at step 520, proceeding to "Yes," one or more files on a predetermined computing node may be monitored for updates at step 530. Monitoring for updates may be performed by a first computing node of the plurality of computing nodes, e.g., first computing node 410 or second computing node 460 of system 400. Monitoring for updates may also be performed across the entire data replication system. While monitoring for updates to one or more files, an update directed to a particular file (e.g., a first file) may be detected at step 540. For example, the update may be a file creation, deletion, append, insert, remove, or other access. If the file update is not a copy at step 550, proceeding to "No," method 500 ends at step 595.

[0062] At step 550, if the file update is a copy, proceed to "Yes," and metadata associated with the file may be updated at step 560. The updated metadata may be based on the copy operation. For example, details of the copy operation, such as the path or location of the copied file, may be updated in the metadata. At step 570, the command for the updated metadata may be sent to other computing nodes of the plurality of computing nodes. For example, given system 400, if a file is copied on second computing node 460, then metadata about the copy may be sent to first computing node 410. After the metadata is sent at step 570, method 500 ends at step 595.

[0063] 6 illustrates an example method 600 for performing data replication consistent with some embodiments of the present disclosure. Method 600 may be performed by a data replication system, such as data replication system 400. Method 600 may be performed by a single computing node among multiple computing nodes of a distributed storage system. For example, method 600 may be performed by first computing node 410 of data replication system 400.

[0064] From start step 605, updates to one or more files on a computing node may be monitored at step 610. The monitoring may be performed by monitoring an input queue of the computing node, e.g., the input queue of the first computing node 410. The monitoring may be for updates received from additional computing nodes. For example, the first computing node 410 may be monitored for updates from other computing nodes of the plurality of computing nodes. At step 620, an update to a first file on the computing node may be detected. The update to the first file may be an update to the content of the file, e.g., new data from other computing nodes in a data replication system. The update to the first file may be an update to file metadata, e.g., an update to one or more values or extended attributes of the first file. The update to the first file may be in the form of a command directed to the first file, e.g., a delete command or a copy command. The update may be in a predetermined format, e.g., a multipart command with associated values. For example, the command may be in the form of a first value of "copy," a second value of "C:\directory1\file1.txt," and a third value of "C:\directory2\file1-copy.txt" included as values of metadata associated with the local file ".txt" on the receiving computing node. If, at step 630, the update is not a command or the update is not a metadata update, proceed "no" and method 600 may end at step 695.

[0065] At step 630, if the update is a command or metadata update, proceed "yes" and method 600 may continue by identifying the type of operation within the command or metadata. At step 650, if the command is not a copy operation, proceed "no" and method 600 may end at step 695. At step 650, if the command is a copy operation, proceed "yes" and a copy operation may then be performed on the computing node that detected the file update. The copy operation may be performed without receiving any actual file data from the input queue or from another of the computing nodes. For example, the copy may be created based on a copy command that was part of the detected file update. The created copy may be placed at a specified path on the local computing node. The specified path may be determined from the copy command included in the detected file update. After the copy is created locally on the receiving computing node, method 600 ends at step 695.

[0066] The present invention may be a system, method, computer program product, or computer program, or any combination thereof, at any level of technical detail that may be integrated. The computer program product may include one or more computer-readable storage media having computer-readable program instructions for causing a processor to perform aspects of the present invention.

[0067] The computer-readable storage medium can be a tangible device capable of holding and storing instructions for use by an instruction execution device. The computer-readable storage medium can be, for example, but not limited to, an electronic storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination thereof. A non-exhaustive list of more specific examples of computer-readable storage media includes the following: a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a static random access memory (SRAM), a portable compact disc read-only memory (CD-ROM), a digital versatile disk (DVD), a memory stick, a floppy disk, a mechanically encoded device such as a punch card or a ridge structure in a groove in which instructions are recorded, or any suitable combination thereof. As used herein, a computer-readable storage medium should not be construed as a transitory signal per se, such as an electric wave or other freely propagating electromagnetic wave, an electromagnetic wave propagating through a waveguide or other transmission medium (e.g., a light pulse passing through a fiber optic cable), or an electrical signal transmitted over an electrical wire.

[0068] The computer-readable program instructions described herein can be downloaded from a computer-readable storage medium to an individual computing device / processing device, or to an external computer or external storage device via a network, such as the Internet, a local area network, a wide area network, or a wireless network, or a combination thereof. The network may be comprised of copper transmission cables, optical fiber transmissions, wireless transmissions, routers, firewalls, switches, gateway computers, or edge servers, or a combination thereof. A network adapter card or network interface in each computing device / processing device receives the computer-readable program instructions from the network and transfers the computer-readable program instructions to the individual computing device / processing device for storage in a computer-readable storage medium.

[0069] Computer-readable program instructions for carrying out operations of the present invention may be either assembler instructions, instruction-set-architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, state-setting data, configuration data for an integrated circuit, or source or object code written in any combination of one or more programming languages, such as object-oriented programming languages, e.g., Smalltalk, C++, etc., or conventional procedural programming languages (e.g., the "C" programming language or similar programming languages). The computer-readable program instructions may execute entirely on the user's computer, partially on the user's computer, partially on the user's computer as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the latter scenario, the remote computer may be connected to the user's computer via any type of network, such as a local area network (LAN) or a wide area network (WAN), or the connection may be to an external computer (e.g., over the Internet using an Internet Service Provider). In some embodiments, electronic circuits, such as programmable logic circuits, field-programmable gate arrays (FPGAs), or programmable logic arrays (PLAs), may execute computer-readable program instructions by utilizing state information of the computer-readable program instructions to personalize the electronic circuitry to perform aspects of the invention.

[0070] Aspects of the present invention are described herein with reference to flowchart illustrations or block diagrams, or combinations thereof, of methods, apparatus (systems), and computer program products or computer programs according to embodiments of the invention. It will be understood that each block of the flowchart illustrations or block diagrams, or combinations thereof, and combinations of blocks in the flowchart illustrations or block diagrams, or combinations thereof, can be implemented by computer-readable program instructions.

[0071] These computer-readable program instructions may be provided to a processor of a computer or other programmable data processing apparatus, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate means for implementing the functions / acts specified in one or more blocks of the flowchart diagrams or block diagrams, or a combination thereof, to produce a machine. These computer-readable program instructions may also be stored in a computer-readable storage medium that can direct a computer-programmable data processing apparatus or other device, or a combination thereof, to function in a particular manner, such that a computer-readable storage medium having stored instructions includes an article of manufacture including instructions that implement aspects of the functions / acts specified in one or more blocks of the flowchart diagrams or block diagrams, or a combination thereof.

[0072] The computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device such that the instructions, which execute on the computer, other programmable data processing apparatus, or other device, implement the functions / acts identified in one or more blocks of the flowchart diagrams or block diagrams, or a combination thereof, to cause the computer, other programmable apparatus, or other device to perform a series of operational steps to generate a computer-implemented process.

[0073] The flowcharts and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products or computer programs according to various embodiments of the present invention. In this regard, each block in the flowcharts or block diagrams may represent a module, segment, or portion of instructions, which includes one or more executable instructions for implementing one or more specified logical functions. In some alternative implementations, the functions shown in the blocks may occur out of the order shown in the figures. For example, two blocks shown in succession may actually be accomplished as a single step performed simultaneously, substantially simultaneously, partially, or fully in a time-overlapping manner, depending on the functionality involved, or the blocks may be performed in the reverse order. It should be noted that each block of the block diagrams or flowchart diagrams or combinations thereof, and combinations of multiple blocks in the block diagrams or flowchart diagrams or combinations thereof, may be implemented by a special-purpose hardware-based system that performs the specified functions or operations, or may execute a combination of special-purpose hardware and computer instructions.

[0074] The description of various embodiments of the present disclosure has been presented for illustrative purposes and is not intended to be exhaustive or limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. The terms used in this specification have been selected to best explain the principles of the embodiments, practical applications, or technical improvements over technologies found in the market, or to enable those skilled in the art to understand the embodiments disclosed herein.

Claims

1. a data replication system monitoring updates to one or more files located on multiple computing nodes; the data replication system detecting, based on the monitoring, a first file operation related to a first file, where the first file is located on a first computing node of the plurality of computing nodes and the first file is related to a first copy file located on a second computing node of the plurality of computing nodes; the data replication system determining, based on the first file operation, that the first file operation is a copy operation of the first file to a second file, where the second file is located on the first computing node of the plurality of computing nodes; the data replication system forwarding a copy command associated with the first file to a second computing node of the plurality of computing nodes in response to the copy operation; and the data replication system creating a second copy file on a second computing node of the plurality of computing nodes in response to the copy command, wherein the second copy file is associated with the second file located on the first computing node of the plurality of computing nodes, and wherein the creating is based solely on information from the first copy file and within the copy command; A method comprising: forwarding the copy command, the data replication system updating metadata associated with the first file with the copy command in response to the copy operation; and the data replication system responsively forwarding the updated metadata for the first file to the second computing node of the plurality of computing nodes. Further comprising: The method.

2. The method of claim 1 , wherein the first file is not transferred to a second computing node of the plurality of computing nodes.

3. the monitoring is based on a predetermined performance threshold, wherein the predetermined performance threshold is based on a network of the data replication system; and The data replication system monitors the bandwidth of the network prior to the transmission. The method of claim 1 further comprising:

4. said monitoring being based on a predetermined performance threshold, wherein said predetermined performance threshold is based on a file input / output queue; and the data replication system monitoring a queue size of one or more of the plurality of computing nodes prior to the transferring; The method of claim 1 further comprising:

5. the monitoring is based on a predetermined performance threshold, wherein the predetermined performance threshold is based on processing load; and the data replication system monitoring the processing load of one or more of the plurality of computing nodes prior to the transferring; The method of claim 1 further comprising:

6. The method of claim 1 , wherein updating the metadata comprises updating one or more extended attributes associated with the first file.

7. The method of claim 1 , wherein updating the metadata comprises creating one or more values in a predetermined format.

8. The method of claim 7 , further comprising the data replication system deleting the one or more values in the metadata of the first file after creating the second copy file.

9. 2. The method of claim 1, wherein the second file is located at a specific path within the first computing node, and the second copy file is generated at a specific copy path located on the second computing node, wherein the specific copy path corresponds to the specific path.

10. The method of claim 9 , wherein the metadata includes the particular path.

11. The method of claim 1 , wherein the copy operation is performed by an application running on the first computing node.

12. The method of claim 11 , wherein updating the metadata associated with the first file is performed by a privileged application executing on the first computing node.

13. 12. The method of claim 11, wherein the copy operation is selected from the group consisting of copy and move.

14. 10. The method of claim 1, further comprising deleting the one or more values in the metadata of the first file after transferring the updated metadata to the second computing node.

15. monitoring updates to one or more files located on the first computing node includes monitoring an input queue of the second computing node of the plurality of computing nodes, and detecting updates to a first file includes receiving updated metadata for the first file in the input queue of the second computing node of the plurality of computing nodes. The method of claim 1.

16. The method of claim 15 , wherein the input queue of the second computing node of the plurality of computing nodes does not receive the first file.

17. The method of claim 1, wherein the metadata includes a first key equal to a copy command, a second key equal to a file name at the destination, and a third key equal to a path at the destination.

18. The method of claim 1, wherein the metadata includes a first key equal to a copy command, a second key equal to a file name at the destination, and a third key equal to a path at the destination.

19. 1. A data replication system, comprising: a memory containing one or more instructions; and a processor communicatively coupled to the memory wherein the processor is configured to perform the method of any one of claims 1 to 18 in response to reading the one or more instructions.

20. A computer program causing a processor to carry out the method according to any one of claims 1 to 18.

Images