Minimizing data transfer and storage usage on cloud object storage systems

Abstract

A method and system for minimizing data transfer and storage usage on a cloud object storage system is disclosed. First, a file is replicated from a production system to a cloud object storage system. Next, it is determined whether the number of hard links associated with the file is greater than one. If the number is greater than one, a special object is created on the cloud object storage system for the file and the file is associated with the special object. If a hard link is created on the production system in association with the file, the hard link is replicated from the production system to the cloud object storage system without replicating data associated with the file. Finally, a metadata reference to the special object is added to the hard link on the cloud object storage system.

Description

[Technical Field]

[0001] The present invention relates to a system and method for minimizing data transfer and storage usage on a cloud object storage system. [Background technology]

[0002] In a data replication environment, one site may be designated as the production site (also known as the "cache site") and another site (i.e., the "receiving site") may be designated as the disaster recovery site (also known as the "home site"). In certain configurations, the production site may capture all operations performed on its local file sets / file systems and maintain a local queue of operations on a first-come, first-served basis. This queue may be maintained on one node of the production site cluster. In some cases, this node is designated a gateway node that is configured to maintain the replication queue in memory. Application nodes in the production site cluster may generate remote procedure calls to update the gateway node with local operations on filesets as they are performed.

[0003] In some cases, replication may occur from one clustered file system to another clustered file system or to cloud object storage for backup and disaster recovery purposes. This may be achieved using standards-based protocols (e.g., NFS, S3, etc.), or using the actual file system protocol itself if the production and backup file systems are of the same type (e.g., IBM Spectrum Scale). Although the S3 protocol supported by cloud object storage is not POSIX compliant, all POSIX compliant operations must be replicated via the S3 protocol to reach the remote cloud object storage (i.e., the backup site).

[0004] One feature that helps maintain or minimize storage usage is the hard link feature. When a hard link is created on a traditional file system for a file, access to that file is provided through an entirely new path within the file system. When a new hard link is created for a file, the inode link count (i.e., nlink) associated with that file may be incremented. However, such hard links are typically not directly supported on cloud object storage servers. As a result, when a replication solution attempts to replicate a hard link operation to a cloud object storage server, the hard link may generate a new object at the cloud object storage site.

[0005] This can cause various problems. For example, this can consume additional storage space at the cloud object storage site, as objects (i.e., files) are stored multiple times at the cloud object storage site. This can also result in more data being transmitted from the production site to the cloud object storage site, as the same object may need to be transferred more than once. This problem is exacerbated when additional hard links exist for a file and / or the size of the file to which the hard links are created is larger. In a use case where an object is retrieved from the cloud object storage site to, for example, recreate a new file system at the production site and repopulate it with files from the cloud object storage site, it may be necessary to copy the replica of the file at the cloud object storage site back to the production site. This can result in more data being transmitted from the cloud object storage site to the production site than is required to recreate the file system. Summary of the Invention

[0006] The present invention was developed in response to the current state of the art, and particularly in response to problems and needs in the art not yet fully addressed by currently available systems and methods. Accordingly, the present invention has been developed to provide a system and method for minimizing data transfer and storage usage on a cloud object storage system. The features and advantages of the present invention will become more fully apparent from the following description and appended claims, or may be learned by the practice of the invention as set forth hereinafter.

[0007] Consistent with the foregoing, a method for minimizing data transfer and storage usage on a cloud object storage system is disclosed. In one embodiment, such a method includes replicating a file from a production system to a cloud object storage system. The method includes determining whether a number of hard links associated with the file is greater than one. If the number of hard links is greater than one, the method creates a special object for the file on the cloud object storage system and associates the file with the special object. When a hard link is created on the production system in association with the file, the method replicates the hard link from the production system to the cloud object storage system without replicating data associated with the file. A metadata reference to the special object is added to the hard link on the cloud object storage system.

[0008] In another embodiment of the present invention, a computer program product for minimizing data transfer and storage usage on a cloud object storage system is disclosed. The computer program product includes a non-transitory computer-readable storage medium having computer-usable program code embodied thereon. The computer-usable program code is configured to replicate a file from a production system to a cloud object storage system. The computer-usable program code determines whether a number of hard links associated with the file is greater than one. If the number of hard links is greater than one, the computer-usable program code creates a special object for the file on the cloud object storage system and associates the file with the special object. Upon creating a hard link on the production system in association with the file, the computer-usable program code replicates the hard link from the production system to the cloud object storage system without replicating data associated with the file. A metadata reference to the special object is added to the hard link on the cloud object storage system.

[0009] In another embodiment of the present invention, a system for minimizing data transfer and storage usage on a cloud object storage system includes at least one processor and at least one memory device operatively coupled to the at least one processor and storing instructions for execution on the at least one processor. The instructions cause the at least one processor to replicate a file from a production system to a cloud object storage system. The instructions further cause the at least one processor to determine whether a number of hard links associated with the file is greater than one. If the number of hard links is greater than one, the instructions create a special object for the file on the cloud object storage system and associate the file with the special object. Upon creating a hard link on the production system in association with the file, the instructions replicate the hard link from the production system to the cloud object storage system without replicating data associated with the file. A metadata reference to the special object is added to the hard link on the cloud object storage system.

[0010] In certain embodiments, the special object is created only for files with more than one hard link. In other or the same embodiments, determining whether the number of hard links associated with the file is greater than one includes reading a hard link count maintained for the file. Similarly, in certain embodiments, replicating the hard links from the production system to the cloud object storage system includes replicating the hard links without replicating data associated with the file. In certain embodiments, the special object is a hidden object. In certain embodiments, restoring the file and associated hard links from the cloud object storage system to the production system includes locating the file and hard link objects that reference the special object on the cloud object storage system. [Brief explanation of the drawings]

[0011] So that the advantages of the present invention can be readily understood, a more particular description of the invention briefly described above will be provided by reference to specific embodiments which are illustrated in the accompanying drawings, in which: The invention will be described and explained with additional specificity and detail by means of the accompanying drawings, with the understanding that these drawings depict only typical embodiments of the invention and therefore should not be considered as limiting the scope of the invention.

[0012] [Figure 1] 1 is a schematic block diagram illustrating data replication from a production system to a cloud object storage system;

[0013] [Figure 2] 1 is a schematic block diagram illustrating replication of files and hard links from a production system to a cloud object storage system;

[0014] [Figure 3]1 is a process flow diagram illustrating one embodiment of an improved technique for replicating files and hard links from a production system to a cloud object storage system;

[0015] [Figure 4] 3 is a schematic block diagram illustrating an example of an improved replication technique;

[0016] [Figure 5] 1 is a process flow diagram illustrating one embodiment of an improved technique for restoring files and hard links from a cloud object storage system to a production system;

[0017] [Figure 6] 6 is a schematic block diagram illustrating an example of the improved restoration technique of FIG. 5; and

[0018] [Figure 7] FIG. 1 is a schematic block diagram illustrating an example of a computing environment for executing computer code for performing any of the inventive methods disclosed herein. DETAILED DESCRIPTION OF THE INVENTION

[0019] It will be readily understood that the components of the present invention, as generally described and illustrated in the figures herein, could be arranged and designed in a wide variety of different configurations. Thus, as represented in the figures, the following more detailed description of embodiments of the present invention is not intended to limit the scope of the invention as claimed, but is merely representative of specific examples of embodiments presently contemplated by the present invention. The presently described embodiments can be best understood by reference to the drawings, wherein like parts are designated with like numerals throughout.

[0020] The present invention may be embodied as a system, method, and / or computer program product, which may include one or more computer-readable storage media having computer-readable program instructions for causing a processor to perform aspects of the present invention.

[0021] A computer-readable storage medium may be a tangible device that can retain and store instructions for use by an instruction execution device. The computer-readable storage medium may be, for example, but not limited to, an electronic storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination of the foregoing. A non-exhaustive list of more specific examples of computer-readable storage media includes the following: portable computer diskettes, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), static random access memory (SRAM), portable compact disk read-only memory (CD-ROM), digital versatile disk (DVD), memory sticks, floppy disks, mechanically encoded devices such as punch cards or ridge structures in grooves in which instructions are recorded, and any suitable combination of the foregoing. Computer-readable storage medium, as used herein, should not be construed as a transitory signal per se, such as an electric wave or other freely propagating electromagnetic wave, an electromagnetic wave propagating through a waveguide or other transmission medium (e.g., a light pulse passing through a fiber optic cable), or an electrical signal transmitted through a wire.

[0022] The computer-readable program instructions described herein may be downloaded from a computer-readable storage medium to each computing / processing device or to an external computer or external storage device via a network, such as the Internet, a local area network, a wide area network, and / or a wireless network. The network may comprise copper transmission cables, optical fiber transmissions, wireless transmissions, routers, firewalls, switches, gateway computers, and / or edge servers. A network adapter card or network interface within each computing / processing device receives the computer-readable program instructions from the network and transfers the computer-readable program instructions to a computer-readable storage medium within the respective computing / processing device for storage.

[0023] The computer readable program instructions for carrying out the operations of the present invention may be assembler instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, state setting data, or may be source code or object code written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Smalltalk or C++ or the like, and conventional procedural programming languages, such as the "C" programming language or similar programming languages.

[0024] The computer-readable program instructions may execute entirely on the user's computer as a standalone software package, partially on the user's computer, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the latter scenario, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or the connection may be to an external computer (e.g., via the Internet using an Internet Service Provider). In some embodiments, electronic circuits, including, for example, programmable logic circuits, field programmable gate arrays (FPGAs), or programmable logic arrays (PLAs), may execute the computer-readable program instructions to personalize the electronic circuit by utilizing state information of the computer-readable program instructions to perform aspects of the present invention.

[0025] Aspects of the present invention are described herein with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems) and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer-readable program instructions.

[0026] These computer-readable program instructions may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus to produce a machine, such that the instructions, executed by the processor of the computer or other programmable data processing apparatus, generate means for implementing the functions / acts specified in one or more blocks of the flowcharts and / or block diagrams. These computer-readable program instructions, which can direct a computer, programmable data processing apparatus, and / or other device to function in a particular manner, may be stored on a computer-readable storage medium, such that the computer-readable storage medium having the instructions stored thereon comprises an article of manufacture containing instructions that implement aspects of the functions / acts specified in one or more blocks of the flowcharts and / or block diagrams.

[0027] The computer-readable program instructions may also be loaded into a computer, other programmable data processing apparatus, or other device to create a computer-implemented process by causing the computer, other programmable apparatus, or other device to perform a series of operational steps, such that the instructions executing on the computer, other programmable apparatus, or other device implement the functions / operations specified in one or more blocks of the flowcharts and / or block diagrams.

[0028] Referring to FIG. 1, a schematic block diagram illustrating data replication from a production system 102 to a cloud object storage system 104 is shown. In certain embodiments, the production system 102 may also be referred to as a "cache site," and the cloud object storage system 104 may also be referred to as a "home site" and may function as a disaster recovery site. In certain configurations, the production system 102 may capture all operations performed on local filesets / file systems and maintain a local queue of operations on a first-come, first-served basis. This queue may be maintained on one node of the production site cluster. This node may be designated a gateway node 108 that maintains the replication queue in memory. An application node 106 in the production site cluster 102 may generate a remote procedure call to update the gateway node 108 with local operations on filesets as they are performed.

[0029] In some cases, replication may occur from one clustered file system to another clustered file system or to the cloud object storage system 104 for backup and disaster recovery purposes. This may be achieved using standards-based protocols (e.g., NFS, S3, etc.), or using the actual file system protocol itself if the production and backup file systems are of the same type (e.g., as may occur with IBM Spectrum Scale). Although the S3 protocol supported by cloud object storage is not POSIX compliant, all POSIX compliant operations must be replicated via the S3 protocol to reach the remote cloud object storage server (i.e., the backup site).

[0030] One feature that helps maintain or minimize storage space usage is the hard link feature. When a hard link is created on a traditional file system for a file, access to that file is provided through an entirely new path within the file system. When a new hard link is created for a file, the inode link count (i.e., nlink) associated with the file may be incremented. However, such hard links are typically not directly supported on the cloud object storage server 110. As a result, when a replication solution attempts to replicate the hard link operation to the cloud object storage server 110, the hard link may create a new object at the cloud object storage site 104.

[0031] This can cause various problems. For example, this can consume additional storage space at the cloud object storage site 104, as objects (i.e., files) are stored multiple times at the cloud object storage site 104. This can also result in more data being transmitted from the production system 102 to the cloud object storage system 104, as the same object may need to be transferred more than once. This problem is exacerbated when additional hard links exist for a file and / or the size of the file to which the hard links are created is larger. In a use case where an object is retrieved from the cloud object storage system 104, for example, to recreate a new file system on the production system 102 and repopulate the production system 102 with files from the cloud object storage system 104, it may be necessary to copy the replica of the file in the cloud object storage system 104 back to the production system 102. This can result in more data being transmitted from the cloud object storage system 104 to the production system 102 than would be necessary to recreate the file system.

[0032] FIG. 2 illustrates one example of this problem in more detail. As shown in FIG. 2, assume that multiple files and associated hard links are stored or created on production system 102. In the example shown, these files include "File1" with inode number "1000" and "File2" with inode number "1001." As further shown in FIG. 1, a hard link "HLink_File1" is stored or created for File1, and a hard link "HLink_File2" is stored or created for File2. These hard links allow File1 and File2 to be accessed through entirely new paths in the file system, which may minimize storage usage on production system 102 by eliminating or reducing the need to store multiple copies of files. When these files and hard links are created on application node 106, application node 106 may generate remote procedure calls that update gateway node 108 with these operations, as shown in FIG. 2. These files and hard links may then be replicated from the production system 102 to the cloud object storage system 104 for backup and / or disaster recovery purposes.

[0033] 2, an object may be created in the cloud object storage system 104 for File1, an object may be created for File2, an object may be created for HLink_File1, and an object may be created for HLink_File2. Because hard links may not be supported on the cloud object storage system 104, creating a hard link on the cloud object storage system 104 may generate a new object on the cloud object storage system 104 that contains all of the data from the linked file. Thus, assuming File1 consumes 10 megabytes (MB) of storage space on the production system 102, replicating File1 and its associated hard link, HLink_File1, to the cloud object storage system 104 may double the amount of data transferred to and consumed on the cloud object storage system 104. Similarly, replicating File2 and its associated hard link, HLink_File2, from the production system 102 to the cloud object storage system 104 may double the amount of data transferred to and consumed on the cloud object storage system 104, as shown in FIG. 2.

[0034] 3, a process flow diagram is shown illustrating one embodiment of an improved technique for replicating files and hard links from a production system 102 to a cloud object storage system 104. In such a method 300, when replicating files and hard links to the cloud object storage system 104, the amount of data transferred to and consumed on the cloud object storage system 104 may be minimized.

[0035] As shown, method 300 first determines (302) whether a file or hard links associated with the file need to be replicated from production system 102 to cloud object storage system 104. If so, method 300 determines (304) the number of hard links associated with the file. In particular embodiments, this may include reading the inode link count (nlink) maintained for the file. If the number of hard links is not greater than one, method 300 replicates (314) the file from production system 102 to cloud object storage system 104 in a conventional manner.

[0036] On the other hand, if the number of hard links is greater than one, the method 300 determines whether a special object was created on the cloud object storage system 104 for the file. As described in more detail below in connection with FIG. 4, for files with multiple hard links (e.g., an nlink value greater than one), a method according to the present invention may create a special object on the cloud object storage system 104 in association with the file. In particular embodiments, this special object may be created with the naming convention ".afm / + file inode number / + file generation number / + file creation time / ". An object beginning with ".afm" may be used to designate an internal object within the cloud object storage system 104. Such an internal object may be configured to carry all data related to the original file on the production system 102. In particular embodiments, the special object may store the number of hard links (e.g., the nlink value) associated with the file. This value is useful when restoring a file and associated hard links from the cloud object storage system 104 to the production system 102, as described in more detail below.

[0037] Once the special object is created, all other hard links may be stored on the cloud object storage system 104 as data-less objects (i.e., metadata-only objects). These hard link objects may include metadata references that point to the special object. In this manner, hard links may be independent of the data they point to or reference. As a result, operations such as creating, renaming, or removing performed on a hard link or the original file it references can be easily replicated to the cloud object storage system 104 without having to replicate any data associated with the file or modify or change the data associated with the file on the cloud object storage system 104. This preserves a significant amount of storage space on the cloud object storage system 104 because data duplication is eliminated or reduced. This, in turn, reduces costs associated with maintaining the cloud object storage system 104 and the data stored therein, and costs associated with transferring data from the production system 102 to the cloud object storage system 104.

[0038] In step 306, if the number of hard links is greater than one and a special object has not yet been created for the file or hard link, method 300 creates (308) a special object on cloud object storage system 104 and stores (310) the data associated with the hard link or file in the special object. If in step 306 a special object has already been created on cloud object storage system 104, method 300 provides (312) a metadata reference in the hard link object that points to the special object. In this way, no data is stored with the hard link object, and the data in the special object remains untouched. This allows the hard link object to remain independent of the data it carries or provides an access path to.

[0039] Figure 4 illustrates an example of the method 300 of Figure 3. This example builds on the example shown and discussed in connection with Figure 2. As shown in Figure 4, when files and associated hard links are replicated from the production system 102 to the cloud object storage system 104, the method 300 may determine whether the number of hard links associated with a particular file is greater than one. If the count is greater than one, the method 300 creates a special object for the file on the cloud object storage system 104.

[0040] For example, assuming that the hard link count of each of File1 and File2 is greater than 1, a first special object may be created on the cloud object storage system 104 for File1 (i.e., inode number "1000"), and a second special object may be created on the cloud object storage system 104 for File2 (i.e., inode number "1001"). Data associated with File1 may be stored in the special object associated with File1, and data associated with File2 may be stored in the special object associated with File2. Similarly, for File1 and HLink_File1, data-less objects (i.e., metadata-only objects) are created and stored on the cloud object storage system 104. These objects include metadata references that point to the special object for File1, as indicated by the arrows in FIG. 4. In a similar manner, data-less objects are created and stored on the cloud object storage system 104 for File2 and HLink_File2. These objects also include metadata references that point to the special object for File2, as indicated by the arrows in FIG. 4. 4, this technique results in two special objects, each storing 10 MB, on the cloud object storage system 104. Associated with each special object is a metadata-only file and hard link object (hence, tiny in size) that contains a metadata reference pointing to the special object.

[0041] Referring to Figure 5, a process flow diagram is shown illustrating one embodiment of an improved technique for restoring files and hard links from a cloud object storage system 104 to a production system 102. In some cases, it may be necessary to restore or rebuild one or more file systems on the production system 102 using data on the cloud object storage system 104. Figure 5 illustrates one embodiment of a method 500 for restoring files and hard links to the production system 102, assuming the files and hard links were created and stored on the cloud object storage system 104 using the techniques shown in Figures 3 and 4.

[0042] As shown, method 500 may first determine (502) whether it is time to restore files and / or hard links from production system 102 to cloud object storage system 104. This may correspond to a failure, maintenance, or other event on production system 102. When particular files and / or hard links are ready to be restored, method 500 may determine (506) the old inode numbers for the files or hard links to be restored. In particular embodiments, this may be accomplished by reading a table on gateway node 108 that stores a list of inode numbers (e.g., files and associated hard links) stored on cloud object storage system 104. In particular embodiments, the table may store one or more of the list of old inode numbers stored on cloud object storage system 104, a list of new inode numbers on production system 102 to which the old inode numbers will be restored, and a hard link count (e.g., nlink value) associated with each old inode number.

[0043] 5, once the old inode number associated with the file has been determined, method 500 may determine (506) a new inode number on production system 102 to which the old inode number will be restored. As described above, this may be accomplished by reading a table on gateway node 108. Once the old and new inode numbers have been determined, method 500 may retrieve (508) the special object and data associated with the old inode number from cloud object storage system 104. Method 500 may then store (510) the data associated with the file on production system 102 in a new or reconstructed file system using the new inode number.

[0044] The method 500 may then determine (512) whether the restored file has any associated hard links that also need to be restored. In particular embodiments, this may be accomplished by reading the hard link count (e.g., nlink value) stored for the old inode number in a table. These hard links may then be restored (514) from the cloud object storage system 104 to the production system 102 to provide an access path to the file with the new inode number in the new or restored file system.

[0045] Figure 6 is a schematic block diagram illustrating an example of the improved restore technique described in connection with Figure 5. This example builds on the example shown and discussed in connection with Figures 2 and 4. As shown, cloud object storage system 104 stores special objects for File1 and File2 along with hard link objects for each.

[0046] When a file system needs to be restored on the production system 102, a method 500 consistent with the present invention may determine the old inode numbers for files or hard links in the file system that pushed the objects to the cloud object storage system 104, and the new inode numbers in the new file system on the production system 102 where the files and hard links will be restored. As shown in Figure 6, in certain embodiments, a table 600 may be maintained on the production system 102, and more specifically on the gateway node 108, that maps old inode numbers to new inode numbers and hard link counts for each original inode number. This table 600 may be maintained during the bulk download of objects from the cloud object storage system 104 to be populated into the new file system on the production system 102.

[0047] When objects (including metadata-only objects, such as hardlink objects) to be repopulated into a new file system on production system 102 are transferred from cloud object storage system 104 to production system 102, table 600 may be referenced to determine a new inode number to associate the object with. In particular embodiments, an entry may be removed from table 600 and marked as complete when the number of hardlink objects (i.e., metadata-only hardlink objects) being transferred to production system 102 matches the hardlink count maintained in table 600. Thus, in the example provided in FIG. 6, the entry for File1 or File2 may be removed from table 600 and marked as complete when the number of hardlink objects being restored to production system 102 equals 2.

[0048] 7, various aspects of the present disclosure are described in text, flowcharts, block diagrams of computer systems, and / or block diagrams of machine logic included in embodiments of a computer program product (CPP). With respect to any flowchart, depending on the technology involved, operations may be performed in an order different from that shown in a given flowchart. For example, again depending on the technology involved, two operations shown in successive flowchart blocks may be performed in reverse order, as a single integrated step, simultaneously, or in an at least partially overlapping manner.

[0049] A computer program product embodiment ("CPP embodiment" or "CPP") is a term used in this disclosure to describe any set of one or more storage media (also referred to as "media"), collectively contained in one or more storage devices, that collectively contain machine-readable code corresponding to instructions and / or data for performing the computer operations specified in a given CPP claim. A "storage device" is any tangible device that can retain and store instructions for use by a computer processor. The computer-readable storage medium may be, but is not limited to, an electronic storage medium, a magnetic storage medium, an optical storage medium, an electromagnetic storage medium, a semiconductor storage medium, a mechanical storage medium, or any suitable combination of the foregoing. Some known types of storage devices that include these media include diskettes, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), static random access memory (SRAM), compact disk read-only memory (CD-ROM), digital versatile disk (DVD), memory stick, floppy disk, mechanically encoded devices (such as punch cards or pits / lands formed on a major surface of a disk), or any suitable combination of the foregoing. Computer-readable storage media, as the term is used in this disclosure, is not to be construed as storage of transient signals per se, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through waveguides, light pulses passing through fiber optic cables, electrical signals communicated through wires, and / or other transmission media. As will be appreciated by those skilled in the art, data typically moves at some infrequent time during the normal operation of a storage device, such as during access, defragmentation, or garbage collection, but this does not make the storage device transient, as data is not transient while stored.

[0050] Computing environment 700 includes an example of an environment for executing at least a portion of computer code involved in performing the inventive methodologies, such as code associated with data transfer and storage usage minimization module 750. In particular embodiments, data transfer and storage usage minimization module 750 may be used to perform any of the inventive methods or techniques described in connection with Figures 1 through 6. In addition to block 750, computing environment 700 includes, for example, a computer 701, a wide area network (WAN) 702, an end user device (EUD) 703, a remote server 704, a public cloud 705, and a private cloud 706. In this embodiment, computer 701 includes a set of processors 710 (including processing circuitry 720 and cache 721), a communications fabric 711, volatile memory 712, persistent storage 713 (including operating system 722 and block 750 as shown above), a set of peripheral devices 714 (including a set of user interface (UI) devices 723, storage 724, and a set of Internet of Things (IoT) sensors 725), and a network module 715. The remote server 704 includes a remote database 730. The public cloud 705 includes a gateway 740, a cloud orchestration module 741, a set of host physical machines 742, a set of virtual machines 743, and a set of containers 744.

[0051] Computer 701 may take the form of a desktop computer, a laptop computer, a tablet computer, a smartphone, a smartwatch or other wearable computer, a mainframe computer, a quantum computer, or any other form of computer or mobile device now known or later developed that is capable of executing programs, accessing a network, or querying a database, such as remote database 730. As is well understood in the art of computer technology, depending on the technology, execution of a computer-implemented method may be distributed among multiple computers and / or multiple locations. While this representation of computing environment 700 focuses the detailed discussion on a single computer, specifically computer 701, to keep the presentation as simple as possible. Computer 701 may be located within a cloud, even though it is not depicted within the cloud in FIG. 7 . However, computer 701 need not be within a cloud except to any extent expressly indicated.

[0052] Processor set 710 includes one or more computer processors of any type now known or later developed. Processing circuitry 720 may be distributed across multiple packages, e.g., multiple coordinated integrated circuit chips. Processing circuitry 720 may implement multiple processor threads and / or multiple processor cores. Cache 721 is memory located within the processor chip package and is typically used for data or code that should be available for fast access by threads or cores executing on processor set 710. Cache memory is typically organized in multiple levels depending on relative proximity to the processing circuitry. Alternatively, some or all of the caches for a processor set may be located “off-chip.” In some computing environments, processor set 710 may be designed to operate with qubits and perform quantum computing.

[0053] Computer-readable program instructions are typically loaded into computer 701 and a series of operational steps are executed by processor set 710 of computer 701, thereby performing a computer-implemented method, such that the instructions so executed instantiate the methods specified in the computer-implemented method flowcharts and / or descriptions contained herein (collectively referred to as "invention methods"). These computer-readable program instructions are stored in various types of computer-readable storage media, such as cache 721 and other storage media discussed below. The program instructions and associated data are accessed by processor set 710 to control and direct the execution of the invention methods. In computing environment 700, at least some of the instructions for performing the invention methods may be stored in block 750 in persistent storage 713.

[0054] Communications fabric 711 is the signal-conducting path that allows the various components of computer 701 to communicate with one another. Typically, this fabric is made up of switches and conductive paths, such as those that make up buses, bridges, physical input / output ports, and the like. Other types of signal communication paths, such as fiber optic and / or wireless communication paths, may also be used.

[0055] Volatile memory 712 may be any type of volatile memory now known or later developed. Examples include dynamic random access memory (RAM) or static RAM. Typically, volatile memory 712 is characterized by random access, although unless expressly indicated, this is not required. In computer 701, volatile memory 712 is located in a single package and is internal to computer 701, although alternatively or additionally, volatile memory may be distributed across multiple packages and / or located external to computer 701.

[0056] Persistent storage 713 may be any form of non-volatile computer storage, now known or later developed. The non-volatility of this storage means that stored data is retained regardless of whether power is supplied to computer 701 and / or power is supplied directly to persistent storage 713. Persistent storage 713 may be read-only memory (ROM), but typically at least a portion of persistent storage allows data to be written, data to be deleted, and data to be rewritten. Some well-known forms of persistent storage include magnetic disks and solid-state storage devices. Operating system 722 may take several forms, such as an open-source Portable Operating System Interface-type operating system employing a kernel or various known proprietary operating systems. The code contained in block 750 typically includes at least a portion of the computer code involved in performing the methods of the present invention.

[0057] Peripheral device set 714 includes the set of peripheral devices of computer 701. Data communication connections between peripheral devices and other components of computer 701 may be implemented in various ways, such as Bluetooth connections, near field communication (NFC) connections, connections made by cable (e.g., Universal Serial Bus (USB)-type cables), insertable connections (e.g., Secure Digital (SD) cards), connections made through local area communication networks, and even connections made through wide area networks such as the Internet. In various embodiments, UI device set 723 may include components such as display screens, speakers, microphones, wearable devices (e.g., goggles and smartwatches), keyboards, mice, printers, touchpads, game controllers, and haptic devices. Storage 724 may be external storage, such as an external hard drive, or insertable storage, such as an SD card. Storage 724 may be persistent and / or volatile. In some embodiments, storage 724 may take the form of a quantum computing storage device for storing data in the form of qubits. In embodiments where computer 701 needs to have a large amount of storage (e.g., computer 701 stores and manages a large database locally), this storage may be provided by a peripheral storage device designed to store extremely large amounts of data, such as a storage area network (SAN) shared by multiple geographically distributed computers. IoT sensor set 725 consists of sensors used in Internet of Things applications. For example, one sensor may be a thermometer and another sensor may be a motion detector.

[0058] The network module 715 is a collection of computer software, hardware, and firmware that enables the computer 701 to communicate with other computers over the WAN 702. The network module 715 may include hardware such as a modem or Wi-Fi signal transceiver, software for packetizing and / or depacketizing data for communication network transmission, and / or web browser software for communicating data over the Internet. In some embodiments, the network control and network forwarding functions of the network module 715 are performed on the same physical hardware device. In other embodiments (e.g., embodiments utilizing Software-Defined Networking (SDN)), the control and forwarding functions of the network module 715 are performed on physically separate devices, such that the control function manages several different network hardware devices. Computer-readable program instructions for carrying out the methods of the present invention may be downloaded to the computer 701 from an external computer or external storage device, typically through a network adapter card or network interface included in the network module 715.

[0059] WAN 702 is any wide area network (e.g., the Internet) capable of communicating computer data over non-local distances by any now known or later developed technology for communicating computer data. In some embodiments, WAN 702 may be replaced and / or supplemented by a local area network (LAN) designed to communicate data between devices located in a local area, such as a Wi-Fi network. WANs and / or LANs typically include copper transmission cables, optical fiber transmissions, wireless transmissions, and computer hardware such as routers, firewalls, switches, gateway computers, and edge servers.

[0060] The end-user device (EUD) 703 is any computer system used and controlled by an end user (e.g., a customer of the enterprise operating the computer 701) and may take any of the forms discussed above in connection with the computer 701. The EUD 703 typically receives useful and useful data from the operation of the computer 701. For example, in a hypothetical case where the computer 701 is designed to provide recommendations to the end user, the recommendations would typically be communicated to the EUD 703 from the network module 715 of the computer 701 over the WAN 702. In this manner, the EUD 703 can display or otherwise present the recommendations to the end user. In some embodiments, the EUD 703 may be a client device such as a thin client, a heavy client, a mainframe computer, a desktop computer, and the like.

[0061] Remote server 704 is any computer system that services at least some data and / or functionality to computer 701. Remote server 704 may be controlled and used by the same entity that operates computer 701. Remote server 704 represents a machine that collects and stores useful and useful data for use by other computers, such as computer 701. For example, in the hypothetical case where computer 701 is designed and programmed to provide recommendations based on historical data, this historical data may be provided to computer 701 from remote database 730 of remote server 704.

[0062] A public cloud 705 is any computer system available for use by multiple entities that provides on-demand availability of computer system resources and / or other computer capabilities, particularly data storage (cloud storage) and computing power, without direct, active management by users. Cloud computing typically leverages resource sharing to achieve coherence and economies of scale. Direct, active management of the computing resources of the public cloud 705 is performed by the computer hardware and / or software of a cloud orchestration module 741. The computing resources provided by the public cloud 705 are typically implemented by virtual computing environments running on various computers that comprise a host physical machine set 742, which is the universe of physical computers within and / or available to the public cloud 705. A virtual computing environment (VCE) typically takes the form of a virtual machine from a virtual machine set 743 and / or a container from a container set 744. It is understood that these VCEs may be stored as images and transferred between various physical machine hosts as images or after instantiation of the VCE. The cloud orchestration module 741 manages the transfer and storage of images, deploys new instantiations of VCE, and manages active instantiations of VCE deployments. The gateway 740 is a collection of computer software, hardware, and firmware that enables the public cloud 705 to communicate over the WAN 702.

[0063] We now provide some further explanation of virtualized computing environments (VCEs). A VCE can be stored as an "image." A new, active instance of a VCE can be instantiated from the image. Two well-known types of VCEs are virtual machines and containers. A container is a VCE that uses operating system-level virtualization. This refers to a feature of an operating system in which the kernel allows the existence of multiple isolated user space instances, called containers. These isolated user space instances typically function as actual computers from the perspective of programs running within them. A computer program running on a typical operating system can utilize all of the computer's resources, such as attached devices, files and folders, network shares, CPU power, and quantifiable hardware capabilities. However, a program running within a container can only use the contents of the container and of the devices allocated to the container; this feature is known as containerization.

[0064] Private cloud 706 is similar to public cloud 705, except that the computing resources are available only for use by a single enterprise. While private cloud 706 is shown in communication with WAN 702, in other embodiments, the private cloud may be completely disconnected from the Internet and accessible only through a local / private network. A hybrid cloud is a composite of multiple clouds of different types (e.g., private, community, or public cloud types), often implemented by multiple different vendors. While each of the multiple clouds remains a separate and distinct entity, the larger hybrid cloud architecture is bound together by standardized or proprietary technologies that enable orchestration, management, and / or data / application portability between the constituent clouds. In this embodiment, both public cloud 705 and private cloud 706 are part of a larger hybrid cloud.

[0065] Some examples will now be provided to further clarify various aspects of the present disclosure.

[0066] Example 1: A method for minimizing data transfer and storage usage on a cloud object storage system includes replicating a file from a production system to a cloud object storage system. The method includes determining whether a number of hard links associated with the file is greater than one. If the number of hard links is greater than one, the method includes creating a special object for the file on the cloud object storage system and associating the file with the special object. When a hard link is created on the production system in association with the file, the method includes replicating the hard link from the production system to the cloud object storage system without replicating data associated with the file. A metadata reference to the special object is added to the hard link on the cloud object storage system.

[0067] Example 2: A limitation of Example 1, where the special object is created only for files that have more than one hard link.

[0068] Example 3: A limitation of any of Examples 1 and 2, wherein determining whether the number of hard links associated with the file is greater than one comprises reading a hard link count maintained for the file.

[0069] Example 4: Any of the limitations of Examples 1 to 3, wherein adding the metadata reference from the hard link object to the special object comprises adding an object path to the hard link object.

[0070] Example 5: Any of the limitations of Examples 1 to 4, wherein replicating the hard link from the production system to the cloud object storage system comprises replicating the hard link without replicating data associated with the file.

[0071] Example 6: The limitation of any of Examples 1 to 5, wherein the special object is a hidden object.

[0072] Example 7: The limitation of any of Examples 1 to 6, further comprising locating the file and associated hard link objects that reference the special object on the cloud object storage system when restoring the file and associated hard link from the cloud object storage system to the production system.

[0073] Example 8: A system including one or more processors and one or more computer-readable storage media collectively storing program instructions configured, when executed by the processors, to cause the processors to perform a method according to any of Examples 1 to 7.

[0074] Example 9: A computer program product comprising: one or more computer-readable storage media; and program instructions collectively stored on the one or more computer-readable storage media, wherein the program instructions comprise instructions configured to cause one or more processors to perform a method according to any one of Examples 1 to 7.

Claims

1. 1. A method for minimizing data transfer and storage usage on a cloud object storage system, comprising: replicating files from the production system to a cloud object storage system; determining whether the number of hard links associated with the file is greater than one; creating a special object for the file on the cloud object storage system if the number of hard links is greater than one; associating said file with said special object; Upon creating a hard link on the production system in association with the file, replicating the hard link from the production system to the cloud object storage system and storing an associated hard link object on the cloud object storage system; and adding a metadata reference from the hardlink object to the special object on the cloud object storage system. A method for providing the above.

2. 2. The method of claim 1, wherein the special object is created only for files that have more than one hard link.

3. 2. The method of claim 1, wherein determining whether the number of hard links associated with the file is greater than one comprises reading a hard link count maintained for the file.

4. The method of claim 1 , wherein adding the metadata reference from the hard link object to the special object comprises adding an object path to the hard link object.

5. The method of claim 1 , wherein replicating the hard link from the production system to the cloud object storage system comprises replicating the hard link without replicating data associated with the file.

6. The method of claim 1 , wherein the special object is a hidden object.

7. 2. The method of claim 1, further comprising: locating, on the cloud object storage system, the file and hard link objects that reference the special object when restoring the file and associated hard link from the cloud object storage system to the production system.

8. 1. A computer program product for minimizing data transfer and storage usage on a cloud object storage system, the computer program product comprising: a non-transitory computer-readable storage medium having computer-usable program code embodied thereon, the computer-usable program code, when executed by at least one processor, performing: A procedure for replicating files from the production system to a cloud object storage system; determining whether the number of hard links associated with the file is greater than one; creating a special object for the file on the cloud object storage system if the number of hard links is greater than one; associating said file with said special object; Upon creating a hard link on the production system in association with the file, replicating the hard link from the production system to the cloud object storage system and storing an associated hard link object on the cloud object storage system; and adding a metadata reference from the hardlink object to the special object on the cloud object storage system; 2. A computer program product configured to execute

9. 9. The computer program product of claim 8, wherein the special object is created only for files that have more than one hard link.

10. 9. The computer program product of claim 8, wherein determining whether the number of hard links associated with the file is greater than one comprises reading a hard link count maintained for the file.

11. 9. The computer program product of claim 8, wherein adding the metadata reference from the hard link object to the special object comprises adding an object path to the hard link object.

12. 9. The computer program product of claim 8, wherein replicating the hard link from the production system to the cloud object storage system comprises replicating the hard link without replicating data associated with the file.

13. The computer program product of claim 8 , wherein the special object is a hidden object.

14. 9. The computer program product of claim 8, wherein the computer usable program code is further configured to perform a procedure for locating the file and hard link objects that reference the special object on the cloud object storage system when restoring the file and associated hard link from the cloud object storage system to the production system.

15. 1. A system for minimizing data transfer and storage usage on a cloud object storage system, comprising: at least one processor; and at least one memory device operatively coupled to the at least one processor and storing instructions for execution on the at least one processor, wherein the instructions cause the at least one processor to: Replicating files from the production system to a cloud object storage system; determining whether the number of hard links associated with the file is greater than one; creating a special object for the file on the cloud object storage system if the number of hard links is greater than one; associating said file with said special object; Upon creating a hard link on the production system in association with the file, replicating the hard link from the production system to the cloud object storage system and storing an associated hard link object on the cloud object storage system; and Adding a metadata reference from the hardlink object to the special object on the cloud object storage system. to carry out A system comprising:

16. 16. The system of claim 15, wherein the special objects are created only for files that have more than one hard link.

17. 16. The system of claim 15, wherein determining whether the number of hard links associated with the file is greater than one comprises reading a hard link count maintained for the file.

18. 16. The system of claim 15, wherein adding the metadata reference from the hard link object to the special object comprises adding an object path to the hard link object.

19. 16. The system of claim 15, wherein replicating the hard link from the production system to the cloud object storage system comprises replicating the hard link without replicating data associated with the file.

20. The system of claim 15 , wherein the special object is a hidden object.

Images