Object metadata conflict resolution

Abstract

In some examples, a first computing system receives a first metadata change made to metadata of a first instance of an object stored at a storage location of the first computing device. The first computing system associates a time of the first metadata change with the first metadata change. Based on receiving a copy of the second instance of the object from the second computing system, the first computing system determines that a second metadata change was made to the metadata of the second instance of the object. Further, the second instance of the object includes a time of the second metadata change. The first computing system performs at least one of resolving a conflict between the first metadata change and the second metadata change based at least on the comparison of the times or merging the first metadata change with the second metadata change into the metadata of the object.

Description

[Technical field]

[0001] The present disclosure relates to the field of data storage. [Background technology]

[0002] A data object, such as a file or other object type, may typically include object data and object metadata. For example, the object data includes the contents of the object. The object metadata may include information about the object data, such as the location of the object data in a file system, system-generated information about the object, user-generated information about the object, etc. Summary of the Invention [Problem to be solved by the invention]

[0003] When metadata replication is performed across multiple computing systems, such as to provide redundancy protection, the same metadata may exist on multiple different computing systems. In such a topology, it is possible that the same object metadata may be modified simultaneously by different users on different computing systems. When metadata is modified differently on different computing systems, this may result in data inconsistencies, unintended data loss, or other undesirable results. [Means for solving the problem]

[0004] In some implementations, a first computing device of a first computing system receives a first metadata change made to metadata of a first instance of an object stored at a storage location of the first computing device. The first computing device associates a time of the first metadata change with the first metadata change. Based on receiving a copy of the second instance of the object from a second computing device of a second computing system, the first computing device determines that a second metadata change has been made to metadata of the second instance of the object. Additionally, the second instance of the object includes a time of the second metadata change. In some cases, the first computing device may resolve a conflict between the first metadata change and the second metadata change based on at least comparing the time of the first metadata change to the time of the second metadata change. Additionally or alternatively, in some cases, the first computing device may merge the first metadata change with the second metadata change into the metadata for the object. [Brief description of the drawings]

[0005] The detailed description is now described with reference to the accompanying drawings, in which the left-most digit(s) of a reference number identifies the figure in which the reference number first appears, and the use of the same reference number in different figures indicates similar or identical items or features.

[0006] [Figure 1] FIG. 1 illustrates an example architecture for a system capable of replicating data among multiple computing systems while allowing merging of metadata changes made across the multiple computing systems according to some implementations.

[0007] [Diagram 2] FIG. 2 illustrates an example process for metadata conflict resolution according to some implementations herein.

[0008] [Diagram 3]FIG. 3 is a flow diagram illustrating an example process for detecting metadata inconsistencies according to some implementations.

[0009] [Figure 4] FIG. 4 is a flow diagram illustrating an example process for conflict resolution and merging metadata changes according to some implementations.

[0010] [Diagram 5] FIG. 5 is a flow diagram illustrating an example process for performing cleanup following conflict resolution and change merging according to some implementations.

[0011] [Figure 6] FIG. 6 illustrates an example of storing last update time and user information in association with metadata changes according to some implementations.

[0012] [Figure 7] FIG. 7 illustrates an example of merging metadata changes according to some implementations herein.

[0013] [Figure 8] FIG. 8 illustrates example selected components of an example computing system that may be used to implement some of the functionality of the systems described herein. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0014] Some implementations herein are directed to techniques and configurations for managing changes made to metadata in multiple different computing systems in a reliable manner, not only by tracking the time each metadata change was made, but also by determining the user role (e.g., user privileges) for the user who changed the metadata in each computing system. For example, by tracking the last modified time (LMT) and user role, such as user privileges, for changes to individual pieces of metadata in each of the computing systems, implementations herein can handle multiple metadata changes made across multiple computer systems to avoid situations where some metadata changes may be lost due to replication processes or the like. Techniques herein allow for the preservation of individual metadata changes associated with a single object made in different computing systems, and can persist changes made by different users at the same time (e.g., within the same replication cycle) in different computing systems without losing the individual changes.

[0015] Data loss due to incorrect conflict resolution is a very common and complex problem in object storage. For example, in an active-active replication topology, replication occurs in two directions between at least two computing systems. In an active-active replication topology, it is possible that the same object can be modified simultaneously in different computing systems by different users with the same or different privileges. However, implementations herein can resolve any conflicts in modifications and prevent data loss even when two or more computing systems are configured for replication in an active-active multi-way configuration.

[0016] Examples herein can maintain and merge metadata changes by tracking last update times and user permissions of changes to individual pieces of metadata, thereby enabling resolution of inconsistencies in metadata changes across, for example, clusters of computing devices, such as those that may be arranged in a ring topology or other replicated configuration. Additionally, implementations herein are not limited to resolving inconsistencies solely by checking the last update times of individual metadata, but may also apply preferences based on user roles (e.g., user permissions) associated with each metadata change.

[0017] In some examples herein, each computing system participating in the replication is responsible for detecting and tracking changes to the object's metadata and storing the changes back in the object so that they can be subsequently forwarded to another system in the next replication cycle. Each system may employ the same logic to compare the locally stored data with the remote side data forwarded in a replication cycle to merge the metadata changes at the individual metadata level so that the final merged object's metadata can include the changes from each side without losing the changes from each side. Furthermore, even in the case of a highly scalable model, it is still possible that multiple systems may enable changes to the object's metadata at the same time. In this situation, the role of the user who changed the metadata on each side may be used as a tiebreaker to resolve the conflict.

[0018] Merging metadata changes at the object level according to the implementations herein is advantageous in avoiding data loss across computing systems in both synchronous and asynchronous modes of replication. Furthermore, the solutions herein are highly scalable and can be used in N-dimensional replication systems. In addition, the solutions herein can also be used in heterogeneous environments, such as when data and metadata are stored on a primary disk drive or similar on one side, and data and metadata are stored on a cloud-based storage on a network on another side. Furthermore, although the amount of additional space used to store the additional information used to track the last update time of each piece of metadata may not be large in some instances, the additional information does not need to persist on each computing system for a long period of time. Conversely, the metadata change information can be deleted from the computing system after the information is used in conflict resolution and change merging.

[0019] In addition, the solutions herein work in situations where the metadata characteristics are heterogeneous, such as when an object on a first computing system has N types of metadata and an object on a second computing system has M types of metadata. In some examples herein, preferences may be granted to the user who performed the most recent change based on the user's role, thereby providing a relatively stable conflict resolution solution when the last update time for an object's metadata does not provide a resolution. Furthermore, while the techniques herein are capable of resolving conflicts in object metadata changes, the solutions herein may also be used to resolve conflicts between changes to different types of object data.

[0020] For purposes of explanation, some example implementations are described in the context of a multiple computing system environment performing data replication in two directions, however, the implementations herein are not limited to the particular examples provided and may be extended to other types of computing system architectures, other types of storage environments, other types of client configurations, other types of data, etc., as will be apparent to one of ordinary skill in the art in view of the disclosure herein. For example, while some examples are described in the context of resolving conflicts in object metadata, similar techniques may be applied to resolve object-level conflicts in the data itself for some types of objects.

[0021] As an example, consider an object as a ZIP file that contains two files, file a and file b. In a dual computing system replication topology, file a of the ZIP file can be modified on a first computing system and file b of the ZIP file can be modified on a second computing system. Thus, by maintaining and using the conflict resolution techniques herein on a per file system basis, any conflicts can be resolved and the final merged ZIP file can contain both the modified version of file a and the modified version of file b.

[0022] As another example, if the object is a type of text file, the last update time and user information of the user who made the change may be maintained for each line in the text file. Thus, multiple lines of a text file may be modified on different computing systems, and the conflict resolution and merging techniques herein may be used to merge modified text files from two or more computing systems to include the modified lines from each computing system without losing the changes made in any of the computing systems.

[0023] As yet another example, suppose the object is a type of image that contains pixels. Changes to the pixels may be tracked by maintaining a timestamp of each pixel change made in each computing system that maintains a copy of the image. The conflict resolution techniques herein may be used to maintain and resolve the changes in the pixels to generate a merged image following duplication.

[0024] As yet another example, the conflict resolution techniques herein may also be applied to subdivided object data, such as those with each subdivision containing a separate label. In this situation, each labeled subdivision of data modified at one of the individual computing systems may have a last update time and user information associated with its individual label. The modifications may then be merged and / or conflict resolution may be performed after replication, similar to the techniques described herein for metadata. Numerous other variations will be apparent to those of skill in the art having the benefit of the disclosure herein.

[0025] 1 illustrates an example architecture of a system 100 that may replicate data between multiple computing systems while enabling merging of metadata changes made in the multiple computing systems, according to some implementations. The system 100 includes multiple computing systems 102, each of which individually includes one or more computing devices 104. For example, a first computing system 102(1) includes one or more computing devices 104(1), a second computing system 102(2) includes one or more computing devices 104(2), and a third computing system 102(3) includes one or more computing devices 104(3). In some examples, the multiple computing devices 104 in each computing system 102 may form a cluster of computing devices or the like, although implementations are not limited to any particular configuration of computing systems 104.

[0026] Computing systems 102 may be located at individual sites 105 and may be in communication with each other over one or more networks 106. For example, a first computing system 102(1) may be physically located at a first site 105(1) at a first geographic location, a second computing system 102(2) may be physically located at a second site 105(2) at a second geographic location remote from the first geographic location, and computing system 102(3) may be physically located at a third site 105(3) remote from the first geographic location and the second geographic location. For example, the second and third geographic locations may be sufficiently distant from each other from the first geographic location, such as in another city, another state, another country, etc., such that a disaster affecting the first computing system 102(1) at the first site 105(1) is unlikely to affect the second computing system 102(2) at the second site 105(2) or the third computing system 102(3) at the third site 105(3), and vice versa. This topology thus provides redundancy in the data stored on computing systems 102(1)-102(3) to avoid catastrophic loss of data.

[0027] In some examples, computing system 102 can communicate with one or more client devices 108 over network 106. For example, computing system 102 may include access nodes, server nodes, management nodes, and / or other types of service nodes that provide storage services to client devices 108 to enable client devices 108 to store data, as well as to perform other management and control functions, as described further below. Client devices 108 may be any of a variety of types of computing devices, as described further below.

[0028] The one or more networks 106 may include any suitable network including a wide area network such as the Internet, a local area network (LAN) such as an intranet, a wireless network such as a cellular network, a local wireless network such as Wi-Fi, and / or a wired network including short-range wireless communication such as BLUETOOTH, Fibre Channel, optical fiber, Ethernet, or any other network, a direct wired connection, or any combination thereof. Thus, the one or more networks 106 may include both wired and / or wireless communication technologies. The components used for such communication may depend at least in part on the type of network, the selected environment, or both. Protocols for communicating over such networks are well known and may not be described in detail herein. As an example, the network 106 may include a private network, such as a LAN, a storage area network (SAN), or a Fibre Channel network. Additionally, the network 106 may include a public network or a combination of public and private networks, which may include the Internet. Implementations herein are not limited to any particular type of network such as the network 106.

[0029] In some examples, the computing systems 102 may each be configured to provide storage and data management services to client users 112 via client devices 108. As some non-limiting examples, the client users 112 may include users performing functions for a business, enterprise, organization, government entity, academic entity, or the like, and in some examples may include storage of very large amounts of data. Additionally, in some examples, the client users 112 may include administrative users who use the client devices 108 to manage one or more computing systems 102. However, implementations herein are not limited to any particular use or application of the system 100 and other example systems and configurations described herein. For example, in some examples, the client devices 108 may not be included, or may be an entirely different type of client device.

[0030] Each client device 108 may be any suitable type of computing device, such as a desktop, laptop, tablet computing device, mobile device, smartphone, wearable device, terminal, and / or any other type of computing device capable of transmitting data over a network. A client user 112 may be associated with a client device 108, such as through an individual user account, user login credentials, or the like. Further, a client device 108 may be configured to communicate with the computing system 102 through one or more networks 106, through a separate network, or through any other suitable type of communications connection. Numerous other variations will be apparent to those of ordinary skill in the art having the benefit of the disclosure herein.

[0031] In some implementations, each client device 108 may include a respective instance of a client application 114 that may run on the client device 108 to communicate with a web application 116 that may run on one or more computing devices 104 of the computing system 102, to transmit user data for storage on the network storage 104, and / or to receive stored data from the network storage 104, through data instructions such as write operations, read operations, delete operations, or the like. In some cases, the application 114 may include or run through a browser, while in other cases, the application 114 may include any other type of application having communication capabilities that enable communication with the web application 116 or other applications on the computing system 102 over one or more networks 106. Additionally, in the case of an administrative user, the web application 116 may provide remote management capabilities. Alternatively, in other examples, any of a number of other types of software configurations may be utilized to perform these functions, as would be apparent to one of ordinary skill in the art having the benefit of the disclosure herein.

[0032] Computing systems 102(1), 102(2), and 102(3) may each respectively execute instances of storage programs 120(1), 120(2), and 102(3), which may include, access, or otherwise execute instances of replication programs 122(1), 122(2), and 122(3) and conflict resolution programs 124(1), 124(2), and 124(3). For example, storage program 120 may provide access to object data 130 stored at each computing system 102, and may further manage metadata 132 associated with object data 130. Thus, storage program 120(1) may access and store object data 130(1) and manage associated metadata 132(1), storage program 120(2) may access and store object data 130(2) and manage associated metadata 132(2), and storage program 120(3) may access and store object data 130(3) and manage associated metadata 132(3). For example, storage program 120 may receive object data 130 from client device 108, store object data 130 on one or more storage devices of network storage 104, and / or retrieve and transmit requested object data 130 to client device 108, such as in response to a client read request or the like.

[0033] The storage program 120 may manage metadata 132 associated with the object data 130. For example, the metadata 132 may be maintained separately from the object data 130, such as in a metadata database or any other suitable data structure. In some cases, the storage program 120 may generate a portion of the metadata 132, such as to indicate a storage location of corresponding object data 130 for individual objects on individual computing systems 102.

[0034] In addition, storage program 120 may include, execute, access, or otherwise coexist with data replication program 122, which may be configured to perform data replication, as indicated by data replication link 144, between computing systems 102(1), 102(2), and 102(3). For example, data replication program 122 may configure computing systems 102(1)-102(3) to perform asynchronous or synchronous data replication between computing systems 102(1)-102(3). Furthermore, in some examples herein, computing systems 102 may utilize a time synchronization program, such as NTP Chrony, NTPD, or the like (not shown in FIG. 1 ), to maintain accurate synchronized time with respect to one another.

[0035] Some examples herein may utilize an active-active data replication configuration. For example, in an active-active replication topology, replication may occur in two directions on a single link between two computing systems. As an example, in a simple active-active replication topology, two computing systems may replicate the same object data 130 and metadata 132 to each other on an active-active link. The items to be replicated may be originally created on any computing system 102. The items may be read-written on the computing system 102 where they were created, and after they are replicated, they may be read-written on the other computing system 102. With active-active replication, client requests may be routed to any of the computing systems 102. All changes performed on each computing system 102, including both metadata changes and changes to object content, are replicated to the other computing system 102.

[0036] The example of FIG. 1 shows an active-active ring topology in which three computing systems 102(1)-102(3) are connected by active-active replication links, indicated at 144. With this topology, if one computing system 102 becomes unavailable (e.g., unexpectedly or due to scheduled maintenance), the other two computing systems 102 can still provide all required functionality. Furthermore, in the active-active ring topology, each computing system 102 has two active-active links connecting it to two other computing systems 102 such that all systems 102(1)-102(3) are linked in the form of a ring. Identical object data 130 and metadata 132 may be replicated on each data replication link 144. Additionally, changes made on one system may travel in both directions along the ring from that computing system until the changes reach the computing system where those changes have already occurred.

[0037] Further, in this example, a ring topology of three computing systems 102 is shown, but in other examples, there may be a greater number of computing systems 102 included in a ring topology. Additionally, while some implementations herein may be applied to an active-active replication topology, the implementations herein are not limited thereto. For example, some implementations may be applied to various other replication scenarios, such as many-to-one, one-to-many, chained replication, etc. Furthermore, these configurations may be combined to form complex replication topologies, as known in the art, and implementations of the conflict resolution techniques herein may be applied to these topologies as well.

[0038] In the illustrated example of FIG. 1 , a first client user 112(1) may make a metadata change 150 to metadata 132(1) of an object stored by a first computing system 102(1), and a second client user 112(2) may make a metadata change 152 to metadata 132(2) of the same object stored by a second computing system 102(2). Further, the changes 150, 152 to the metadata 132(1) and 132(2) of the same object may be inconsistent with each other. As an example, metadata change 150 may be made at time t 1 While the metadata change 152 removes a portion of the metadata from the object's metadata at time t 2 Suppose that a new metadata is added to an object in first computing system 102(1). The object with change 150 to metadata 132(1) is subsequently replicated to computing systems 102(2) and 102(3), and the object with change 152 to metadata 132(2) is subsequently replicated to computing systems 102(1) and 102(3). In a conventional replication process, only the version of the object with the most recent change to its metadata would survive the replication process. Thus, change 150 made in first computing system 102(1) may be lost because it was made earlier than later change 152 in second computing system 102(2). Examples herein may preserve both changes and may further provide for conflict resolution following replication, e.g., as described below in connection with FIGS. 2-5.

[0039] 2-5 include flow diagrams illustrating example processes according to some implementations. The processes are illustrated in logical flow diagrams as a collection of blocks, which represent sequences of operations, some or all of which may be implemented in hardware, software, or a combination thereof. In the context of software, the blocks may represent computer-executable instructions stored on one or more computer-readable media that, when executed by one or more processors, programs the processors to perform the described operations. Generally, computer-executable instructions include routines, programs, objects, components, data structures, and the like that perform particular functions or implement particular data types. The order in which the blocks are described should not be construed as limiting. Any number of the described blocks may be combined in any order and / or in parallel to implement a process or alternative processes, and not all of the blocks need be executed. For purposes of illustration, the processes are described herein with reference to the environments, frameworks, and systems described in the examples, although the processes may be implemented in a variety of other environments, frameworks, and systems.

[0040] FIG. 2 illustrates an example process 200 for metadata conflict resolution according to some implementations herein. The conflict resolution techniques herein may be utilized whenever N systems participate in N-side replication. In this example, conflict resolution program 124(1) may run on a first computing system 102(1) and conflict resolution program 124(2) may run simultaneously on a second computing system 102(2) to perform a series of operations to preserve and merge metadata changes 150 and 152 as described above in connection with the example of FIG. 1. Additionally, in this example, for ease of illustration, a third computing system 102(3) is not shown, although some of the operations may be performed on the third computing system 102(3), as described further below.

[0041] At 202, first computing system 102(1) may detect metadata changes 150 made by user 112(1) to metadata 132(1) stored by first computing system 102(1), as described above in connection with the example of FIG. 1. As an example, conflict resolution program 124(1) may run as a background process on computing system 102(1) and may detect the changes made to metadata 132(1) stored by first computing system 102(1). Upon detecting the changes to metadata 132(1), conflict resolution program 124(1) may record the exact time that metadata change 150 was made and may also record user information associated with the user who made metadata change 150.

[0042] At 204, second computing system 102(2) may also execute conflict resolution program 124(2) to detect metadata changes 152 made by user 112(2) to metadata 132(2) stored by second computing system 102(2), as described above in connection with the example of Figure 1. Conflict resolution program 124(2) may record the exact time that metadata changes 152 were made and user information for the user who made metadata changes 152.

[0043] At 206, the first computing system 102(1) may update metadata 132(1) of the particular object corresponding to the modified metadata to store the last modified time (LMT) and the user information of the user who last modified the particular object's respective metadata. For example, storing the last modified time and the user information directly with the metadata of the particular affected object (i.e., as part of the data object itself) simplifies tracking of the modification information and reduces overhead, such as in cases where it would be necessary to maintain the modification information separately outside of the object. This may also aid in replicating the modification information, such that the last modified time and the user information of the user who last modified the particular metadata are replicated with the object itself, thereby eliminating the need to separately track and transfer the modification information to transmit it to other computing systems 102(2) and 102(3) (not shown in FIG. 2). Prior to the next replication cycle, any number of modifications to the metadata may be made by one or more users, such as when modifications are made to different parts of the metadata, with each metadata change being maintained with its own modification information. On the other hand, if the same portion of the metadata is modified and then modified again by the same computing system, only the subsequent metadata changes to that portion of the metadata may be retained along with the last update time and user information since the previous changes to that particular portion of the metadata are overwritten and do not need to be replicated.

[0044] At 208, the second computing system 102(2) may update the metadata of the particular object that corresponds to the changed metadata to store the last update time and user information of the user who directly changed the metadata with the metadata of the particular affected object.

[0045] At 210, the first computing system 102(1) may replicate objects to the second computing system 102(2). For example, the computing systems 102(1)-102(3) may be configured by running individual instances of the replication program 122 to replicate updated objects to the other computing systems 102(1)-102(3) participating in the object-level replication via the computing systems 102(1)-102(3), such as based on a periodic trigger or other trigger mechanism. For example, instead of replicating all changed objects at regular or irregular time periods, the system may instead be configured to replicate any objects whose data or metadata has changed, such as within a prescribed time of the change being made. Regardless of the type of replication trigger, the replicated objects may include object data and object metadata for objects for which one or more metadata changes have been made. The replicated object metadata may further include the last update time and user information for individual users who made changes to the metadata of the object.

[0046] At 212, second computing system 102(2) replicates the object, including the changed object metadata and modification information (the LMT and user information for the user who modified the metadata), to first computing system 102(1). Additionally, while the example of FIG. 2 only shows replication from first computing system 102(1) to second computing system 102(2) and vice versa, bidirectional replication also occurs between first and second computing systems 102(1), 102(2) and third computing system 102(3). Thus, third computing system 102(3) (not shown in FIG. 2) also receives objects with changed metadata and respective modification information from each of first computing system 102(1) and second computing system 102(2).

[0047] At 214, the first computing system 102(1) may resolve the conflict in the changes to the metadata, if a conflict exists, such as based on a comparison of the last update times received from the second computing system 102(2) compared to the last update times recorded by the first computing system 102(1) of the metadata changes made at the first computing system 102(1). For example, the conflict resolution program 124(1) of the first computing system 102(1) may select the most recent last update time as the metadata change to implement to resolve the conflict. If the last update times are identical, the conflict resolution program 124(1) may refer to the user information of the user who made the change to determine which user may have higher privileges with respect to the object, and may select the change made by the user with higher privileges or other higher priority user role. As an example, a first metadata change 150 changes the user privileges of the object to a first setting, and a second metadata change 152 changes the user privileges to a second setting. In this case, the last update time may be applied to select the second setting of user privileges. Additionally, for some types of changes, the role of the user who made the change may override conflict resolution based on the last update time in cases such as when the first change prohibits subsequent changes by other users. Further, other existing conflict resolution rules may be applied depending on the type of metadata changed, such as when both metadata changes were to change the object's retention period, and then the longest retention period may be applied to resolve the conflict. Additionally, in cases where the changes to the metadata are not conflicting, for example, one metadata change is to the user's permissions to access the object and the other metadata change is to the object's retention period, the changes from both sides are merged and both metadata changes from both computing systems are retained as a single instance of the object.

[0048] At 216, concurrent with block 214, the second computing system 102(2) may resolve inconsistencies in the metadata, if any, based on a comparison of the last update times received from the first computing system 102(1) compared to the last update times recorded by the second computing system 102(2) of the metadata changes made at the second computing system 102(2). The third computing system 102(3) may perform similar inconsistency resolution actions. Thus, following the replication, each computing system 102(1)-102(3) participating in the replication now has access to both the last update times of each metadata change 150, 152 and the corresponding user information regarding the user who last modified each metadata change. Thus, each computing system 102(1)-102(3) may resolve any inconsistencies at the level of each metadata change in parallel with the other systems 102(1)-102(3) by comparing each of the last update times of each metadata change and, if necessary, the user information. For example, the conflict resolution program 124 in each computing system 102 may select the most recent last update time as the change to implement to resolve the direct conflict. If the last update times are the same, the conflict resolution program 124 may refer to the user information of the user who made the change to determine which user may have higher authority over the object, and may select the change made by the user with higher authority or other higher priority user role. In addition, for some types of changes, the role of the user who made the change may override the conflict resolution based on the last update time, such as in cases where the first change prohibits subsequent changes by other users. Furthermore, for some types of conflicts, other existing conflict resolution rules may be applied. In addition, in cases where the changes to the metadata are not contradictory to each other, the changes from both sides are merged and both metadata changes are kept in a single updated object.

[0049] At 218, following conflict resolution, the first computing system 102(1) may locally merge changes to the object metadata based on the conflict resolution decision. For example, metadata changes from both computing systems 102(1) and 102(2) may be merged and retained unless they directly conflict with each other, as described above.

[0050] At 220, following conflict resolution, the second computing system 102(2) may locally merge changes to the object metadata based on the conflict resolution decision. For example, metadata changes from both computing systems 102(1) and 102(2) may be merged and retained unless they directly conflict with each other, as described above. A third computing system 102(3) may also perform this function.

[0051] At 222, the first computing system 102(1) may replicate the merged object back to the second computing system 102(2) and then to the third computing system 102(3) as part of the next replication cycle, as well as receive the merged object from the third computing system 102(3). For example, the merged object may be merged based on metadata changes from each of computing systems 102(1) and 102(2) and may be stored locally by each system.

[0052] At 224, the second computing system 102(2) may replicate the merged object back to the first computing system 102(1) as part of the next replication cycle, and further replicates the merged object to, as well as receives, the merged object from, the third computing system 102(3).

[0053] At 226, the first computing system 102(1) may delete any modification information (LMT and user information) that was stored and / or used for conflict resolution. For example, there is no longer any need to store this information since all of the computing systems 102 have performed their own conflict resolution and merging processes and any conflicts have been resolved. Thus, any previously stored modification information, including LMT and user information, of the object may be locally deleted to reduce the overall size of the object on each computing system 102.

[0054] At 228, the second computing system 102(2) may delete any change information (LMT and user information) that was saved and / or used for conflict resolution. The third computing system 102(3) may perform a similar function.

[0055] 3 is a flow diagram illustrating an example process 300 for detecting metadata conflicts according to some implementations. In some cases, the process 300 may be performed at least in part by one or more computing systems 102, such as by one or more computing devices 104 of the computing system 102 executing the conflict resolution program 124 and the replication program 122.

[0056] At 302, the computing system may monitor for changes to the metadata initiated by a user. For example, when the metadata is maintained in a metadata database or other data structure, the computing system may monitor for any changes to the metadata in the data structure.

[0057] At 304, the computing system may determine whether a user has initiated a change to the object's metadata. If the determination is true, the process proceeds to 306. If the determination is not true, the process returns to 302 to continue monitoring.

[0058] At 306, when the system detects that a user has changed the metadata of an object, the computing system may determine the time the change was made and user information for the user who initiated the change. Examples of user information may include user permissions or other user role information, such as a priority level according to the user's role.

[0059] At 308, the computing system may store the time of the change as a last-update time and the user information in metadata associated with the target object corresponding to the changed metadata. If a previous last-update time is already stored in the metadata associated with the target object, the system may replace the last-update time with the new time as the last-update time.

[0060] At 310, the computing system may determine whether the object is ready to replicate in the next replication cycle. If the determination is true, the process proceeds to 312. If the determination is not true, the process proceeds to 302 to monitor for any further changes to the metadata. As mentioned above, in some cases replication may occur periodically, while in other examples replication may occur when a change to the object is detected locally at the computing system. Furthermore, in some examples, when replication is triggered at one of the computing systems, other computing systems participating in replication with that computing system may also be triggered to perform replication of any objects that have received changes since the last replication cycle. Implementations herein are not limited to any particular type of replication trigger or replication cycle.

[0061] At 312, the computing system may replicate the object to other computing systems participating in the replication and may include the object metadata with the replicated object along with the last update time and user information for the user who modified the metadata.

[0062] At 314, the computing system may determine whether the object was successfully sent to other systems during replication. If the determination is not true, the process proceeds to 316. If the determination is true, the process returns to 302 to monitor for new changes to the metadata made by the user.

[0063] At 316, if the object is not successfully transmitted to the other systems participating in the replication, the computing system may retry transmitting the object to the other systems.

[0064] 4 is a flow diagram illustrating an example process 400 for conflict resolution and merging of metadata changes according to some implementations. In some examples, process 400 may be performed by one or more computing devices 104 of computing system 102, such as by execution of conflict resolution program 124 and replication program 122.

[0065] At 402, a computing system may monitor for objects received during replication from other computing systems.

[0066] At 404, the computing system may determine whether an object was received through replication that has metadata changes that are inconsistent with metadata changes to stored metadata already stored in the computing system, or, if no inconsistencies exist, whether the received metadata changes can be merged with metadata changes made to the object already stored in the computing system. If these determinations are true, the process proceeds to 406. If these determinations are not true, the process proceeds to 402 to continue monitoring for objects received during replication.

[0067] At 406, if a conflict exists, the computing system may attempt to resolve the conflict based at least in part on a comparison of the last update times of the received and stored object metadata changes. For example, the computing system may determine the latest time between the received and stored last update times for a metadata change to the object. Based on the determination of the latest last update time, in some cases, the computing system may select the metadata change associated with the latest last update time as the metadata change to be implemented. Additionally, for some types of metadata changes, the role of the user who made the change may override conflict resolution based on the last update time, such as in cases where the first change prohibits subsequent changes by other users. In this situation, the role of the individual user who made the metadata change may be used to resolve the conflict in the metadata changes, or alternatively, other conflict resolution rules may be applied, as described elsewhere herein.

[0068] At 408, the computing system may determine whether the conflict was resolved using a comparison of the last update times. If the determination is true, the process proceeds to 412. If the determination is not true, the process proceeds to 410.

[0069] At 410, if the conflict is not resolved based on a comparison of the last update times, the computing system may resolve the conflict based on the user information by determining the authority associated with the object of the user who made the change to the object metadata. As an example, if the last update times are identical, or if the conflict cannot otherwise be resolved based on a comparison of the last update times, the computing system may select the metadata change made by the user with a higher level of user authority (e.g., corresponding to a higher priority role) for the particular data object. Additionally, in some cases, other rules for resolving metadata conflicts may be applied, such as in cases where the retention period has been changed, in which case the longest retention period is applied. Thus, various other conflict resolution rules may be applied in certain situations, as known in the art.

[0070] At 412, when non-conflicting metadata changes exist and / or when any conflicts have been resolved, the computing systems may merge the non-conflicting metadata changes, if any. For example, in some cases, two or more metadata changes may be non-conflicting, in which case the metadata changes from the two or more computing systems may be retained or all non-conflicting metadata changes may be merged into a single instance of the object.

[0071] At 414, the computing system may determine whether the object is ready to be replicated in the next replication cycle. For example, the computing system may wait until the next replication cycle before sending the object with the merged metadata to another computing system. If it is ready, the process proceeds to 416. If it is not ready, the process proceeds to 418.

[0072] At 416, the computing system may replicate the object including the merged object metadata to other computing systems participating in the replication.

[0073] At 418, if it is not yet time for the next replication cycle, the system may wait until the next replication cycle arrives.

[0074] 5 is a flow diagram illustrating an example process 500 for performing cleanup following conflict resolution and merging of changes according to some implementations. In some examples, process 500 may be performed by one or more computing devices of computing system 102, such as by execution of conflict resolution program 124 and replication program 122.

[0075] At 502, a computing system may monitor for objects received through replication from other computing systems.

[0076] At 504, the computing system may determine whether an object has been received that corresponds to a metadata conflict or other multi-system metadata data changes that have been locally resolved or merged, or both. For example, if the computing system has locally resolved a metadata conflict or otherwise merged metadata changes made in multiple computing systems for a particular object, the computing system may determine whether the same object has been received through replication from another computing system. If the determination is true, the process proceeds to 506. If the determination is not true, the process returns to 502 to monitor for additional objects received through replication.

[0077] At 506, the computing system may also determine whether conflicts or other metadata changes have been resolved and / or whether metadata changes have been merged into the metadata of the received object that corresponds to the locally resolved object. If the determination is true, the process proceeds to 510. If the determination is not true, the process proceeds to 508.

[0078] At 508, the computing system may wait until the next replication cycle to observe whether other computing systems resolve the multi-system changes in the metadata.

[0079] At 510, the computing system may determine whether the object is ready for change cleanup. For example, when the computing system determines that any conflicts have been resolved and / or metadata changes have been merged by all of the computing systems, the computing system may proceed with local change cleanup. If the determination is true, the process proceeds to 512. If the determination is not true, the process proceeds to 508 to wait until the next replication cycle.

[0080] At 512, the computing system may remove the local modification information, including the last update time and corresponding user information. For example, the location in the metadata of the object may be cleared of this information.

[0081] At 514, the computing system may locally update the object based on the received copy, e.g., if any other changes were made to the object, these changes may be updated in the local object.

[0082] The example processes described herein are merely example processes provided for illustrative purposes. Many other variations will be apparent to those skilled in the art in view of the disclosure herein. In addition, the disclosure herein describes some examples of suitable frameworks, architectures, and environments for carrying out the processes, but the implementations herein are not limited to the specific examples shown and described. Furthermore, the disclosure provides various example implementations as described and shown in the drawings. However, the disclosure is not limited to the implementations described and shown herein, and may extend to other implementations as understood or known by those skilled in the art.

[0083] FIG. 6 illustrates an example 600 of storing last update times and user information for metadata changes according to some implementations. In this example, assume that a user makes a first metadata change 604(1) to an object 602, as shown at 601. Based on detecting this, the computing system may associate with the first metadata change 604(1) a last update time 606(1) indicating the time when the first metadata change 604(1) was made by the user. In addition, the computing system may also associate with the first metadata change 604(1) user information 608(1) of the user who made the first metadata change 604(1). By way of example, the user information 608(1) may include at least the role of the user who made the first metadata change 604(1), which may include the user's priority and / or user privileges with respect to the object 602. Thus, the last update time 606(1) and the user information 608(1) may be stored as part of the metadata of the object 602, and thus may form part of the object 602.

[0084] Suppose the same or a different user subsequently makes a second metadata change 604(2) to some other metadata of the object 602 at the same computing system, as shown at 610. Upon detection of the second metadata change 604(2), the computing system associates a last update time 606(2) with the second metadata change 604(2). The computing system also associates with the second metadata change 604(2) the user information 608(2) of the user who made the second metadata change 604(2). As mentioned above, any number of changes to the metadata may be made by one or more users at the same computing system prior to the next replication cycle. For example, when the changes are made to different portions of the metadata, each metadata change is retained in the object 602 along with its own change information (LMT and user information). On the other hand, if the same portion of metadata is changed, for example in the first computing system 102(1) and then changed again in the first computing system 102(1), then only the later metadata changes to that portion of metadata will be retained in the object 602 along with the last update time and user information since the previous changes to that particular portion of metadata will be overwritten and do not need to be replicated.

[0085] Upon replication of object 602 to another computing system, both first metadata change 604(1) and second metadata change 604(2) along with their respective change information (i.e., LMT 606(1), user info 608(1), LMT 606(2), and user info 608(2)) may be replicated to the other computing systems participating in the replication topology. Also, in some examples, if another user makes a third metadata change to the instance of object 602 stored in the second computing system (not shown in FIG. 6 ), after replication, both first metadata change 604(1) and second metadata change 604(2) may be merged with the third metadata change made in the second computing system if they do not conflict with the third metadata change, or alternatively, if a conflict exists, the conflict may be resolved as described above.

[0086] 7 illustrates an example 700 of metadata change merging according to some implementations of the present disclosure. In some examples, first computing system 102(1) corresponds to first computing system 102(1) described above with respect to Figures 1 and 2, and second computing system 102(2) corresponds to second computing system 102(2) described above with respect to Figures 1 and 2 (computing system 102(3) has been omitted for clarity of illustration, or may be omitted entirely in this example).

[0087] In 701, before the replication is performed, it is assumed that a first computing system 102(1) includes a first instance of an object 702(1) and a second computing system 102(2) includes a second instance of an object 702(2). Further, it is assumed that a first user makes a first metadata change 704(1) to the metadata of the first instance 702(1) of object 702(1) at the first computing system 102(1) and a second user makes a second metadata change 704(2) to the metadata of the second instance of object 702(2) at the second computing system 102(2). Based on the metadata change 704(1), the first computing system 102(1) associates a last update time 706(1) and user information 708(1) with the first metadata change 704(1). Additionally, based on the metadata change 704(2), the second computing system 102(2) associates a last update time 706(2) and user information 708(2) with the second metadata change 704(2).

[0088] Subsequently, data replication 710 may be performed to replicate the first instance of object 702(1) to second computing system 102(2), and data replication 712 may be performed to replicate the second instance of object 702(2) to first computing system 102(1). As shown at 714, each of first computing system 102(1) and second computing system 102(2) may execute its respective conflict resolution program 124(1) or 124(2), respectively, to resolve conflicts, if any, between first metadata change 704(1) and second metadata change 704(2). In this example, it is assumed that there are no conflicts between first metadata change 704(1) and second metadata change 704(2). For example, first metadata change 704(1) may be a change to user permissions for object 702, and second metadata change 704(2) may be a change to the retention period for object 702. In this case, because no conflict resolution is required, the first metadata change 704(1) and the second metadata change 704(2) are merged into a single object 702 in each computing system 102(1) and 102(2) such that the metadata changes from both computing systems 102(1) and 102(2) are maintained in a single instance of objects 702(1) and 702(2) stored in each system 102(1) and 102(2), respectively.

[0089] Furthermore, in some instances, as described above, there may be more than two metadata changes, some of which may conflict with metadata changes made in other computing systems, and some of which are not conflicting. In this situation, conflicts between two or more metadata changes may be resolved, such as by using last update time, user role, or other conflict resolution rules. As described above, following resolution of any conflicting changes, the resolved metadata changes and any other non-conflicting metadata changes are merged into a single object in the respective computing systems.

[0090] FIG. 8 illustrates selected example components of an example computing system 102 that may be used to implement some of the system functionality described herein. The computing system 102 includes one or more computing devices 104, which may include one or more servers or other types of computing devices that may be implemented in any number of ways. In addition, in some examples, the computing device 104 may include or be in communication with one or more storage systems, storage controllers, network-attached storage, storage arrays, storage area networks, or the like. For example, in the case of a server, the programs, other functional components, and data may be implemented on a single server, a cluster of servers, a server farm or data center, a cloud-hosted computing service, etc., although other computer architectures may be used in addition or instead. Multiple computing devices 104 may be located together or separately and may be organized, for example, as virtual servers, server banks, and / or server farms. The functionality described may be provided by a server of a single entity or enterprise, or may be provided by servers and / or services of multiple different entities or enterprises.

[0091] In the illustrated example, the computing device 104 may include or be associated with one or more processors 802, one or more computer-readable media 804, and one or more communication interfaces 806. Each processor 802 may be a single processing unit or several processing units and may include single or multiple computing units or multiple processing cores. The processor 802 may be implemented as one or more central processing units, microprocessors, microcomputers, microcontrollers, digital signal processors, image processing units, system-on-chip processors, state machines, logic circuits, and / or any device that manipulates signals based on operational instructions. By way of example, the processor 802 may include one or more hardware processors and / or any suitable type of logic circuitry specifically programmed or configured to execute the algorithms and processes described herein. The processor 802 may be configured to fetch and execute computer-readable instructions stored in the computer-readable media 804, which may program the processor 802 to perform the functions described herein.

[0092] The computer readable medium 804 may include volatile and non-volatile memory and / or removable and non-removable media implemented in any type of technology for storage of information, such as computer readable instructions, data structures, program modules, or other data. For example, the computer readable medium 804 may include, but is not limited to, RAM, ROM, EEPROM, flash memory or other memory technology, optical storage, solid-state storage, magnetic tape, and magnetic disk storage, or any other medium that can be used to store the desired information and that can be accessed by a computing device. Furthermore, in some examples, the computer readable medium 804 includes a networked storage system, which may include a storage array, network attached storage, storage area network, cloud storage, and the like, as described elsewhere herein.

[0093] Depending on the configuration of the computing system 102, the computer readable medium 804 may be a tangible non-transitory medium to the extent that non-transitory computer readable medium, when referred to, excludes media such as energy, carrier signals, electromagnetic waves, and / or the signals themselves. In some cases, the computer readable medium 804 may be co-located with the computing system 102, while in other instances, the computer readable medium 804 may be partially remote from the computing system 102.

[0094] The computer readable medium 804 may be used to store any number of functional components executable by the processor 802. In many implementations, these functional components comprise instructions or programs executable by the processor 802 and that, when executed, specifically program the processor 802 to perform the actions ascribed to the computing system 102 herein. The functional components stored within the computer readable medium 804 may include the web application 116 and storage program 120, including the replication program 122 and the conflict resolution program 124, each of which may include one or more computer programs, applications, modules, executable code, or portions thereof. Furthermore, although these programs are shown together in this example, in some examples, these programs may be separate programs and / or, in use, some or all of these programs may execute on separate computing devices 104 in an individual computing system 102.

[0095] In addition, the computer-readable medium 804 may store data, data structures, and other information used to perform the functions and services described herein. For example, the computer-readable medium 804 may store one or more data structures including metadata 132, and the computer-readable medium 804 may also store object data 130. The computing system 102 may also include or maintain other functional components or data, including programs, drivers, and other data used or generated by the functional components. Moreover, the computing system 102 may include many other logical, programmatic, and physical components, the ones described are merely examples relevant to the discussion herein.

[0096] The one or more communications interfaces 806 may include one or more software and hardware components for enabling communications with various other devices, such as over one or more networks 106. For example, the communications interface 806 may enable communications over one or more of a LAN, the Internet, a cable network, a cellular network, a wireless network (e.g., Wi-Fi) and a wired network (e.g., Fibre Channel, fiber optic, Ethernet), a direct connection as well as short-range communications such as BLUETOOTH®, and the like, as additionally enumerated elsewhere herein.

[0097] Various instructions, methods, and techniques described herein may be considered in the general context of computer-executable instructions, such as computer programs and applications, stored on a computer-readable medium and executed by a processor herein. In general, the terms programs and applications may be used interchangeably and may include instructions, routines, scripts, modules, objects, components, data structures, executable code, and the like, for performing particular tasks or implementing particular data types. These programs, applications, and the like may be executed as native code or downloaded and executed, such as in a virtual machine or other just-in-time compilation execution environment. Typically, the functionality of the programs and applications may be combined or distributed as desired in various implementations. Implementations of these programs, applications, and techniques may be stored on computer storage media or transmitted across some form of communication medium.

[0098] Although the subject matter has been described above in language specific to structural features and / or method acts, it is to be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or acts described. Rather, the specific features and acts are disclosed as example forms of implementing the claims.

Claims

1. 1. A system comprising: a first computing device of a first computing system capable of communicating over a network with a second computing device of a second computing system, where data is replicated between the first computing system and the second computing system, the first computing device comprising: receiving, by the first computing device, a first metadata change made to metadata of a first instance of an object stored at a storage location associated with the first computing device; associating, by the first computing device, a time of the first metadata change with the first metadata change; determining, based on receiving a copy of the second instance of the object from the second computing device of the second computing system, that a second metadata change has been made to metadata of the second instance of the object at the second computing system, the second instance of the object including a time of the second metadata change; resolving, by the first computing device, a conflict between the first metadata change and the second metadata change based at least in part on comparing the time of the first metadata change to the time of the second metadata change; or merging, by the first computing device, the first metadata changes with the second metadata changes into the metadata for the object; and A system configured to perform operations including:

2. 2. The system of claim 1, wherein the operations further include resolving the conflict based in part on a comparison, by the first computing device, of user information associated with a first user indicated to be a source of the first metadata change and user information associated with a second user indicated to be a source of the second metadata change.

3. The user information relating to the first user: the role of the first user with respect to the object; or the first user's authority with respect to the object; The system of claim 2 , further comprising at least one of:

4. 2. The system of claim 1 , wherein the operations further include replicating, by the first computing device, the first instance of the object to the second computing device of the second computing system, the second computing device of the second computing system configured to resolve the discrepancy between the first metadata change and the second metadata change based at least in part on a comparison of the time of the first metadata change and the time of the second metadata change.

5. The operation includes: replicating, by the first computing device, the first instance of the object to a third computing device of a third computing system; replicating, by the second computing device, the second instance of the object to the third computing device of the third computing system; Further comprising:

2. The system of claim 1, wherein the third computing device of the third computing system is configured to resolve the discrepancy between the first metadata change and the second metadata change based at least in part on a comparison of the time of the first metadata change and the time of the second metadata change.

6. The system of claim 5 , wherein the first computing system, the second computing system, and the third computing system are configured in an active-active ring replication configuration.

7. 2. The system of claim 1, wherein the time of the first metadata change is stored with the metadata of the first instance of the object and is replicated with the metadata of the first instance of the object when data is replicated to the second computing device of the second computing system.

8. The act of merging the first metadata change with the second metadata change into the metadata of the object comprises: storing, by the first computing device, both the first metadata change and the second metadata change in the metadata of the first instance of the object stored by the first computing device; The system of claim 1 further comprising:

9. The operation includes: receiving, by the first computing device, upon duplication of data received from the second computing device, a third instance of the object in which the first metadata changes are merged with the second metadata changes; The system of claim 8 further comprising:

10. The operation includes: based at least on determining that the metadata of the third instance of the object includes the first metadata change and the second metadata change, removing the time associated with the first metadata change and user information associated with the first metadata change from the metadata associated with the first instance of the object; The system of claim 9 further comprising:

11. A third metadata change is received from the second computing device of the second computing system along with a copy of the second instance of the object, the operation comprising: resolving a conflict between the first metadata change and the second metadata change based at least in part on a comparison of the time of the first metadata change and the time of the second metadata change to select a metadata change from the first metadata change and the second metadata change based on the resolution of the conflict; determining that the third metadata change is consistent with the selected metadata change; and merging the third metadata change and the selected metadata change into a single instance of the object in the first computing system; The system of claim 1 further comprising:

12. 1. A method comprising: receiving, by a first computing device associated with a first computing system, a first metadata change made to metadata of a first instance of an object stored at a storage location associated with the first computing device; associating, by the first computing device, a time of the first metadata change with the first metadata change; determining, based on receiving a copy of a second instance of the object from a second computing device associated with a second computing system, that a second metadata change has been made to metadata of the second instance of the object in the second computing system, the second instance of the object including a time of the second metadata change; resolving, by the first computing device, a conflict between the first metadata change and the second metadata change based at least in part on a comparison of the time of the first metadata change and the time of the second metadata change; or merging, by the first computing device, the first metadata changes with the second metadata changes into the metadata for the object; and The method includes:

13. 13. The method of claim 12, further comprising resolving the conflict based in part on a comparison, by the first computing device, of user information associated with a first user indicated as being a source of the first metadata change and user information associated with a second user indicated as being a source of the second metadata change.

14. One or more non-transitory computer readable media having stored thereon one or more programs executable by a first computing device of a first computing system to configure the first computing device to perform operations, the operations including: receiving, by the first computing device, a first metadata change made to metadata of a first instance of an object stored at a storage location associated with the first computing device; associating, by the first computing device, a time of the first metadata change with the first metadata change; determining, based on receiving a copy of a second instance of the object from a second computing device associated with a second computing system, that a second metadata change has been made to metadata of the second instance of the object in the second computing system, the second instance of the object including a time of the second metadata change; resolving, by the first computing device, a conflict between the first metadata change and the second metadata change based at least in part on a comparison of the time of the first metadata change and the time of the second metadata change; or merging, by the first computing device, the first metadata changes with the second metadata changes into the metadata for the object; and One or more non-transitory computer-readable media.

15. 15. The one or more non-transitory computer-readable media of claim 14, wherein the operations further include resolving the conflict based in part on a comparison, by the first computing device, of user information associated with a first user indicated to be a source of the first metadata change and user information associated with a second user indicated to be a source of the second metadata change.

Images