Conflict resolution for object metadata

Abstract

In some examples, a first computing system receives a first metadata change to metadata of a first instance of an object stored at a storage location of a first computing device. The first computing system associates a time of the first metadata change with the first metadata change. Based on receiving a copy of a second instance of the object from a second computing system, the first computing system determines that a second metadata change has been made to metadata of the second instance of the object. Further, the second instance of the object includes a time at which the second metadata changes. The first computing system at least one of resolving a conflict between the first metadata change and the second metadata change based at least on comparing the times, or merging the first metadata change and the second metadata change into the metadata of the object.

Description

Technical Field

[0001] The present disclosure relates to the technical field of data storage. Background Art

[0002] A data object such as a file or other object type may generally include object data and object metadata. For example, the object data includes the content of the object. The object metadata may include information about the object data, such as the location of the object data in a file system, system-generated information about the object, user-generated information about the object, etc.

[0003] When replication of metadata is performed across multiple computing systems, such as to provide redundancy protection, the same metadata may exist on multiple different computing systems. In such a topology, the same object metadata may be modified simultaneously by different users at different computing systems. When metadata is changed differently at different computing systems, the result may result in inconsistent data, unexpected data loss, or other undesirable consequences. Summary of the invention

[0004] In some embodiments, a first computing device of a first computing system receives a first metadata change made to metadata of a first instance of an object stored at a storage location of the first computing device. The first computing device associates a time of the first metadata change with the first metadata change. Based on receiving a copy of a second instance of the object from a second computing device of a second computing system, the first computing device determines that a second metadata change was made to metadata of the second instance of the object. In addition, the second instance of the object includes the time of the second metadata change. In some cases, the first computing device may resolve a conflict between the first metadata change and the second metadata change based at least on comparing the time of the first metadata change with the time of the second metadata change. Additionally or alternatively, in some cases, the first computing device may merge the first metadata change and the second metadata change into the metadata of the object. BRIEF DESCRIPTION OF THE DRAWINGS

[0005] The detailed description is set forth with reference to the accompanying drawings. In the drawings, the leftmost digit(s) of a reference number identifies the drawing in which the reference number first appears. The use of the same reference numbers in different drawings indicates similar or identical items or features.

[0006] Figure 1 An example architecture of a system capable of replicating data between multiple computing systems while enabling merging of metadata changes made at the multiple computing systems is illustrated in accordance with some implementations.

[0007] Figure 2 An example process of metadata conflict resolution according to some implementations herein is illustrated.

[0008] Figure 3 is a flow chart illustrating an example process for detecting metadata conflicts according to some implementations.

[0009] Figure 4 is a flow diagram illustrating an example process for conflict resolution and merging of metadata changes in accordance with some implementations.

[0010] Figure 5 is a flow diagram illustrating an example process for performing cleanup after conflict resolution and merging of changes in accordance with some implementations.

[0011] Figure 6 Illustrated is an example of storing last modification time and user information in association with metadata changes in accordance with some implementations.

[0012] Figure 7 An example of merging metadata changes according to some implementations herein is illustrated.

[0013] Figure 8 Illustrated are selected example components of an example computing system that may be used to implement some of the functionality of the systems described herein. DETAILED DESCRIPTION

[0014] Some embodiments herein relate to techniques and arrangements for managing changes made to metadata at multiple different computing systems in a reliable manner by tracking the time at which each metadata change was made and by determining the user role (e.g., user privileges) of the user who modified the metadata at each computing system. For example, by tracking the last modification time (LMT) and user role (such as user privileges) of changes to individual metadata segments at each computing system, embodiments herein are able to handle multiple metadata changes made across multiple computer systems to avoid situations where some metadata changes may be lost due to replication processing, etc. The techniques herein enable individual metadata changes associated with a single object made at different computing systems to be saved, and changes that have been made by different users at different computer systems to be saved simultaneously (e.g., within the same replication cycle) without losing the changes.

[0015] Data loss due to inaccurate conflict resolution is a very common and complex problem in object storage. For example, in an active-active replication topology, replication occurs in two directions between at least two computing systems. In an active-active replication topology, it is possible for the same object to be modified simultaneously by different users with the same or different privileges at different computing systems. However, even when two or more computing systems are configured for replication in an active-active multi-directional arrangement, embodiments herein can resolve any conflicts in the changes and prevent the loss of data.

[0016] Examples herein can save and merge metadata changes by tracking last modification times and user privileges for changes to individual metadata segments, which enables resolution of conflicts in metadata changes such as across a cluster of computing devices such as those that may be arranged in a ring topology or other replication configuration. Furthermore, implementations herein are not limited to resolving conflicts by only checking the last modification times of individual metadata, but can also apply preferences based on the user role (e.g., user privileges) associated with each metadata change.

[0017] In some examples herein, each computing system participating in replication is responsible for detecting and tracking changes to object metadata and storing them back in the object so that the changes can be subsequently transmitted to another system during the next replication cycle. Each system can use the same logic to compare the locally stored data with the remote side data transmitted under the replication cycle to merge metadata changes at the individual metadata level so that the metadata of the object finally merged can contain changes from each side without losing the changes on each side. In addition, even in a highly scalable model, multiple systems can still implement changes to the metadata of an object at the same time. In this case, the role of the user who modifies the metadata on each side can be used as a tie breaker for resolving conflicts.

[0018] According to the embodiments herein, merging metadata changes at the object level is beneficial to avoiding data loss across computing systems in synchronous replication mode and asynchronous replication mode. In addition, the solution herein is highly scalable and can be used for N-dimensional replication systems. In addition, the solution herein can even be used in heterogeneous environments (such as where data and metadata are stored in a primary disk drive, etc. on one side, and data and metadata are stored in a cloud-based storage device via a network on the other side). In addition, although the amount of additional space used to store the last modification time of tracking individual metadata fragments may not be important in some examples, the additional information does not need to be stored for a long time on each computing system. On the contrary, the metadata change information can be deleted from the computing system after the information is used for conflict resolution and merged changes.

[0019] In addition, the solutions herein also work in situations where metadata is inherently heterogeneous (such as when an object has N types of metadata on a first computing system and M types of metadata on a second computing system). In some examples herein, preference can be given to the user who made the most recent modification based on the user's role, and when the last modification time of the object metadata does not provide a solution, thereby providing a more robust conflict resolution. Furthermore, while the techniques herein are capable of resolving conflicts in metadata changes for an object, the solutions herein can also be used to resolve conflicts between changes to different types of object data.

[0020] For purposes of discussion, some example implementations are described in the context of multiple computing systems that perform data replication in both directions. However, the implementations herein are not limited to the specific examples provided, and may be extended to other types of computing system architectures, other types of storage environments, other types of client configurations, other types of data, etc., as will be apparent to those skilled in the art based on the disclosure herein. For example, while some examples are described in the context of resolving conflicts within metadata of an object, similar techniques may be applied to resolve object-level conflicts in the data of some types of objects themselves.

[0021] As an example, assume that the object is a zip file containing two files, file a and file b. In a dual computing system replication topology, it is possible to modify file a of the zip file on a first computing system and file b of the zip file on a second computing system. Therefore, by maintaining and using the conflict resolution techniques herein on a per-file basis, any conflicts can be resolved, and the final merged zip file can contain both the modified version of file a and the modified version of file b.

[0022] As another example, if the object is a type of text file, the last modification time and user information of the user who made the change can be maintained for each line in the text file. Thus, multiple lines of the text file can be modified on different computing systems, and the conflict resolution and merging techniques herein can be used to merge the changed text files from two or more computing systems to include the modified lines from each computing system without losing the changes made at any computing system.

[0023] As yet another example, assume that the object is an image type that includes pixels. Changes to the pixels can be tracked by maintaining a time record of each pixel change made at each computing system that maintains a copy of the image. The conflict resolution techniques herein can be used to preserve and resolve pixel changes to generate a merged image after replication.

[0024] As yet another example, the conflict resolution techniques herein may also be applied to segmented object data, such as where each segment includes a separate tag. In this case, each tagged segment of data that is changed at one of the various computing systems may have a last modification time and user information associated with its corresponding tag. Subsequently, changes may be merged and / or conflict resolution may be performed after replication, similar to the techniques discussed herein for metadata changes. Many other variations will be apparent to those skilled in the art having the benefit of the disclosure herein.

[0025] Figure 1 An example architecture of a system 100 capable of replicating data between multiple computing systems while enabling merging of metadata changes made at multiple computing systems in accordance with some embodiments is illustrated. System 100 includes multiple computing systems 102, each of which includes one or more computing devices 104. For example, a first computing system 102(1) includes one or more computing devices 104(1), a second computing system 102(2) includes one or more computing devices 104(2), and a third computing system 102(3) includes one or more computing devices 104(3). In some examples, multiple computing devices 104 at each computing system 102 may form a cluster of computing devices, etc., but embodiments are not limited to any particular configuration of computing systems 104.

[0026] The computing systems 102 may be located at respective locations 105 and may communicate with each other via one or more networks 106. For example, a first computing system 102(1) may be physically located at a first location 105(1) at a first geographic location, a second computing system 102(2) may be physically located at a second location 105(2) at a second geographic location remote from the first geographic location, and computing system 102(3) may be physically located at a third location 105(3) remote from the first geographic location and the second geographic location. For example, the second and third geographic locations may be sufficiently remote from the first geographic location and sufficiently remote from each other, such as in another city, another state, another country, etc., such that a disaster affecting the first computing system 102(1) at the first location 105(1) is unlikely to affect the second computing system 102(2) at the second location 105(2) or the third computing system 102(3) at the third location 105(3), and vice versa. Thus, the topology provides redundancy in the data stored on the computing systems 102(1)-102(3) to avoid catastrophic loss of data.

[0027] In some examples, computing system 102 can communicate with one or more client devices 108 via network(s) 106. For example, computing system 102 may include access nodes, server nodes, management nodes, and / or other types of service nodes that provide storage services to client devices 108 for enabling client devices 108 to store data and perform other management and control functions, as discussed further below. Client device(s) 108 may be any of various types of computing devices, as discussed further below.

[0028] The one or more networks 106 may include any suitable network, including: a wide area network, such as the Internet; a local area network (LAN), such as an intranet; a wireless network, such as a cellular network, a local wireless network (such as Wi-Fi), and / or a short-range wireless communication (such as Bluetooth ); a wired network including fiber channel, optical fiber, Ethernet, or any other such network, a direct wired connection, or any combination thereof. Thus, one or more networks 106 may include wired and / or wireless communication technologies. The components used for such communications may depend, at least in part, on the type of network, the environment selected, or both. Protocols for communicating over such networks are well known and will not be discussed in detail herein. As an example, (one or more) networks 106 may include a private network, such as a LAN, a storage area network (SAN), or a fiber channel network. In addition, (one or more) networks 106 may include a public network, which may include the Internet, or a combination of public and private networks. The embodiments herein are not limited to any particular type of network as network 106.

[0029] In some examples, computing system 102 may be configured to provide storage and data management services to client users 112, respectively, via (one or more) client devices 108. As several non-limiting examples, client users 112 may include users that perform functions for businesses, enterprises, organizations, government entities, academic entities, etc., and in some examples may include storing very large amounts of data. Additionally, in some examples, client users 112 may include administrative users that use client devices 108 to manage one or more of computing systems 102. However, the embodiments herein are not limited to any particular use or application of system 100 and other example systems and arrangements described herein. For example, in some examples, client device 108 may not be included or may be a completely different type of client device.

[0030] Each client device 108 may be any suitable type of computing device, such as a desktop computer, a laptop computer, a tablet computing device, a mobile device, a smart phone, a wearable device, a terminal, and / or any other type of computing device capable of sending data over a network. Client users 112 may be associated with (one or more) client devices 108, such as through corresponding user accounts, user login credentials, etc. In addition, (one or more) client devices 108 may be configured to communicate with computing system 102 over one or more networks 106, over a separate network, or over any other suitable type of communication connection. Many other variations will be apparent to those skilled in the art having the benefit of the disclosure herein.

[0031] In some implementations, each client device 108 may include a respective instance of a client application 114 that may be executed on the client device 108, such as for communicating with a web application 116 that may be executed on one or more computing devices 104 of the computing system 102, such as for sending user data for storage on the network storage 104 and / or for receiving stored data from the network storage 104 through data instructions (such as write operations, read operations, delete operations, etc.). In some cases, the application 114 may include a browser or may be operated by a browser, while in other cases, the application 114 may include any other type of application with communication functionality that enables communication with the web application 116 or other applications on the computing system 102 through one or more networks 106. In addition, in the case of managing users, the web application 116 may provide remote management functionality. Alternatively, in other examples, any of many other types of software arrangements may be employed to perform these functions, as will be apparent to those skilled in the art having the benefit of the disclosure herein.

[0032] Computing systems 102(1), 102(2), and 102(3) may each execute an instance of storage program 120(1), 120(2), and 102(3), respectively, and storage program 120(1), 120(2), and 102(3), respectively, may include or may access or otherwise execute instances of replication program 122(1), 122(2), and 122(3) and conflict resolution program 124(1), 124(2), and 124(3), respectively. For example, storage program 120 may provide access to object data 130 stored at each computing system 102, and may also manage metadata 132 associated with object data 130. Thus, storage program 120(1) may access and store object data 130(1), and may manage associated metadata 132(1); storage program 120(2) may access and store object data 130(2), and may manage associated metadata 132(2); and storage program 120(3) may access and store object data 130(3), and may manage associated metadata 132(3). For example, storage program 120 may receive object data 130 from client device 108, may store object data 130 on one or more storage devices at network storage 104, and / or may retrieve requested object data 130 and send it to client device 108, such as in response to a client read request.

[0033] The storage program 120 may also manage metadata 132 associated with the object data 130. For example, the metadata 132 may be maintained separately from the object data 130, such as in a metadata database or any other suitable data structure. In some cases, the storage program 120 may generate a portion of the metadata 132, such as to indicate a storage location at a respective computing system 102 of the corresponding object data 130 for the respective object.

[0034] In addition, storage program 120 may include, may execute, may access, or may otherwise coexist with data replication program 122, which may be configured to perform data replication between computing system 102(1), computing system 102(2), and computing system 102(3), as indicated by data replication link 144. For example, data replication program 122 may configure computing systems 102(1)-102(3) to perform asynchronous or synchronous data replication between computing systems 102(1)-102(3). Furthermore, in some examples herein, computing system 102 may employ a time synchronization program, such as NTP chrony, NTPD, etc. ( Figure 1 ), to maintain precise synchronization time with each other.

[0035] Some examples herein may employ an active-active data replication configuration. For example, in an active-active replication topology, replication may occur in both directions over a single link between two computing systems. As an example, in a simple active-active replication topology, two computing systems may replicate the same object data 130 and metadata 132 to each other over an active-active link. The replicated items may be initially created on either computing system 102. Items may be read and written on the computing system 102 where they were created, and after being replicated, may also be read and written on the other computing system 102. With active-active replication, client requests may be directed to any computing system in the computing systems 102. All changes made on each computing system 102 (including both metadata changes and changes to object content) are replicated to (one or more) other computing systems 102.

[0036] Figure 1 The example illustrates a ring active-active topology in which three computing systems 102(1)-102(3) are connected by active-active replication links indicated at 144. With this topology, if one computing system 102 becomes unavailable (e.g., unexpectedly or for scheduled maintenance), the other two computing systems 102 can still provide all required functionality. In addition, in the ring active-active topology, each computing system 102 has two active-active links connecting the computing system 102 to two other computing systems 102 in such a way that all systems 102(1)-102(3) are linked in the form of a ring. The same object data 130 and metadata 132 can be replicated on each data replication link 144. In addition, changes made on one system can travel around the ring in both directions from that computing system until the changes reach those computing systems where the changes have occurred.

[0037] In addition, although a ring topology of three computing systems 102 is illustrated in this example, in other examples, a greater number of computing systems 102 may be included in the ring topology. In addition, although some embodiments herein may be applied to active-active replication topologies, the embodiments herein are not limited thereto. For example, some embodiments may be applied to various other replication scenarios, such as many-to-one, one-to-many, chained replication, etc. In addition, as is known in the art, these configurations may be combined to form complex replication topologies, and embodiments of the conflict resolution techniques herein may also be applied to these topologies.

[0038] exist Figure 1In the illustrated example, assume that a first client user 112(1) makes a metadata change 150 to metadata 132(1) of an object stored by a first computing system 102(1), and that a second client user 112(2) makes a metadata change 152 to metadata 132(2) of the same object stored by a second computing system 102(2). Furthermore, assume that the changes 150, 152 to metadata 132(1) and 132(2) of the same object conflict with each other. As an example, assume that metadata change 150 removes a piece of metadata from the metadata of the object at time t1, and metadata change 152 adds new metadata to the object at time t2. The object having change 150 to metadata 132(1) is subsequently replicated to computing systems 102(2) and 102(3), and the object having change 152 to metadata 132(2) is subsequently replicated to computing systems 102(1) and 102(3). In a conventional replication process, only the version of the object having the most recent change to the metadata may survive the replication process. Thus, the change 150 made at the first computing system 102(1) may be lost because it was made earlier than the subsequent change 152 at the second computing system 102(2). The examples herein are able to save both changes and may further provide conflict resolution after replication, as described below, for example, with respect to Figures 2 to 5 discussed.

[0039] Figures 2 to 5 Including a flowchart illustrating an example process according to some embodiments. The process is illustrated as a set of boxes representing a series of operations in a logical flow chart, some or all of which can be implemented in hardware, software or a combination thereof. In the context of software, a box can represent a computer executable instruction stored on one or more computer-readable media, and when executed by one or more processors, the computer executable instruction programs the processor to perform the operations listed. Typically, computer executable instructions include routines, programs, objects, components, data structures, etc. that perform specific functions or implement specific data types. The order of describing the boxes should not be interpreted as limiting. Any number of described boxes can be combined in any order and / or in parallel to implement the process or alternative processes, and all boxes do not need to be executed. For the purpose of discussion, the environment, framework and system described in the examples herein are referred to to describe the process, but the process can be implemented in a variety of other environments, frameworks and systems.

[0040] Figure 2An example process 200 for metadata conflict resolution according to some embodiments herein is illustrated. Whenever N systems participate in N-side replication, the conflict resolution techniques described herein may be employed. In this example, conflict resolution program 124(1) may be executed on first computing system 102(1) and conflict resolution program 124(2) may be executed simultaneously on second computing system 102(2) to perform operations for preserving and merging the above-described conflict resolution procedures. Figure 1 150 and 152. In addition, although third computing system 102(3) is not shown in this example for clarity of illustration, some operations may also be performed on third computing system 102(3), as discussed further below.

[0041] At 202, first computing system 102(1) may detect metadata changes 150 made by user 112(1) to metadata 132(1) stored by first computing system 102(1), as described above with respect to Figure 1 As an example, conflict resolver 124(1) may run as a background process on computing system 102(1) and may detect changes made to metadata 132(1) stored by first computing system 102(1). Upon detecting the change to metadata 132(1), conflict resolver 124(1) may record the exact time when metadata change 150 was made and may further record user information associated with the user who made metadata change 150.

[0042] At 204, second computing system 102(2) may also execute conflict resolution program 124(2) to detect metadata changes 152 made by user 112(2) to metadata 132(2) stored by second computing system 102(2), as described above with respect to Figure 1 Conflict resolver 124(2) may record the exact time when metadata change 152 was made and user information of the user who made metadata change 152.

[0043] At 206, first computing system 102(1) may update metadata 132(1) for a particular object corresponding to the changed metadata to store a last modification time (LMT) and user information of a user who last modified the individual metadata of the particular object. For example, by storing the last modification time and user information directly with the metadata of the particular affected object (i.e., as part of the data object itself), tracking of change information is simplified and overhead is reduced, such as in situations where it is necessary to separately maintain change information external to the object. This may also facilitate replication of change information, such that the last modification time and user information of the user who last modified the individual metadata are replicated with the object itself, thereby eliminating the need to separately track and transmit change information to send the change information to other computing systems 102(2) and 102(3) ( Figure 2 (not shown in FIG. 1 ). Any number of changes may be made to the metadata by one or more users before the next replication cycle. For example, if changes are made to different segments of metadata, each metadata change is retained along with its own change information. On the other hand, if the same metadata segment is changed and then changed again at the same computing system, only the subsequent metadata changes to that metadata segment may retain the last modification time and user information, since the previous changes to that particular metadata segment have been overwritten and do not need to be replicated.

[0044] At 208, second computing system 102(2) may update metadata of a particular object corresponding to the changed metadata to store the last modification time and user information of the user who changed the metadata directly with the metadata of the particular affected object.

[0045] At 210, the first computing system 102(1) may replicate an object to the second computing system 102(2). For example, computing systems 102(1)-102(3) may be configured to replicate updated objects across computing systems 102(1)-102(3) to other computing systems 102(1)-102(3) participating in object-level replication, such as based on a periodic trigger or other triggering mechanism, by executing respective instances of replication program 122. For example, instead of replicating all changed objects at regular or irregular time periods, the system may instead be configured to replicate any object that has had its data or metadata changed, such as within a specified time of performing the change. Regardless of the type of replication trigger, the replicated object may include object data and object metadata for the object to which one or more metadata changes have been made. The replicated object metadata may also include last modification time(s) and user information for the respective user(s) that made the changes to the object's metadata.

[0046] At 212, second computing system 102(2) copies the object including the modified object metadata and change information (LMT and user information of the user who changed the metadata) to first computing system 102(1). Figure 2 The example only illustrates replication from first computing system 102(1) to second computing system 102(2) and vice versa, but bidirectional replication also occurs between first computing system 102(1) and second computing system 102(2) and third computing system 102(3). Thus, third computing system 102(3) ( Figure 2 ) also receives an object with modified metadata and corresponding change information from each of the first computing system 102 and the second computing system 102.

[0047] At 214, if a conflict exists, first computing system 102(1) may resolve the conflict in the change to the metadata, such as based on a comparison of the last modification time received from second computing system 102(2) with the last modification time recorded by first computing system 102(1) for the metadata change made at first computing system 102(1). For example, conflict resolver 124(1) at first computing system 102(1) may select the most recent last modification time as the metadata change to implement for resolving the conflict. If the last modification times are the same, conflict resolver 124(1) may reference user information of the user making the change to determine which user may have higher privileges with respect to the object, and may select the change made by the user with the higher privilege or other higher priority user role. As an example, assume that first metadata change 150 changes the user permissions of the object to a first specification, and second metadata change 152 changes the user permissions to a second specification. In this case, the last modification time may be applied to select the second specification of the user permissions. Additionally, for some types of changes, the role of the user making the change may override conflict resolution based on the last modification time, such as in the event that the first change prohibits subsequent changes by other users. In addition, other existing conflict resolution rules can be applied based on the type of metadata being changed, such as in the case where both metadata changes change the retention period of the object, then the longest retention period can be applied to resolve the conflict. In addition, in the case where the changes to metadata do not conflict, for example, one metadata change is to user privileges for accessing the object, and the other metadata change is to the retention period of the object, then the changes from both sides are merged, and the two metadata changes from the two computing systems are retained in a single instance of the object.

[0048] At 216, concurrently with block 214, if a conflict exists based on a comparison of the last modification time received from first computing system 102(1) with the last modification time recorded by second computing system 102(2) for the metadata change made at second computing system 102(2), second computing system 102(2) may resolve the conflict in the metadata. Third computing system 102(3) may also perform similar conflict resolution actions. Thus, after replication, each computing system 102(1)-102(3) participating in the replication now has access to both the last modification time of the individual metadata changes 150, 152 and the corresponding user information regarding the user who last modified the individual metadata. Thus, each computing system 102(1)-102(3) may resolve any conflicts at the level of individual metadata changes in parallel with the other systems 102(1)-102(3) by comparing each of the last modification times of the individual metadata and, if necessary, the user information. For example, conflict resolver 124 at each computing system 102 may select the most recent last modification time as the change to be implemented to resolve an immediate conflict. If the last modification times are the same, the conflict resolver 124 can reference the user information of the user making the change to determine which user has higher privileges with respect to the object, and can select the change made by the user with higher privileges or with other higher priority user roles. In addition, for some types of changes, the role of the user making the change can override conflict resolution based on the last modification time, such as in the case where the first change prohibits subsequent changes by other users. In addition, other existing conflict resolution rules can be applied to certain types of conflicts. In addition, in the case where the changes to the metadata do not conflict with each other, then the changes from both sides are merged and the two metadata changes are retained in a single updated object.

[0049] At 218, after the conflict is resolved, first computing system 102(1) may locally merge changes to the object metadata based on the conflict resolution decision. For example, metadata changes from both computing systems 102(1) and 102(2) may be merged and retained unless they directly conflict with each other, as discussed above.

[0050] At 220, after the conflict is resolved, second computing system 102(2) may locally merge changes to the object metadata based on the conflict resolution decision. For example, metadata changes from both computing systems 102(1) and 102(2) may be merged and retained unless they directly conflict with each other, as discussed above. Third computing system 102(3) may also perform this function.

[0051] At 222, first computing system 102(1) may replicate the merged object back to second computing system 102(2) as part of the next replication cycle, and also replicate back to third computing system 102(3), and receive the merged object from third computing system 102(3). For example, the merged object may be merged based on metadata changes from each of computing systems 102(1) and 102(2), and may be saved locally by each system.

[0052] At 224, as part of the next replication cycle, second computing system 102(2) may replicate the merged object back to first computing system 102(1) and also replicate the merged object to and receive the merged object from third computing system 102(3).

[0053] At 226, first computing system 102(1) may delete any change information (LMT and user information) saved and / or used for conflict resolution. For example, since all computing systems 102 have performed the conflict resolution and merge process themselves and have resolved any conflicts, there is no need to save this information. Therefore, change information including any previously saved LMT and user information for the object may be deleted locally to reduce the overall size of the object at each computing system 102.

[0054] At 228, second computing system 102(2) may delete any change information (IMT and user information) saved and / or used for conflict resolution. Third computing system 102(3) may perform similar functions.

[0055] Figure 3 is a flow chart illustrating an example process 300 for detecting metadata conflicts according to some implementations. In some cases, process 300 may be performed at least in part by one or more computing systems in computing system 102, such as by one or more computing devices 104 of computing system 102 executing conflict resolution program 124 and replication program 122.

[0056] At 302, the computing system may monitor changes to metadata initiated by a user. For example, when metadata is maintained in a metadata database or other data structure, the computing system may monitor any changes to the metadata in the data structure.

[0057] At 304, the computing system may determine whether the user has initiated a change to the metadata of the object. If so, the process proceeds to 306. If not, the process returns to 302 to continue monitoring.

[0058] At 306, when the system detects that a user has changed the metadata of an object, the computing system can determine the time when the change was made and the user information of the user who initiated the change. Examples of user information may include user privileges or other user role information, such as a priority level consistent with the user role.

[0059] At 308, the computing system may save the change time and user information as the last modification time in the metadata associated with the target object corresponding to the changed metadata. If a previous last modification time is already stored in the metadata associated with the target object, the system may replace the last modification time with the new time as the last modification time.

[0060] At 310, the computing system may determine whether the object is ready to be replicated in the next replication cycle. If so, the process proceeds to 312. If not, the process proceeds to 302 to monitor any additional changes to the metadata. As described above, in some cases, replication may occur periodically, while in other examples, replication may occur when changes to the object are detected locally at the computing system. In addition, in some examples, when replication is triggered at one of the computing systems, other computing systems participating in the replication of that computing system may also be triggered to perform replication of any objects that have received changes since the last replication cycle. The embodiments herein are not limited to any particular type of replication trigger or replication cycle.

[0061] At 312 , the computing system may replicate the object to other computing systems participating in the replication, and may include object metadata with the replicated object along with a last modification time and user information of the user who changed the metadata.

[0062] At 314, the computing system may determine whether the object was successfully sent to the other systems in the replication. If not, the process goes to 316. If yes, the process returns to 302 to monitor new changes made by the user to the metadata.

[0063] At 316 , if the object was not successfully sent to the other systems participating in the replication, the computing system may retry sending the object to the other systems.

[0064] Figure 4 is a flow chart illustrating an example process 400 for conflict resolution and merging of metadata changes according to some implementations. In some examples, process 400 may be performed by one or more computing devices 104 of computing system 102, such as by executing conflict resolution program 124 and replication program 122.

[0065] At 402 , a computing system may monitor objects received in replication from other computing systems.

[0066] At 404, the computing system may determine whether an object having metadata changes that conflict with metadata changes to a storage object already stored at the computing system has been received through replication, or if no conflict exists, whether the received metadata changes can be merged with metadata changes made to the object already stored at the computing system. If so, the process proceeds to 406. If not, the process proceeds to 402 to continue monitoring objects received in replication.

[0067] At 406, when there is a conflict, the computing system may attempt to resolve the conflict, at least in part, by comparing the last modification time of the received object metadata change(s) and the stored object metadata change(s). For example, the computing system may determine, for the metadata change of the object, the most recent time between the received last modification time and the stored last modification time. Based on determining the most recent last modification time, in some cases, the computing system may select the metadata change associated with the most recent last modification time as the metadata change to be implemented. In addition, for some types of metadata changes, the role of the user making the change may override conflict resolution based on the last modification time, such as in the case where the first change prohibits subsequent changes by other users. In this case, the role of the corresponding user making the metadata change may be used to resolve the conflict in the metadata change, or alternatively, other conflict resolution rules may be applied, as discussed elsewhere herein.

[0068] At 408, the computing system can determine whether the conflict has been resolved using the comparison of the last modification time. If so, the process proceeds to 412. If not, the process proceeds to 410.

[0069] At 410, when the conflict is not resolved based on the comparison of the last modification time, the computing system can resolve the conflict based on the user information by determining the privileges of the object of the user who made the change to the object metadata. As an example, if the last modification times are the same, or if the conflict cannot be resolved based on the last modification time comparison, the computing system can select the metadata change made by a user with a higher level of user privileges (e.g., corresponding to a higher priority role) with respect to the particular data object. In addition, in some cases, other rules for resolving metadata conflicts can be applied, such as in the case of changing the retention period, in which case the longest retention period is applied. Therefore, various other conflict resolution rules known in the art can be applied in specific situations.

[0070] When non-conflicting metadata changes exist and / or when any conflicts have been resolved, the computing systems can merge the non-conflicting metadata changes (if any) at 412. For example, in some cases, two or more of the metadata changes may not conflict, in which case the metadata changes from the two or more computing systems can be retained and all non-conflicting metadata changes can be merged into a single instance of the object.

[0071] At 414, the computing system may determine whether the object is ready to be replicated in the next replication cycle. For example, the computing system may wait until the next replication cycle before sending the object with the merged metadata to other computing systems. If yes, the process proceeds to 416. If not, the process proceeds to 418.

[0072] At 416 , the computing system may replicate the object including the merged object metadata to other computing systems participating in the replication.

[0073] At 418, if it is not yet time for the next replication cycle, the system may wait until the next replication cycle arrives.

[0074] Figure 5 is a flow chart illustrating an example process 500 for performing cleanup after conflict resolution and merging of changes according to some implementations. In some examples, process 500 may be performed by one or more computing devices of computing system 102, for example, by executing conflict resolution program 124 and replication program 122.

[0075] At 502 , a computing system may monitor objects received from other computing systems via replication.

[0076] At 504, the computing system can determine whether an object corresponding to a metadata conflict or other multi-system metadata changes that have been locally resolved, merged, or both has been received. For example, if the computing system has locally resolved a metadata conflict or otherwise merged metadata changes made at multiple computing systems for a particular object, the computing system can determine whether the same object has been received from another computing system via replication. If so, the process proceeds to 506. If not, the process returns to 502 to monitor for additional objects received via replication.

[0077] At 506, the computing system may determine whether conflicts or other metadata changes have also been resolved and / or merged in the metadata of the received object, the received object corresponding to the object for which the metadata conflicts have been resolved locally. If so, the process proceeds to 510. If not, the process proceeds to 508.

[0078] At 508 , the computing system may wait for the next replication cycle to see if the other computing system resolves the multi-system changes in the metadata.

[0079] At 510, the computing system may determine whether the object is ready to purge the change information. For example, when the computing system determines that any conflicts have been resolved by all computing systems and / or metadata changes have been merged by all computing systems, the computing system may proceed with the purge of the change information locally. If yes, the process proceeds to 512. If no, the process proceeds to 508 to wait for the next replication cycle.

[0080] At 512, the computing system may delete the local change information including the last modification time(s) and the corresponding user information. For example, the location in the metadata of the object may clear the information.

[0081] At 514, the computing system can update the object locally based on the received copy. For example, if any other changes are made to the object, these changes can be updated on the local object.

[0082] The example processes described herein are merely examples of processes provided for the purpose of discussion. Many other changes will be apparent to those skilled in the art based on the disclosure herein. In addition, although the disclosure herein sets forth several examples of suitable frameworks, architectures, and environments for executing processes, the embodiments herein are not limited to the specific examples shown and discussed. In addition, the present disclosure provides various example embodiments as described and as shown in the accompanying drawings. However, the present disclosure is not limited to the embodiments described and illustrated herein, but can be extended to other embodiments known or to become known to those skilled in the art.

[0083] Figure 6An example 600 of storing a last modification time and user information in association with a metadata change in accordance with some embodiments is illustrated. In this example, as indicated at 601, assume that a user makes a first metadata change 604(1) to an object 602. Upon detecting this, the computing system may associate a last modification time 606(1) indicating the time at which the user made the first metadata change 604(1) with the first metadata change 604(1). Additionally, the computing system may associate user information 608(1) of the user who made the first metadata change 604(1) with the first metadata change 604(1). As an example, the user information 608(1) may include at least a role of the user who made the first metadata change 604(1), which may include a priority of the user relative to the object 602 and / or privileges of the user. Thus, the last modification time 606(1) and the user information 608(1) may be stored as part of the metadata of the object 602 and, therefore, form part of the object 602.

[0084] Subsequently, as indicated at 610, assume that the same user or a different user makes a second metadata change 604(2) to some other metadata of object 602 at the same computing system. Based on detecting the second metadata change 604(2), the computing system associates a last modification time 606(2) with the second metadata change 604(2). The computing system also associates user information 608(2) of the user who made the second metadata change 604(2) with the second metadata change 604(2). As previously described, any number of changes may be made to the metadata by one or more users at the same computing system prior to the next replication cycle. For example, if changes are made to different pieces of metadata, each metadata change is retained in object 602 along with its own change information (LMT and user information). On the other hand, if the same piece of metadata is changed at, for example, first computing system 102(1), and then changed again at first computing system 102(1), only the subsequent metadata changes to that piece of metadata may be retained in object 602 having the last modification time and user information because previous changes to that particular piece of metadata have already been overwritten and do not need to be copied.

[0085] During replication of object 602 to another computing system, both first metadata change 604(1) and second metadata change 604(2) along with their corresponding change information (i.e., LMT 606(1), user information 608(1), LMT 606(2), and user information 608(2)) may be replicated to the other computing system(s) participating in the replication topology. In some examples, if another user also makes changes to the metadata stored on the second computing system ( Figure 6If a third metadata change is made to an instance of object 602 (not shown in FIG. 1 ), then after replication, if both the first metadata change 604(1) and the second metadata change 604(2) do not conflict with the third metadata change, they can be merged with the third metadata change made at the second computing system, or alternatively, if a conflict exists, the conflict can be resolved as described above.

[0086] Figure 7 An example 700 of merging metadata changes according to some embodiments herein is illustrated. In some examples, assume that the first computing system 102(1) corresponds to the Figure 1 and Figure 2 The first computing system 102(1) discussed above and the second computing system 102(2) correspond to Figure 1 and Figure 2 A second computing system 102(2) is discussed (computing system 102(3) is omitted for clarity of illustration, or may be omitted entirely in this example).

[0087] At 701, assume that before performing replication, first computing system 102(1) includes a first instance of object 702(1) and second computing system 102(2) includes a second instance of object 702(2). Further, assume that a first user makes a first metadata change 704(1) to metadata of the first instance of object 702(1) at first computing system 102(1), and a second user makes a second metadata change 704(2) to metadata of the second instance of object 702(2) at second computing system 102(2). Based on metadata change 704(1), first computing system 102(1) associates a last modification time 706(1) and user information 708(1) with first metadata change 704(1). Further, based on metadata change 704(2), second computing system 102(2) associates a last modification time 706(2) and user information 708(2) with second metadata change 704(2).

[0088] Subsequently, assume that data replication 710 is performed to replicate the first instance of object 702(1) to second computing system 102(2), and data replication 712 is performed to replicate the second instance of object 702(2) to first computing system 102(1). As indicated at 714, each of first computing system 102(1) and second computing system 102(2) may execute its respective conflict resolution program 124(1) or 124(2), respectively, to resolve conflicts, if any, between first metadata change 704(1) and second metadata change 704(2). In this example, assume that no conflicts exist between first metadata change 704(1) and second metadata change 704(2). For example, first metadata change 704(1) may be a change to user privileges for object 702, and second metadata change 704(2) may be a change to a retention period for object 702. In this case, since no conflict resolution is required, the first metadata change 704(1) and the second metadata change 704(2) are merged into a single object 702 at each computing system 102(1) and 102(2), so that the metadata changes from both computing systems 102(1) and 102(2) are retained in a single instance of objects 702(1) and 702(2) stored at each system 102(1) and 102(2), respectively.

[0089] Furthermore, in some examples, as described above, there may be more than two metadata changes, some of which may conflict with metadata changes made at other computing systems, and some of which do not conflict. In such cases, conflicts between two or more metadata changes may be resolved, such as by using last modification time, user role, or other conflict resolution rules. After resolving any conflicting changes, the resolved metadata changes and any other non-conflicting metadata data changes are merged into a single object at each computing system, as discussed above.

[0090] Figure 8Selected example components of an example computing system 102 that can be used to implement some functions of the system described herein are illustrated. The computing system 102 includes one or more computing devices 104, which can include one or more servers or other types of computing devices that can be embodied in any number of ways. In addition, in some examples, the computing device 104 can also include one or more storage systems, storage controllers, network-attached storage devices, storage arrays, storage area networks, etc., or can communicate with one or more storage systems, storage controllers, network-attached storage devices, storage arrays, storage area networks, etc. For example, in the case of a server, programs, other functional components and data can be implemented on a single server, a server cluster, a server group or data center, a cloud-hosted computing service, etc., but other computer architectures can be used in addition or alternatively. Multiple computing devices 104 can be located together or separately, and are organized, for example, as virtual servers, server libraries and / or server groups. The described functions can be provided by servers of a single entity or enterprise, or can be provided by servers and / or services of multiple different entities or enterprises.

[0091] In the illustrated example, the computing device 104 includes or may be associated with one or more processors 802, one or more computer-readable media 804, and one or more communication interfaces 806. Each processor 802 may be a single processing unit or multiple processing units, and may include a single or multiple computing units or multiple processing cores. The (one or more) processors 802 may be implemented as one or more central processing units, microprocessors, microcomputers, microcontrollers, digital signal processors, graphics processing units, system-on-chip processors, state machines, logic circuits, and / or any device that manipulates signals based on operating instructions. As an example, the (one or more) processors 802 may include one or more hardware processors and / or logic circuits of any suitable type that are specifically programmed or configured to perform the algorithms and processes described herein. The (one or more) processors 802 may be configured to obtain and execute computer-readable instructions stored in the computer-readable medium 804, which may program the processor 802 to perform the functions described herein.

[0092] The computer-readable medium 804 may include volatile and non-volatile memory and / or removable and non-removable media implemented with any type of technology for storing information (such as computer-readable instructions, data structures, program modules, or other data). For example, the computer-readable medium 804 may include, but is not limited to, RAM, ROM, EEPROM, flash memory or other memory technology, optical storage devices, solid-state storage devices, tape and disk storage devices, or any other medium that can be used to store the desired information and can be accessed by a computing device. In addition, in some examples, the computer-readable medium 804 includes a network storage system, which may include a storage array, a network-attached storage device, a storage area network, cloud storage, etc., as described elsewhere herein.

[0093] Depending on the configuration of computing system 102, computer readable medium 804 may be a tangible, non-transitory medium to the extent that when referred to, non-transitory computer readable media do not include media such as energy, carrier signals, electromagnetic waves, and / or signals themselves. In some cases, computer readable medium 804 may be co-located with computing system 102, while in other examples, computer readable medium 804 may be partially remote from computing system 102.

[0094] The computer-readable medium 804 may be used to store any number of functional components that can be executed by (one or more) processors 802. In many implementations, these functional components include instructions or programs that can be executed by (one or more) processors 802 and when executed, specifically program (one or more) processors 802 to perform actions attributed to the computing system 102 in this article. The functional components stored in the computer-readable medium 804 may include web applications 116 and stored programs 120 (including replication programs 122 and conflict resolution programs 124), each of which may include one or more computer programs, applications, modules, executable codes, or portions thereof. In addition, although these programs are illustrated together in this example, in some examples, these programs may be separate programs and / or during use, some or all of these programs may be executed on separate computing devices 104 at the corresponding computing system 102.

[0095] In addition, the computer-readable medium 804 can store data, data structures, and other information used to perform the functions and services described herein. For example, the computer-readable medium 804 can store one or more data structures including metadata 132, and the computer-readable medium 804 can also store object data 130. The computing system 102 can also include or maintain other functional components and data, which can include programs, drivers, etc., and data used or generated by the functional components. In addition, the computing system 102 can include many other logical, programming, and physical components, of which those described above are merely examples relevant to the discussion herein.

[0096] The one or more communication interfaces 806 may include one or more software and hardware components for enabling communication with various other devices, such as through one or more networks 106. For example, the communication interface(s) 806 may be connected via a LAN, the Internet, a cable network, a cellular network, a wireless network (e.g., Wi-Fi) and a wired network (e.g., Fiber Channel, fiber optic, Ethernet), a direct connection, and a network such as Bluetooth. Communication can be achieved by one or more of the short-range communications, etc., as otherwise listed elsewhere in this document.

[0097] The various instructions, methods and techniques described herein can be considered in the general context of computer executable instructions (such as computer programs and applications stored on computer readable media and executed by (one or more) processors herein). Generally, the terms "program" and "application" can be used interchangeably and can include instructions, routines, scripts, modules, objects, components, data structures, executable code, etc. for performing specific tasks or implementing specific data types. These programs, applications, etc. can be executed as native code, or can be downloaded and executed, such as in a virtual machine or other just-in-time compilation execution environment. Generally, the functions of programs and applications can be combined or distributed as needed in various embodiments. The implementation methods of these programs, applications and techniques can be stored on computer storage media or transmitted through some form of communication media.

[0098] Although the subject matter has been described in language specific to structural features and / or methodological acts, it is to be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or acts described. Rather, the specific features and acts are disclosed as example forms of implementing the claims.

Claims

1. A system comprising: A first computing device of a first computing system, the first computing device being capable of communicating with a second computing device of a second computing system via a network, wherein data is replicated between the first computing system and the second computing system, the first computing device being configured to perform operations including: receiving, by the first computing device, a first metadata change made to metadata of a first instance of an object stored at a storage location associated with the first computing device; associating, by the first computing device, a time of the first metadata change with the first metadata change; determining, based on receiving a copy of the second instance of the object from the second computing device of the second computing system, that a second metadata change was made to metadata of the second instance of the object at the second computing system, wherein the second instance of the object includes a time of the second metadata change; And at least one of the following: resolving, by the first computing device, a conflict between the first metadata change and the second metadata change based at least in part on comparing a time of the first metadata change to a time of the second metadata change, or The first metadata change and the second metadata change are merged, by the first computing device, into the metadata of the object.

2. The system of claim 1 , the operations further comprising resolving, by the first computing device, the conflict based in part on a comparison of user information associated with a first user indicated as a source of the first metadata change and user information associated with a second user indicated as a source of the second metadata change.

3. The system according to claim 2, wherein: The user information associated with the first user indicates at least one of the following: the role of the first user relative to the object, and The privileges of the first user with respect to the object.

4. According to the system of claim 1, the operation also includes copying, by the first computing device, the first instance of the object to the second computing device at the second computing system, wherein the second computing device at the second computing system is configured to resolve the conflict between the first metadata change and the second metadata change based at least in part on comparing the time of the first metadata change with the time of the second metadata change.

5. The system of claim 1, wherein the operations further comprise: copying, by the first computing device, the first instance of the object to a third computing device at a third computing system; copying, by the second computing device, the second instance of the object to the third computing device at the third computing system; and Wherein the third computing device at the third computing system is configured to resolve a conflict between the first metadata change and the second metadata change based at least in part on comparing a time of the first metadata change with a time of the second metadata change.

6. The system according to claim 5, wherein: The first computing system, the second computing system, and the third computing system are arranged in a ring-shaped active-active replication configuration.

7. The system according to claim 1, wherein: The time at which the first metadata was changed is stored with the metadata of the first instance of the object and is replicated with the metadata of the first instance of the object during replication of data to the second computing device at the second computing system.

8. The system according to claim 1, wherein: The operation of merging the first metadata change and the second metadata change into the metadata of the object further comprises: Both the first metadata change and the second metadata change are stored, by the first computing device, in the metadata of the first instance of the object stored by the first computing device.

9. The system of claim 8, wherein the operations further comprise: A third instance of the object is received by the first computing device in a replication of the data received from the second computing device, wherein the first metadata change has been merged with the second metadata change.

10. The system of claim 9, the operations further comprising: Based at least on determining that the metadata of the third instance of the object includes the first metadata change and the second metadata change, a time associated with the first metadata change and user information associated with the first metadata change are deleted from the metadata associated with the first instance of the object.

11. The system according to claim 1, wherein: A third metadata change is received from the second computing device of the second computing system along with the copy of the second instance of the object, the operations further comprising: resolving a conflict between the first metadata change and the second metadata change based at least in part on comparing a time of the first metadata change to a time of the second metadata change, thereby selecting a metadata change from the first metadata change and the second metadata change based on resolving the conflict; determining that the third metadata change does not conflict with the selected metadata change; and The third metadata change and the selected metadata change are merged at the first computing system into a single instance of the object.

12. A method comprising: receiving, by a first computing device associated with a first computing system, a first metadata change made to metadata of a first instance of an object stored at a storage location associated with the first computing device; associating, by the first computing device, a time of the first metadata change with the first metadata change; determining, based on receiving a copy of a second instance of the object from a second computing device associated with a second computing system, that a second metadata change was made to metadata of the second instance of the object at the second computing system, wherein the second instance of the object includes a time of the second metadata change; And at least one of the following: resolving, by the first computing device, a conflict between the first metadata change and the second metadata change based at least in part on comparing a time of the first metadata change to a time of the second metadata change, or The first metadata change and the second metadata change are merged, by the first computing device, into the metadata of the object.

13. The method of claim 12 further includes resolving the conflict by the first computing device based in part on a comparison of user information associated with a first user indicated as the source of the first metadata change and user information associated with a second user indicated as the source of the second metadata change.

14. One or more non-transitory computer-readable media storing one or more programs executable by a first computing device of a first computing system to configure the first computing device to perform operations comprising: receiving, by the first computing device, a first metadata change made to metadata of a first instance of an object stored at a storage location associated with the first computing device; associating, by the first computing device, a time of the first metadata change with the first metadata change; determining, based on receiving a copy of a second instance of the object from a second computing device associated with a second computing system, that a second metadata change was made to metadata of the second instance of the object at the second computing system, wherein the second instance of the object includes a time of the second metadata change; And at least one of the following: resolving, by the first computing device, a conflict between the first metadata change and the second metadata change based at least in part on comparing a time of the first metadata change to a time of the second metadata change, or The first metadata change and the second metadata change are merged, by the first computing device, into the metadata of the object.

15. According to one or more non-transitory computer-readable media of claim 14, the operations further comprising resolving, by the first computing device, the conflict based in part on a comparison of user information associated with a first user indicated as a source of the first metadata change and user information associated with a second user indicated as a source of the second metadata change.

Images