Distributed object storage system multi-site data synchronization method and device and medium

Abstract

The application discloses a kind of distributed object storage system multi-site data synchronization method, device and medium, it is related to distributed storage technical field, for realizing the data synchronization between sites.A kind of method is provided for the problem of unable synchronization or imbalance when the hard disk configuration between sites is different in current data synchronization scheme.The filtering strategy is configured for each site, the filtering strategy includes multiple filtering rules, and each filtering rule includes the storage class associated with the storage pool in the target site.When data synchronization is needed, the metadata of the source object is matched with the corresponding filtering rule of the target site.And the storage pool of the source object is specified according to the storage class in the successfully matched filtering rule, to realize accurate data synchronization between sites.In addition, based on the setting of filtering rule, the source object can be allocated to any storage pool of the target site based on different needs, to avoid the problem of data imbalance.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of distributed storage, in particular to a multi-site data synchronization method and device for a distributed object storage system and a medium. BACKGROUND

[0002] For a distributed object storage system, when a bucket is created in the system, an object policy must be specified. The object policy defines a storage class (or storage category). The storage class is associated with a storage pool in the site. Then, when an object is uploaded to the bucket, the user can specify the storage class to specify that the object data is stored in a certain storage pool.

[0003] When creating an object policy in a multi-site environment, the same-named object policy must be created on each site in turn to create a bucket on the master site using the object policy, and then the bucket is synchronized to each slave site. However, in actual applications, the configurations of each site are not exactly the same. For example, the disk configurations of different sites are different, so the storage classes included in the same-named object policies of different sites are also different.

[0004] Suppose that an object needs to be synchronized from site A to site B after being written to site A. The storage class of the object in site A is Aclass1, but site B does not have the storage class Aclass1. At this time, data synchronization will fail. If a fixed storage class is specified in advance for the synchronization of data without the same storage class, all the data will be written to the fixed storage class, which will cause an imbalance in data and affect system performance.

[0005] Therefore, there is an urgent need in the art for a multi-site data synchronization method for a distributed object storage system to solve the problem of how to achieve data synchronization between sites when the site configurations are inconsistent and ensure the balance of data synchronization. SUMMARY

[0006] The purpose of the present application is to provide a multi-site data synchronization method, device and medium for a distributed object storage system to solve the problem of how to achieve data synchronization between sites when the site configurations are inconsistent and ensure the balance of data synchronization.

[0007] To solve the above technical problems, the present application provides a multi-site data synchronization method for a distributed object storage system, comprising:

[0008] When a source object on a source site needs to be synchronized to a target site, metadata information of the source object and a filtering policy corresponding to the target site are acquired; the filtering policy comprises a plurality of filtering rules; each filtering rule comprises a set of matching items and a storage class corresponding to each other; the storage class is associated with any storage pool in the target site; the matching items comprise one or more kinds of metadata information;

[0009] The metadata information of the source object is matched with the matching items in each filtering rule;

[0010] According to the storage class in the filtering rule hit by the source object, the source object is stored in a designated storage pool in the target site.

[0011] In a possible embodiment, the matching of the metadata information of the source object with the matching items in each filtering rule comprises:

[0012] The metadata information of the source object is matched in order from high to low according to the priority of each filtering rule; the priority of different filtering rules is different;

[0013] When any filtering rule is matched successfully, subsequent matching is stopped, and the step of storing the source object in a designated storage pool in the target site according to the storage class in the filtering rule hit by the source object is performed.

[0014] In a possible embodiment, the matching items comprise a matching condition corresponding to any kind of metadata information, or a plurality of matching conditions corresponding to the same or different kinds of metadata information;

[0015] The kinds of metadata information comprise object prefix, object suffix, object size, object type, object label, metadata, and object owner.

[0016] In a possible embodiment, when the matching items comprise a plurality of matching conditions, a logical relationship is further included between each matching condition;

[0017] The logical relationship comprises and relationship and or relationship.

[0018] If the logical relationship is and relationship, the source object is considered to be matched successfully with the filtering rule only when the metadata information of the source object satisfies all the matching conditions in the matching items;

[0019] If the logical relationship is or relationship, the source object is considered to be matched successfully with the filtering rule when the metadata information of the source object satisfies any matching condition in the matching items.

[0020] In a possible embodiment, before obtaining the filtering policy corresponding to the target site, the method further includes:

[0021] Determine whether the state of the source object storage class configuration item is on; wherein the source object storage class configuration item is a configurable state bit, including two states: on and off;

[0022] If not, proceed to the step of obtaining a filtering policy corresponding to the target site;

[0023] If so, the object policy of the source site is obtained, and the source object is stored in a designated storage pool in the target site according to the object policy of the source site.

[0024] In a possible embodiment, if the filtering policy obtained in the step of obtaining the filtering policy corresponding to the target site is an empty set, or if no filtering rule is matched in the step of matching the metadata information of the source object with the matching items in each filtering rule, the method further includes:

[0025] Obtain a default storage class; wherein the default storage class is a storage class included in each site;

[0026] According to the default storage class, the source object is stored in a designated storage pool in the target site.

[0027] In a possible embodiment, applied to any site, the method further includes:

[0028] Upon receiving the filtering policy, obtaining bucket metadata of the current site cluster;

[0029] Parsing the bucket metadata and adding the filtering policy to the bucket metadata;

[0030] Rewriting the bucket metadata after adding the filtering policy into the current site cluster;

[0031] This method is applied to a slave site. When the slave site needs to synchronize buckets from the master site, the method further includes:

[0032] Obtaining the metadata synchronization log of the primary site, and parsing to obtain the bucket metadata corresponding to the primary site as the first metadata;

[0033] erasing the filtering policy in the first metadata to obtain second metadata;

[0034] Loading the bucket metadata of the current site cluster into memory as third metadata;

[0035] parsing the third metadata to obtain a metadata part of the filtering strategy corresponding to the current site as fourth metadata;

[0036] writing the fourth metadata into the second metadata to obtain fifth metadata;

[0037] writing the fifth metadata into the current site cluster to complete the bucket synchronization.

[0038] To solve the above technical problems, the present application further provides a distributed object storage system multi-site data synchronization device, comprising:

[0039] an acquisition module, configured to acquire metadata information of a source object and a filtering strategy corresponding to a target site when it is required to synchronize the source object on a source site to the target site; the filtering strategy comprises a plurality of filtering rules, each filtering rule comprises a set of matching items and a storage class corresponding to each other; the matching items comprise one or more kinds of metadata information;

[0040] a matching module, configured to match the metadata information of the source object with the matching items in each filtering rule;

[0041] a synchronization module, configured to store the source object in a designated storage pool in the target site according to the storage class in the filtering rule hit by the source object.

[0042] To solve the above technical problems, the present application further provides a distributed object storage system multi-site data synchronization device, comprising:

[0043] a memory, configured to store a computer program;

[0044] a processor, configured to execute the computer program to realize the steps of the distributed object storage system multi-site data synchronization method as described above.

[0045] To solve the above technical problems, the present application further provides a non-volatile storage medium, wherein the non-volatile storage medium stores a computer program, and the computer program is executed by a processor to realize the steps of the distributed object storage system multi-site data synchronization method as described above.

[0046] The application provides a multi-site data synchronization method of a distributed object storage system, which comprises the following steps: configuring a corresponding filtering strategy for each site in a multi-site scenario of the distributed object storage system, wherein the filtering strategy comprises a plurality of filtering rules, and each filtering rule comprises a storage class associated with a storage pool in a target site; when object data needs to be synchronized between sites, matching the metadata of a source object with the corresponding filtering rule of the target site; and specifying the storage pool of the source object according to the storage class in the filtering rule that is matched successfully, so that accurate data synchronization between sites can be realized even if the object strategies of the source site and the main site are different. In addition, based on the setting of the filtering rule, different matching items can be set for each storage pool of the target site, so that the source objects with different metadata can be allocated to any storage pool of the target site based on different needs, and the problem of data imbalance caused by the allocation of all synchronization data to the same storage pool can be avoided.

[0047] The application provides a multi-site data synchronization device and a nonvolatile storage medium of a distributed object storage system, which correspond to the above method and have the same effect. BRIEF DESCRIPTION OF DRAWINGS

[0048] In order to more clearly illustrate the embodiments of the application, the drawings needed in the embodiments will be briefly introduced below. Obviously, the drawings in the following description are only some embodiments of the application, and other drawings can be obtained by those skilled in the art without creative labor on the basis of these drawings.

[0049] Figure 1 A flowchart of a multi-site data synchronization method of a distributed object storage system provided by the embodiments of the application is shown in the figure.

[0050] Figure 2 A flowchart of another multi-site data synchronization method of a distributed object storage system provided by the embodiments of the application is shown in the figure.

[0051] Figure 3 A structure diagram of a multi-site data synchronization device of a distributed object storage system provided by the embodiments of the application is shown in the figure.

[0052] Figure 4 A structure diagram of another multi-site data synchronization device of a distributed object storage system provided by the embodiments of the application is shown in the figure. DETAILED DESCRIPTION

[0053] With reference to the accompanying drawings, the technical solutions in the embodiments of the present application will be described clearly and completely. Obviously, the described embodiments are only a part of the embodiments of the present application, but not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative work fall within the protection scope of the present application.

[0054] The core of the present application is to provide a multi-site data synchronization method and device for a distributed object storage system and a medium.

[0055] In order for those skilled in the art to better understand the present application, the present application will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0056] In a distributed object storage system, multiple sites can be deployed. A site is an independent object storage system with its own storage gateway and storage pool. A site can have multiple object storage gateway instances and can independently provide business access.

[0057] When creating an object policy in a multi-site environment, the same-named object policy needs to be created on each site in turn, and finally the bucket can be created on the master site using the object policy, and then the bucket is synchronized to each slave site. If the bucket is synchronized to the slave site, and it is found that the slave site does not have the same-named object policy used by the bucket, the bucket will fail to synchronize at the slave site. The object policy defines an index pool, a hierarchical pool and a data pool, among which the index pool and the hierarchical pool are only one, and the data pool can have multiple, each data pool is associated with a storage class. Based on this, by creating a storage class and specifying the storage class when uploading an object, users can upload objects to different data pools in the object storage cluster to achieve data filtering of objects at the business end.

[0058] In a multi-site scenario, when creating an object policy, all sites are required to create the same-named object policy, and there is no requirement that the specific storage pool specified in the object policy is also the same, and there is no requirement that the storage class associated with the data pool in the same object policy on the master and slave sites is also the same. Therefore, when the disk configurations of the sites are different, it may cause problems such as data synchronization failure between sites.

[0059] For example, assume that there are storage pools Apool1, Apool2, and Apool3 on site A. There is also an object policy policy, which creates a total of three storage classes Aclass1, Aclass2, and Aclass3 inside. Aclass1 is associated with Apool1, Aclass2 is associated with Apool2, and Aclass3 is associated with Apool3. There are storage pools Bpool1, Bpool2, Bpool3, and Bpool4 on site B. There is also an object policy policy with the same name, which creates a total of four storage classes Bclass1 to Bclass4 inside. Bclass1 is associated with Bpool1, Bclass2 is associated with Bpool2, Bclass3 is associated with Bpool3, and Bclass4 is associated with Bpool4.

[0060] In a multi-site environment, when an object is written to a site, a synchronization log is recorded, and other sites synchronize the newly written object to each target site by comparing the synchronization log. However, data synchronization is an automatic backup function, and users do not perceive the data synchronization process, nor can they specify a storage class to write different objects to different data pools when the data is synchronized to the target site. If the object is synchronized to site B according to the storage class of the source object, such as Aclass1, the storage is still performed according to the source storage class. Since site B does not have a storage class Aclass1, data synchronization will fail. If data is written according to a fixed storage class such as Bclass1, when data on site A is synchronized to site B, all data will be written to the fixed data pool Bpool1 on site B, and no synchronization object will be written to the other three data pools, resulting in data imbalance and poor performance.

[0061] To solve the above problems, the present application provides a multi-site data synchronization method for a distributed object storage system, as shown in Figure 1 The method comprises the following steps:

[0062] S11: When a source object on a source site needs to be synchronized to a target site, the metadata information of the source object and a filtering policy corresponding to the target site are obtained.

[0063] The filtering policy comprises a plurality of filtering rules. Each filtering rule comprises a set of matching items and a storage class corresponding to each other. The storage class is associated with any storage pool in the target site. The matching item comprises one or more kinds of metadata information.

[0064] S12: The metadata information of the source object is matched with the matching items in each filtering rule.

[0065] S13: The source object is stored in a specified storage pool in the target site according to the storage class in the filtering rule hit by the source object.

[0066] For step S11, the matching item in the filter rule is specifically used to achieve the matching with the metadata information of the source object. The matching item includes one or more kinds of metadata information to match with the corresponding kind of metadata information in all metadata information of the source object in step S12. When the metadata information of the source object meets the matching item set in a filter rule, it is considered that the source object hits the filter rule. At this time, step S13 can store the source object in the storage pool associated with the storage class in the hit filter rule according to the storage class in the hit filter rule.

[0067] It is easy to understand that the filter strategy corresponds to one site. And all the storage classes in the filter rules contained in the filter strategy corresponding to the site are associated with the respective storage pools of the site. That is, as long as the source object hits any filter rule in the filter strategy corresponding to the target site, the source object can be stored in a specified storage pool in the target site according to the storage class in the hit filter rule, so that the data synchronization between sites can be achieved.

[0068] In addition, the matching item in the filter rule is the condition for filtering the source object. As known from the above, a site usually includes multiple storage pools. Therefore, how to store the synchronized object data (i.e. the source object) in different storage pools in the site in a more balanced manner is a key to achieving balanced data storage in the storage pool. In the method, this process is achieved through the matching item in the filter rule. A filter rule includes a matching item and a storage class. Therefore, when the source object hits a matching item (i.e. the source object meets a certain condition), it will be stored in a specified storage pool according to the storage class of the filter rule to which the hit matching item belongs. When the filter strategy includes multiple filter rules with different storage classes and matching items, balanced data synchronization between sites can be achieved.

[0069] However, it should be noted that there should not be two filter rules with the same matching item among multiple filter rules belonging to one filter strategy. If the matching item is the same and the storage class is also the same, the two filter rules are set repeatedly and are meaningless. If the matching item is the same but the storage class is different, the two filter rules are set in conflict, which will cause the storage of the source object hitting the matching item to be chaotic and cause more serious problems.

[0070] To sum up, the multi-site data synchronization method of the distributed object storage system provided by the application sets a filtering strategy for each site. The filtering strategy includes multiple filtering rules, and each filtering rule includes a matching item and a storage class. The storage class is associated with a storage pool of the corresponding site, so that when the filtering rule is hit, the source object can be accurately stored in a certain storage pool of the target site through the storage class in the filtering rule, thereby realizing data synchronization. The matching item provides corresponding filtering conditions for multiple different storage pools in the target site, that is, only when the filtering conditions of a certain storage pool are met (i.e., the corresponding matching item is hit), the object data can be stored in the storage pool. That is, on-demand allocation storage in different storage pools can be realized, and various synchronization requirements of users can be met. In addition, the method can also realize balanced storage of data synchronization between multiple sites, thereby enhancing the robustness of the object storage multi-site data synchronization function.

[0071] On the other hand, in the above embodiment, the specific implementation of the matching item is not limited. In actual application, appropriate matching items can be freely set to store object data meeting specific conditions in specific storage pools based on different needs of object storage in different storage pools in the site.

[0072] However, the embodiment also provides a possible implementation scheme for the matching item. The above matching item can specifically include a matching condition corresponding to any kind of metadata information, or multiple matching conditions corresponding to the same or different kinds of metadata information.

[0073] The kinds of metadata information include object prefix, object suffix, object size, object type, object label, metadata, and object owner.

[0074] It should be noted that one matching condition corresponds to one kind of metadata information. The specific content of the matching condition should be determined according to the kind of metadata information. For example, for metadata information such as object size represented by a numerical value, the corresponding matching condition can be a numerical value or an array, and can also be a continuous or discontinuous numerical range. The numerical range can have only an upper limit value or a lower limit value, or have both an upper limit value and a lower limit value. For metadata information such as object type represented by a string of specific content, the corresponding matching condition can be a specific object type. When the object type of the source object is consistent with the matching condition, it is considered to be hit. Similarly, for metadata types such as object prefix and object suffix, the corresponding matching condition can be a specific prefix name or suffix name. If the prefix name or suffix name of the source object includes the prefix name or suffix name set in the matching condition, the matching item is hit. In addition, the object label refers to the label of the source object containing the object label set in the matching condition; the metadata refers to the custom metadata of the source object containing the custom metadata set in the matching condition; and the object owner refers to the owner of the source object being consistent with the object owner set in the matching condition.

[0075] In the embodiment, the matching item of one filter rule can include one or more matching conditions. When multiple matching conditions are involved, each matching condition can involve metadata information of the same or different kind. This can achieve more flexible and comprehensive matching, and can adapt to more diverse filtering needs of users.

[0076] Further, when the matching item of one filter rule includes multiple matching conditions, the embodiment also provides a corresponding possible implementation scheme:

[0077] The matching conditions also include a logical relationship. The logical relationship includes an AND relationship and an OR relationship.

[0078] If the logical relationship is an AND relationship, only when the metadata information of the source object satisfies all the matching conditions in the matching item, is the source object considered to be successfully matched with the filter rule.

[0079] If the logical relationship is an OR relationship, when the metadata information of the source object satisfies any matching condition in the matching item, the source object is considered to be successfully matched with the filter rule.

[0080] It should be noted that in the embodiment, one matching item includes only one kind of logical relationship. That is, the multiple matching conditions in the same matching item are either all AND relationships or all OR relationships, and AND and OR cannot be mixed.

[0081] The purpose of such arrangement is twofold: firstly, to reduce the complexity of the matching item. When the matching item contains multiple logical relationships, it will bring higher difficulty and complexity to the parsing of the matching item and the matching with the metadata information of the source object, which is not conducive to implementation and affects efficiency. Secondly, it is also to avoid the confusion of logical relationships. When AND and OR exist between three or more matching conditions at the same time, there may be ambiguity. If parentheses are used to clarify the logical relationship, it will further increase the complexity of the matching item, increase the parsing difficulty and reduce the parsing efficiency.

[0082] In addition, a complex matching item that simultaneously includes two logical relationships can also be disassembled into multiple simple matching items that only contain one logical relationship. That is, a complex filtering rule that contains multiple logical relationships can be implemented by multiple simple filtering rules of the same storage class.

[0083] Furthermore, since the number of filtering rules contained in the filtering strategy and the specific matching conditions contained in each filtering rule are not limited in the above embodiments, there may be a scenario in actual application in which the metadata information of a source object can hit different filtering rules of multiple matching items at the same time. At this time, if the storage classes contained in these hit filtering rules are the same, it is still good to solve. However, if the storage classes in these filtering rules are different, how to determine the specific storage of the source object to which storage pool, the present embodiment provides a possible implementation scheme:

[0084] Each filtering rule corresponds to a different priority. When the metadata information of the source object simultaneously satisfies the matching items in multiple filtering rules, the filtering rule with the highest priority is hit.

[0085] That is, the present embodiment also allocates a corresponding priority to each filtering rule when configuring the filtering rule, and the priority of each filtering rule is different. So that when the source object satisfies multiple arbitrary filtering rules, there is only one filtering rule with the highest priority, thereby determining that the only one filtering rule indicates the storage of the source object in the target site.

[0086] In addition, for the configuration of the priority, it can be known from the above scheme provided by the present embodiment that each filtering rule uniquely corresponds to a priority. That is, in the corresponding relationship, a filtering rule corresponds to a matching item, a storage class, and a priority. Therefore, when configuring the filtering strategy, a corresponding matching item, a storage class, and a priority can be configured as a group, that is, the configuration of a filtering rule is completed. If the filtering strategy exists in the form of a configuration file, the filtering rule is a specific configuration item in the configuration file that includes a matching item, a storage class, and a priority.

[0087] In addition, based on the priority of the filtering rules configured in the above embodiment, when the source object meets the hit condition of multiple filtering rules at the same time, the source object can still be accurately stored in the specified storage pool. However, the embodiment does not limit how the source object meets the unique filtering rule in step S12. In a possible implementation, all filtering rules can be matched with the metadata information of the source object. If multiple filtering rules are matched, the filtering rule with the highest priority is selected as the filtering rule hit in the present step S12.

[0088] However, in addition to the above implementation, the embodiment further provides another filtering rule hit scheme in step S12. The above step S12 specifically includes:

[0089] The metadata information of the source object is matched with the filtering rules in order from high to low according to the priority of the filtering rules. When any filtering rule is matched successfully, the subsequent matching is stopped, and the step S13 is performed.

[0090] The priority of different filtering rules is different.

[0091] That is, in the embodiment, all filtering rules are not directly matched with the metadata information of the source object. Instead, the filtering rules are sorted according to the priority, and then matched with the metadata information of the source object in order. At this time, if the source object meets the matching item of one of the filtering rules in the matching process, the source object directly hits the filtering rule, and does not match the other filtering rules with lower priority, even though the source object can also meet the matching item of the other filtering rules.

[0092] Based on the above, the matching scheme provided by the embodiment can also ensure that, when the source object is matched each time, the filtering rule with the highest priority among the filtering rules that can meet the matching item condition of the source object is hit. Such matching does not need to match all filtering rules each time. When one filtering rule meets the condition, it is hit, and subsequent matching is not needed, thereby further improving the matching efficiency.

[0093] On the other hand, the above embodiments are further implementation schemes provided by the filtering rules. In addition to this, the embodiment further provides another possible implementation. Before step S11, the above method further includes: Figure 2 as shown in the following table.

[0094] S10: determining whether the state of the source object storage class configuration item is an open state. If not, the step S11 is performed. If yes, the step S14 is performed.

[0095] The source object storage class configuration item is a configurable state bit, including two states of opening and closing.

[0096] S14: Obtain the object policy of the source site, and store the source object in the specified storage pool in the target site according to the object policy of the source site.

[0097] That is, the embodiment also provides a data synchronization scheme for the same application scenario of multi-site hard disk configuration. When the multi-site hard disk configurations are the same, the requirement of the user that the data in a certain site is synchronized to other sites in the same way according to the storage distribution mode of the data can be met. It should be noted that when the hard disk configurations of the sites are completely the same and the user needs to synchronize the data in the same storage distribution mode, the filtering mode provided in steps S11-S13 is no longer needed. As shown in step S14, the object policy of the source site is directly obtained. Since the hard disk configurations of the sites are the same, the storage pool associated with each storage class in the object policy of the source site also exists in the target site. Therefore, based on the object policy of the source site, the source object can be stored in the same storage pool in the target site, so as to meet the more diverse data synchronization requirements of the user.

[0098] In addition, whether to enable complete synchronization (the "complete" refers to that the storage pool in which the object data is stored is also completely the same) between the sites is controlled by a configuration item. The configuration item is equivalent to a switch, and the control of whether to enable complete synchronization can be realized by changing the state of the switch. When the complete synchronization is enabled, the complete synchronization of the source object in the target site is realized by step S14. When the complete synchronization is not enabled, the filtering synchronization of the source object in the target site is realized by steps S11-S13.

[0099] It should be noted that whether to enable the complete synchronization function needs to consider not only the requirement of the user, but also whether the hard disk configurations of the source site and the target site are the same. If the hard disk configurations of the source site and the target site are different, the complete synchronization function is prohibited to be enabled.

[0100] Further, the embodiment also provides another possible implementation scheme:

[0101] When the cross-site data synchronization with filtering rules is performed based on steps S11-S13, if the filtering policy obtained in step S11 is an empty set (no valid filtering policy is obtained, or the target site is not configured with a filtering policy), or the source object does not hit any filtering rule in step S12, as shown in step S15, the method further includes: Figure 2

[0102] S151: Obtain a default storage class.

[0103] ​The default storage class is a storage class automatically generated when the same-name object policy is created, and does not need to be manually set additionally.

[0104] S152: According to the default storage class, the source object is stored in the specified storage pool in the target site.

[0105] As can be seen from the above, the embodiment provides a possible implementation scheme for ensuring that the source object can be successfully synchronized to the target site when the filtering policy is invalid (no matter which one of the invalid conditions such as acquisition invalidity, non-configuration invalidity, or matching invalidity). The default storage class automatically generated when the same-name object policy is created in the multi-site scenario is used. The default storage class is a storage class contained in the same-name object policy of each site. That is, when the storage of the source object is specified to use the storage class, a specific storage pool can be determined in any site to implement the storage of the object data. The embodiment protects a data synchronization scheme in an extreme scenario, ensures that the cross-site synchronization of the object data can be implemented even if the filtering policy is invalid, and ensures the robustness of data synchronization.

[0106] On the other hand, the above embodiment does not limit where the filtering policy is configured to enable each site to synchronize data according to the distributed object storage system multi-site data synchronization method provided in the above embodiment when the object data of other sites is uploaded.

[0107] In a possible implementation, the filtering policy can be configured in the metadata of the bucket. At this time, if there is a master-slave relationship between the multi-sites, the master site has the need to automatically synchronize data to each slave site. Therefore, when the corresponding filtering policy is set for the bucket of the master site, the master site records the configuration in the metadata synchronization log. When each slave site synchronizes the bucket of the master site to the slave site according to the metadata synchronization log, the filtering policy of the master site will overwrite the filtering policy in the slave site, thereby causing the filtering invalidity.

[0108] To solve this problem, the embodiment provides a filtering policy synchronization prohibition scheme. Specifically, the scheme mainly includes a common part of each site and a characteristic part for the slave site. First, for the common part applied to each site, the method further includes:

[0109] S21: When the filtering policy is received, the storage bucket metadata of the current site cluster is acquired.

[0110] S22: The storage bucket metadata is parsed, and the filtering policy is added to the storage bucket metadata.

[0111] S23: The storage bucket metadata to which the filtering policy is added is written back to the current site cluster.

[0112] For the application of the method to the characteristic part of the slave site, when the slave site needs to synchronize the storage bucket from the master site, the method further comprises:

[0113] S31: Obtain the metadata synchronization log of the master site, and parse the obtained storage bucket metadata corresponding to the master site to serve as the first metadata.

[0114] S32: Erase the filtering strategy in the first metadata to obtain the second metadata.

[0115] S33: Load the storage bucket metadata of the current site cluster into the memory to serve as the third metadata.

[0116] S34: Parse the third metadata to obtain the metadata part of the filtering strategy corresponding to the current site to serve as the fourth metadata.

[0117] S35: Write the fourth metadata into the second metadata to obtain the fifth metadata.

[0118] S36: Write the fifth metadata into the current site cluster to complete the synchronization of the storage bucket.

[0119] Specifically, the steps S21-S23 described above are the steps for any site to make the filtering strategy effective after receiving the filtering strategy input by the user or the operation and maintenance personnel. The filtering strategy is written into the storage bucket metadata and written into the cluster of the current site to make it effective. Subsequently, when the current site receives the object data sent by other sites, the cluster can filter the object data according to the filtering strategy and store it in the specified storage pool.

[0120] The steps S31-S36 are specific schemes for avoiding synchronizing the filtering policy of the master site to the current site when the slave site synchronizes the bucket of the master site. When the metadata synchronization log of the master site is obtained, the metadata synchronization log is first parsed to obtain first metadata. The first metadata includes both the bucket metadata required for synchronizing the bucket of the master site and the filtering policy of the master site. Therefore, in step S32, the filtering policy of the first metadata is deleted. Since the filtering policy formats of the master site and the slave site are the same, the filtering policy of the master site can be determined from the first metadata. After the filtering policy of the master site is deleted from the first metadata, the remaining metadata is second metadata, that is, only the metadata required for synchronizing the bucket of the master site. If the second metadata is synchronized to the cluster of the current site, the bucket synchronization can be achieved. However, the second metadata does not include the filtering policy, and if only the second metadata is synchronized, the filtering policy of the current site is lost, which causes the distributed object storage system multi-site data synchronization method provided in the above embodiment to fail in the current site. Therefore, in step S33, the bucket metadata in the cluster of the current site is loaded into the memory. Based on the same parsing as in step S32, the filtering policy corresponding to the current site is determined, and in step S34, the metadata corresponding to the filtering policy of the current site is extracted, that is, fourth metadata is obtained. Then, the fourth metadata is written into the second metadata, that is, the bucket metadata of the master site + the bucket metadata of the current site filtering policy, that is, fifth metadata. Finally, based on step S36, the fifth metadata is written into the cluster of the current site, and the bucket synchronization of the master site and the slave site is completed, and only the filtering policy corresponding to the current site is retained in the slave site.

[0121] As known from the above, when the filtering policy takes effect, the filtering policy needs to be written into the bucket metadata of the cluster of the current site. However, when there is a master-slave relationship among multiple sites, the filtering policy of the slave site will be overwritten by the filtering policy of the master site when the slave site synchronizes the bucket of the master site, thereby implementing the method. Based on this, the embodiment provides a scheme for erasing the filtering policy of the master site and rewriting the filtering policy of the current site during bucket synchronization, which ensures that the filtering policy of the slave site is not overwritten by the filtering policy of the master site on the premise that the slave site can normally synchronize the bucket of the master site. That is, the method is ensured to be normally applied in the multiple-site scenario with a master-slave relationship.

[0122] In addition to the embodiment of the distributed object storage system multi-site data synchronization method provided in the above embodiment, the present application also provides an embodiment of a computer program product. The computer program product includes computer programs / instructions, which, when executed by a processor, can implement the steps of the distributed object storage system multi-site data synchronization method according to any one of the above embodiments.

[0123] Since the embodiments of the computer program product part correspond to the embodiments of the method part, the embodiments of the computer program product part refer to the description of the embodiments of the method part, which will not be repeated here.

[0124] In the above embodiments, the distributed object storage system multi-site data synchronization method is described in detail, and the present application also provides a corresponding embodiment of a distributed object storage system multi-site data synchronization device. It should be noted that the embodiments of the device part are described from two angles, one is based on the functional module, and the other is based on the hardware.

[0125] Based on the functional module, the present embodiment provides a distributed object storage system multi-site data synchronization device, as shown in Figure 3 , which comprises:

[0126] The acquisition module 11 is used to acquire the metadata information of the source object and the filtering strategy corresponding to the target site when it is necessary to synchronize the source object on the source site to the target site. The filtering strategy comprises a plurality of filtering rules, and each filtering rule comprises a set of matching items and a storage class corresponding to each other; the matching items comprise one or more metadata information.

[0127] The matching module 12 is used to match the metadata information of the source object with the matching items in each filtering rule.

[0128] The synchronization module 13 is used to store the source object in the designated storage pool in the target site according to the storage class in the filtering rule hit by the source object.

[0129] Since the embodiments of the device part correspond to the embodiments of the method part, the embodiments of the device part refer to the description of the embodiments of the method part, which will not be repeated here.

[0130] Figure 4 The structure diagram of a distributed object storage system multi-site data synchronization device provided by another embodiment of the present application is shown in Figure 4 , which comprises: a memory 20 for storing a computer program;

[0131] A processor 21 is used to execute the computer program to realize the steps of the distributed object storage system multi-site data synchronization method of the above embodiments.

[0132] The distributed object storage system multi-site data synchronization device provided by the present embodiment can include but is not limited to mobile terminals, personal computers, workstations, etc.

[0133] The processor 21 can include one or more processing cores, such as a 4-core processor, an 8-core processor, etc. The processor 21 can be implemented in at least one of a hardware form of a digital signal processor (DSP), a field-programmable gate array (FPGA), a programmable logic array (PLA), etc. The processor 21 can also include a main processor and a coprocessor. The main processor is a processor for processing data in a wake-up state, also known as a central processing unit (CPU). The coprocessor is a low-power processor for processing data in a standby state. In some embodiments, the processor 21 can be integrated with a graphics processor (GPU) for rendering and drawing content to be displayed by the display screen. In some embodiments, the processor 21 can further include an artificial intelligence (AI) processor for processing machine learning-related computing operations.

[0134] The memory 20 can include one or more computer-readable storage media, which can be non-transitory. The memory 20 can further include a high-speed random access memory, and a nonvolatile memory such as one or more disk storage devices, flash storage devices. In this embodiment, the memory 20 is at least used to store the following computer program 201, wherein the computer program is loaded and executed by the processor 21, and can implement the related steps of the distributed object storage system multi-site data synchronization method disclosed in any of the preceding embodiments. In addition, the resources stored by the memory 20 can further include an operating system 202 and data 203, etc., and the storage mode can be temporary storage or permanent storage. The operating system 202 can include Windows, Unix, Linux, etc. The data 203 can include but is not limited to a distributed object storage system multi-site data synchronization method, etc.

[0135] In some embodiments, the distributed object storage system multi-site data synchronization apparatus can further include a display screen 22, an input / output interface 23, a communication interface 24, a power supply 25, and a communication bus 26.

[0136] Those skilled in the art can understand that the structure shown in the above embodiments does not constitute a limitation on the distributed object storage system multi-site data synchronization apparatus, and can include more or fewer components than those shown in the drawings. Figure 4 The structure shown in the above embodiments does not constitute a limitation on the distributed object storage system multi-site data synchronization apparatus, and can include more or fewer components than those shown in the drawings.

[0137] The embodiment of the present application provides a kind of distributed object storage system multi-site data synchronization device, including memory and processor, processor when executing the program stored in memory, can realize the following method: a kind of distributed object storage system multi-site data synchronization method.

[0138] Finally, the present application also provides a kind of nonvolatile storage medium corresponding embodiment. Nonvolatile storage medium stores computer program, computer program is executed by processor to realize the steps as described in the above method embodiment.

[0139] It can be understood that if the method in the above embodiment is realized in the form of software function unit and is sold or used as an independent product, it can be stored in a nonvolatile storage medium. Based on such understanding, the technical solutions of the present application essentially or the part that contributes to the prior art or the whole or part of the technical solutions can be embodied in the form of software product, and the computer software product is stored in a storage medium, and executes all or part of the steps of the method described in each embodiment of the present application. The foregoing storage medium includes: U disk, mobile hard disk, read-only memory (Read-Only Memory, ROM), random access memory (Random Access Memory, RAM), magnetic disk or optical disk and various program code storage media.

[0140] The above provides a kind of distributed object storage system multi-site data synchronization method, device and medium provided by the present application are introduced in detail. The embodiments in the specification are described in a progressive manner, and each embodiment mainly describes the difference from other embodiments, and the same or similar parts of each embodiment can be referred to each other. For the device disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, it is described simply, and the related part is described in the method part. It should be pointed out that, for ordinary skilled in the art, without departing from the principles of the present application, the present application can be improved and modified, and these improvements and modifications also fall within the scope of the present application.

[0141] It also needs to be explained that in the present specification, the relational terms such as first and second and the like are used solely to distinguish one entity or action from another entity or action without necessarily requiring or implying any actual such relationship or order between such entities or actions. Moreover, the terms "comprises", "comprising", or any other variations thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but can include other elements not expressly listed or inherent to such process, method, article, or apparatus. An element proceeded by "comprises... a" does not, without more constraints, exclude the existence of additional identical elements in the process, method, article, or apparatus that comprises the element.

Claims

1. A method for synchronizing multi-site data in a distributed object storage system, characterized in that: The method comprises the following steps: When it is needed to synchronize a source object on a source site to a target site, metadata information of the source object and a filtering strategy corresponding to the target site are acquired; the filtering strategy comprises a plurality of filtering rules; each filtering rule comprises a set of matching items and a storage class corresponding to each other; the storage class is associated with any storage pool in the target site; the matching items comprise one or more kinds of metadata information; The metadata information of the source object is matched with the matching items in each filtering rule; wherein the matching of the metadata information of the source object with the matching items in each filtering rule comprises: According to the priority of each filtering rule, the metadata information of the source object is matched in turn from high to low; wherein the priorities of different filtering rules are different; When any filtering rule is matched successfully, subsequent matching is stopped, and the source object is stored in a specified storage pool in the target site according to the storage class in the matched filtering rule of the source object; The source object is stored in a specified storage pool in the target site according to the storage class in the matched filtering rule of the source object.

2. The method of claim 1, wherein, The matching items comprise a matching condition corresponding to any kind of metadata information, or a plurality of matching conditions corresponding to the same or different kinds of metadata information; Wherein, the kinds of metadata information include object prefix, object suffix, object size, object type, object label, metadata, and object owner.

3. The method of claim 2, wherein, When the matching items comprise a plurality of matching conditions, the logical relationship between each matching condition is also included; Wherein, the logical relationship includes and relationship and or relationship; If the logical relationship is and relationship, only when the metadata information of the source object satisfies all the matching conditions in the matching items, it is considered that the source object is matched with the filtering rule successfully; If the logical relationship is or relationship, when the metadata information of the source object satisfies any matching condition in the matching items, it is considered that the source object is matched with the filtering rule successfully.

4. The method of claim 1, wherein, Before acquiring the filtering strategy corresponding to the target site, the method further comprises: Judging whether the state of the source object storage class configuration item is an open state; wherein the source object storage class configuration item is a configurable state bit, including two states of open and close; If not, go to the step of acquiring the filtering strategy corresponding to the target site; If yes, acquire the object strategy of the source site, and store the source object in a specified storage pool in the target site according to the object strategy of the source site.

5. The method of claim 1, wherein, If the filtering strategy acquired in the step of acquiring the filtering strategy corresponding to the target site is an empty set, or no filtering rule is matched in the step of matching the metadata information of the source object with the matching items in each filtering rule, the method further comprises: Acquiring a default storage class; wherein the default storage class is a storage class contained in each site; Storing the source object in a specified storage pool in the target site according to the default storage class.

6. The method of claim 1 to 5, wherein, The method applied to any site further comprises: When the filtering policy is received, obtaining the bucket metadata of the current site cluster; Parsing the bucket metadata and adding the filtering policy to the bucket metadata; Re-writing the bucket metadata after adding the filtering policy into the current site cluster; The method applied to the slave site further comprises: Obtaining the metadata synchronization log of the master site and parsing to obtain the bucket metadata corresponding to the master site as the first metadata; Erasing the filtering policy in the first metadata to obtain the second metadata; Loading the bucket metadata of the current site cluster into the memory as the third metadata; Parsing the third metadata to obtain the metadata part of the filtering policy corresponding to the current site as the fourth metadata; Writing the fourth metadata into the second metadata to obtain the fifth metadata; Writing the fifth metadata into the current site cluster to complete the bucket synchronization.

7. A distributed object storage system multi-site data synchronization apparatus, characterized by, Comprise: The obtaining module is used for obtaining the metadata information of the source object and the filtering policy corresponding to the target site when the source object on the source site needs to be synchronized to the target site; The filtering policy comprises a plurality of filtering rules, and each filtering rule comprises a set of matching items and a storage class corresponding to each other; the matching items comprise one or more metadata information; The matching module is used for sequentially matching the metadata information of the source object in descending order according to the priority of each filtering rule; wherein the priority of different filtering rules is different; when any filtering rule is matched successfully, the subsequent matching is stopped, and the synchronization module is turned to; The synchronization module is used for storing the source object in the specified storage pool in the target site according to the storage class in the filtering rule hit by the source object.

8. A distributed object storage system multi-site data synchronization apparatus, characterized by, Comprise: The memory is used for storing the computer program; The processor is used for executing the computer program to realize the steps of the multi-site data synchronization method of the distributed object storage system according to any one of claims 1 to 6.

9. A non-volatile storage medium, comprising: The non-volatile storage medium stores the computer program, and the computer program is executed by the processor to realize the steps of the multi-site data synchronization method of the distributed object storage system according to any one of claims 1 to 6.

Images