Distributed object storage system multi-site data synchronization method and device and medium

Abstract

The invention discloses a distributed object storage system multi-site data synchronization method and device and a medium, relates to the technical field of distributed storage, and is used for realizing data synchronization between sites. The method is provided for solving the problem that according to an existing data synchronization scheme, when hard disk configurations between sites are different, synchronization cannot be achieved or imbalance exists. A filtering strategy is configured for each site, the filtering strategy comprises a plurality of filtering rules, and each filtering rule comprises a storage class associated with a storage pool in a target site. And when data synchronization needs to be carried out, matching is carried out according to the metadata of the source object and the filtering rule corresponding to the target site. And the storage pool of the source object is specified according to the storage class in the successfully matched filtering rule, so that accurate data synchronization between the sites is realized. Besides, based on the setting of the filtering rule, the source object can be allocated to any storage pool of the target site based on different requirements, and the problem of data imbalance is avoided.

Description

Technical Field

[0001] The present invention relates to the field of distributed storage technology, and in particular to a multi-site data synchronization method, device and medium for a distributed object storage system. Background Art

[0002] For a distributed object storage system, when you create a bucket in the system, you must specify an object policy. The object policy defines storage classes (or storage categories), which are associated with storage pools within a site. Furthermore, when an object is uploaded to a bucket, the user can specify a storage class to specify that the object data is stored in a certain storage pool.

[0003] When creating an object policy in a multi-site environment, you need to create an object policy with the same name on each site in turn before you can use the object policy to create a bucket on the master site, and then the bucket will be synchronized to each slave site. However, in actual applications, the configurations of each site are not exactly the same. For example, if the disk configurations of different sites are different, the storage classes contained in the objects with the same name are also different.

[0004] Assume that if an object needs to be synchronized to site B after being written to site A, and the storage class of the object in site A is Aclass1, but there is no storage class Aclass1 in site B, then data synchronization will fail. If a fixed storage class is specified in advance to synchronize this part of data that does not have the same storage class, then all this data will be written to the fixed storage class, resulting in data imbalance, which will affect system performance.

[0005] Therefore, technicians in this field are in urgent need of a multi-site data synchronization method for a distributed object storage system to solve the problem of how to achieve data synchronization between sites when the site configurations are inconsistent and ensure the balance of data synchronization. Summary of the invention

[0006] The purpose of the present invention is to provide a distributed object storage system multi-site data synchronization method, device and medium to solve the problem of how to achieve data synchronization between sites when the site configurations are inconsistent and to ensure the balance of data synchronization.

[0007] In order to solve the above technical problems, the present invention provides a distributed object storage system multi-site data synchronization method, comprising:

[0008] When it is necessary to synchronize the source object on the source site to the target site, the metadata information of the source object and the filtering policy corresponding to the target site are obtained; wherein the filtering policy includes a plurality of filtering rules; each filtering rule includes: a set of matching items and storage classes corresponding to each other; the storage class is associated with any storage pool in the target site; the matching items include one or more metadata information;

[0009] Matching the metadata information of the source object with the matching items in each of the filtering rules;

[0010] According to the storage class in the filtering rule matched by the source object, the source object is stored in a designated storage pool in the target site.

[0011] In a possible embodiment, matching the metadata information of the source object with the matching item in each of the filtering rules includes:

[0012] According to the priority of each filtering rule, matching is performed with the metadata information of the source object in order from high to low; wherein different filtering rules have different priorities;

[0013] When any of the filtering rules is matched successfully, subsequent matching is stopped, and the process proceeds to the step of storing the source object in a designated storage pool in the target site according to the storage class in the filtering rule matched by the source object.

[0014] In a possible embodiment, the matching item includes: a matching condition corresponding to any kind of metadata information, or a plurality of matching conditions corresponding to the same or different kinds of metadata information;

[0015] The types of metadata information include: object prefix, object suffix, object size, object type, object tag, metadata, and object owner.

[0016] In a possible embodiment, when the matching item includes a plurality of the matching conditions, the matching conditions further include: a logical relationship;

[0017] Wherein, the logical relationship includes: and relationship, or relationship;

[0018] If the logical relationship is an AND relationship, the source object is considered to successfully match the filtering rule only when the metadata information of the source object satisfies all the matching conditions in the matching items;

[0019] If the logical relationship is an OR relationship, then when the metadata information of the source object satisfies any of the matching conditions in the matching items, it is considered that the source object successfully matches the filtering rule.

[0020] In a possible embodiment, before obtaining the filtering policy corresponding to the target site, the method further includes:

[0021] Determine whether the state of the source object storage class configuration item is an open state; wherein the source object storage class configuration item is a configurable state bit, including two states of open and closed;

[0022] If not, proceed to the step of obtaining a filtering policy corresponding to the target site;

[0023] If so, the object policy of the source site is obtained, and the source object is stored in a designated storage pool in the target site according to the object policy of the source site.

[0024] In a possible embodiment, if the filtering policy obtained in the step of obtaining the filtering policy corresponding to the target site is an empty set, or any filtering rule is not hit in the step of matching the metadata information of the source object with the matching items in each of the filtering rules, the method further includes:

[0025] Obtain a default storage class; wherein the default storage class is a storage class included in each site;

[0026] According to the default storage class, the source object is stored in a designated storage pool in the target site.

[0027] In a possible embodiment, applied to any site, the method further includes:

[0028] When receiving the filtering policy, obtaining bucket metadata of the current site cluster;

[0029] Parsing the bucket metadata and adding the filtering policy to the bucket metadata;

[0030] Rewrite the bucket metadata after adding the filtering policy into the current site cluster;

[0031] This method is applied to a slave site. When the slave site needs to synchronize a bucket from a master site, the method further includes:

[0032] Obtaining the metadata synchronization log of the primary site, and parsing to obtain the bucket metadata corresponding to the primary site as the first metadata;

[0033] erasing the filtering policy in the first metadata to obtain second metadata;

[0034] Loading the bucket metadata of the current site cluster into the memory as the third metadata;

[0035] Parsing the third metadata to obtain a metadata portion of the filtering policy corresponding to the current site as fourth metadata;

[0036] writing the fourth metadata into the second metadata to obtain fifth metadata;

[0037] The fifth metadata is written into the current site cluster to complete the bucket synchronization.

[0038] In order to solve the above technical problems, the present invention further provides a distributed object storage system multi-site data synchronization device, comprising:

[0039] An acquisition module, used for acquiring metadata information of the source object and a filtering policy corresponding to the target site when it is necessary to synchronize the source object on the source site to the target site; the filtering policy includes a plurality of filtering rules, each of which includes: a set of matching items and storage classes corresponding to each other; the matching items include one or more metadata information;

[0040] A matching module, used for matching the metadata information of the source object with the matching items in each of the filtering rules;

[0041] A synchronization module is used to store the source object in a designated storage pool in the target site according to the storage class in the filtering rule hit by the source object.

[0042] In order to solve the above technical problems, the present invention further provides a distributed object storage system multi-site data synchronization device, comprising:

[0043] Memory for storing computer programs;

[0044] The processor is used to implement the steps of the distributed object storage system multi-site data synchronization method as described above when executing the computer program.

[0045] To solve the above technical problem, the present invention also provides a non-volatile storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of the multi-site data synchronization method of the distributed object storage system as described above are implemented.

[0046] The present invention provides a method for synchronizing data in multiple sites of a distributed object storage system. In a distributed object storage system multi-site scenario, a corresponding filtering strategy is configured for each site. The filtering strategy includes multiple filtering rules, and each filtering rule includes a storage class associated with a storage pool in a target site. Furthermore, when object data needs to be synchronized between sites, the metadata of the source object is matched with the corresponding filtering rules of the target site. The storage pool of the source object is specified according to the storage class in the successfully matched filtering rule, ensuring that even if the object strategies of the source site and the main site are different, accurate data synchronization between sites can still be achieved. In addition, based on the setting of the filtering rules, different matching items can be set for each storage pool of the target site, so that source objects with different metadata can be allocated to any storage pool of the target site based on different needs, avoiding the problem of data imbalance caused by all synchronized data being allocated to the same storage pool.

[0047] The distributed object storage system multi-site data synchronization device and non-volatile storage medium provided by the present invention correspond to the above method and have the same effect as above. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] In order to more clearly illustrate the embodiments of the present invention, the following briefly introduces the drawings required for use in the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0049] Figure 1 A flowchart of a multi-site data synchronization method for a distributed object storage system provided by an embodiment of the present invention;

[0050] Figure 2 A flowchart of another distributed object storage system multi-site data synchronization method provided by an embodiment of the present invention;

[0051] Figure 3 A structural diagram of a multi-site data synchronization device for a distributed object storage system provided by an embodiment of the present invention;

[0052] Figure 4 A structural diagram of another distributed object storage system multi-site data synchronization device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0053] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0054] The core of the present invention is to provide a method, device and medium for synchronizing multi-site data in a distributed object storage system.

[0055] In order to enable those skilled in the art to better understand the scheme of the present invention, the present invention is further described in detail below in conjunction with the accompanying drawings and specific implementation methods.

[0056] In a distributed object storage system, multiple sites can be deployed. A site is an independent object storage system with its own storage gateway and storage pool. A site can have multiple object storage gateway instances, which can independently provide external business access.

[0057] When creating an object policy in a multi-site environment, you need to create an object policy with the same name on each site in turn, and finally use the object policy to create a bucket on the master site, and then the bucket will be synchronized to each slave site. If the bucket is synchronized to the slave site, if it is found that there is no object policy with the same name used by the bucket on the slave site, the bucket will fail to synchronize from this slave site. The object policy defines the index pool, classification pool, and data pool respectively, of which there is only one index pool and classification pool, while there can be multiple data pools, each of which is associated with a storage class. Based on this, by creating a storage class and specifying the storage class when uploading the object, the user can upload the object to different data pools in the object storage cluster to achieve data filtering of the object on the business side.

[0058] In a multi-site scenario, when creating an object policy, it is only required that all sites must create an object policy with the same name. There is no restriction on the specific storage pool specified in the object policy, nor is there a requirement that the storage class associated with the data pool in the same object policy on the master and slave sites be the same. Therefore, when the disk configurations between sites are different, problems such as data synchronization failure between sites may occur.

[0059] For example, assume that there are storage pools Apool1, Apool2, and Apool3 on site A. There is also an object policy policy, which creates a total of three storage classes Aclass1, Aclass2, and Aclass3. Among them, Aclass1 is associated with Apool1, Aclass2 is associated with Apool2, and Aclass3 is associated with Apool3. There are storage pools Bpool1, Bpool2, Bpool3, and Bpool4 on site B. There is also an object policy policy with the same name, which creates a total of four storage classes Bclass1 to Bclass4. Among them, Bclass1 is associated with Bpool1, Bclass2 is associated with Bpool2, Bclass3 is associated with Bpool3, and Bclass4 is associated with Bpool4.

[0060] In a multi-site environment, when an object is written to a site, a synchronization log is recorded. Other sites synchronize the newly written objects to each target site by comparing the synchronization log. However, data synchronization is an automatic backup function. Users are not aware of the data synchronization process, and there is no way to specify storage classes to write different objects to different data pools when data is synchronized to the target site. If the object is synchronized to site B according to the storage class of the source object, such as Aclass1, it is still stored according to the source storage class. Since there is no storage class Aclass1 on site B, data synchronization will fail. If data is written uniformly according to a fixed storage class such as Bclass1, then when the data on site A is synchronized to site B, it will all be written to the fixed data pool Bpool1 on site B, and no synchronization objects will be written to the other three data pools, resulting in data imbalance and poor performance.

[0061] To solve the above problems, the present invention provides a distributed object storage system multi-site data synchronization method, such as Figure 1 As shown, including:

[0062] S11: When it is necessary to synchronize the source object on the source site to the target site, metadata information of the source object and a filtering policy corresponding to the target site are obtained.

[0063] The filtering strategy includes multiple filtering rules. Each filtering rule includes: a set of corresponding matching items and storage classes. The storage class is associated with any storage pool in the target site. The matching items include one or more metadata information.

[0064] S12: Match the metadata information of the source object with the matching items in each filtering rule.

[0065] S13: According to the storage class in the filtering rule matched by the source object, the source object is stored in a designated storage pool in the target site.

[0066] For step S11, the matching items in the filtering rules are specifically used to achieve matching with the metadata information of the source object. Among them, the matching items include one or more metadata information, which are matched with the corresponding types of metadata information in all metadata information of the source object in step S12. When the metadata information of the source object meets the matching items set in a certain filtering rule, it is considered that the source object hits this filtering rule. At this time, step S13 can store the source object in the storage pool associated with the storage class in the target site according to the storage class in the hit filtering rule.

[0067] It is easy to understand that the filtering policy corresponds to the site one-to-one. And all the filtering rules contained in the filtering policy corresponding to the site, and the storage classes therein are associated with each storage pool of the site. That is, as long as the source object hits any filtering rule in the filtering policy corresponding to the target site, the source object can be stored in a storage pool specified in the target site according to the storage class in the hitting filtering rule, ensuring that data synchronization between sites can be achieved.

[0068] In addition, the matching items in the filtering rules are the conditions for filtering the source objects. As can be seen from the above, a site usually contains multiple storage pools. Therefore, how to store the synchronized object data (i.e., source objects) in different storage pools in the site in a more balanced manner is a key to achieving balanced data storage in the storage pools. This method implements this process through the matching items in the filtering rules. A filtering rule includes a matching item and a storage class. Therefore, when the source object hits a matching item (i.e., the source object meets a certain condition), it will be stored in the specified storage pool according to the storage class of the same filtering rule as the matching item. When the filtering policy contains multiple filtering rules with different storage classes and matching items, balanced data synchronization between sites can be achieved.

[0069] However, it should be noted that among multiple filter rules belonging to the same filter policy, two filter rules with exactly the same matching items are not allowed. If the matching items are the same and the storage class is also the same, the two filter rules are set repeatedly and meaningless. If the matching items are the same but the storage class is different, the two filter rules are set in conflict, which will cause storage confusion for source objects that hit this matching item, causing more serious problems.

[0070] In summary, the multi-site data synchronization method of a distributed object storage system provided by the present invention sets a filtering strategy for each site. The filtering strategy includes multiple filtering rules, and each filtering rule includes a matching item and a storage class. The storage class is associated with a storage pool of the corresponding site to ensure that when the filtering rule is hit, the source object can be accurately stored in a storage pool of the target site through the storage class in the filtering rule to achieve data synchronization. The matching item provides corresponding filtering conditions for multiple different storage pools in the target site, that is, the object data can only be stored in the storage pool when the filtering condition of a storage pool is met (that is, the corresponding matching item is hit). That is, on-demand allocation of storage in different storage pools can be achieved, which can meet various synchronization requirements of users. In addition, the method can also achieve balanced storage of data synchronization between multiple sites, enhancing the robustness of the multi-site data synchronization function of object storage.

[0071] On the other hand, in the above embodiments, the specific implementation of the matching items is not limited. In practical applications, based on different needs for storing objects in different storage pools within a site, appropriate matching items can be freely set to store object data that meets specific conditions in a specific storage pool.

[0072] However, this embodiment also provides a possible implementation scheme for the matching items, and the above matching items may specifically include: a matching condition corresponding to any kind of metadata information, or multiple matching conditions corresponding to the same or different kinds of metadata information.

[0073] The types of metadata information include: object prefix, object suffix, object size, object type, object tag, metadata, and object owner.

[0074] It should be noted that one matching condition corresponds to only one type of metadata information. The specific content of the matching condition should be determined according to the type of metadata information. Exemplarily, for metadata information such as object size represented by a numerical value, the corresponding matching condition can be a numerical value or an array, or a continuous or discontinuous numerical range, which can have only an upper limit or a lower limit, or both. For metadata information such as object type represented by a string of specific content, the corresponding matching condition can be a specific object type, and the match is hit when the object type of the source object is consistent with the matching condition. Similarly to the object type, for metadata types such as object prefix and object suffix, the corresponding matching condition can be a specific prefix name or suffix name. If the prefix name or suffix name of the source object includes the prefix name or suffix name set in the matching condition, the match item is hit. In addition, the object label refers to the object label set by the matching condition included in the label of the source object; metadata refers to the custom metadata set by the matching condition included in the custom metadata of the source object; the object owner refers to the owner of the source object being consistent with the object owner set by the matching condition.

[0075] In this embodiment, a matching item of a filtering rule may include one or more matching conditions. When multiple matching conditions are involved, each matching condition may involve the same or different types of metadata information, so as to achieve more flexible and comprehensive matching and meet the more diverse filtering needs of users.

[0076] Furthermore, when a matching item of a filtering rule contains multiple matching conditions, this embodiment also provides a corresponding possible implementation scheme:

[0077] The matching conditions also include: logical relationships, wherein the logical relationships include: AND relationship (AND) and OR relationship (OR).

[0078] If the logical relationship is an AND relationship, the source object is considered to match the filtering rule successfully only when the metadata information of the source object meets all the matching conditions in the matching items.

[0079] If the logical relationship is an OR relationship, when the metadata information of the source object meets any of the matching conditions in the matching items, it is considered that the source object successfully matches the filtering rule.

[0080] It should be noted that, in this embodiment, a matching item contains only one logical relationship, that is, multiple matching conditions in the same matching item are either all in an AND relationship or all in an OR relationship, and AND and OR cannot be mixed.

[0081] There are two purposes for this setting: First, it reduces the complexity of the matching items. When the matching items contain multiple logical relationships, it will bring higher difficulty and complexity to the parsing of the matching items and the matching with the source object metadata information, which is not conducive to implementation and affects efficiency. The second is to avoid confusion in logical relationships. When AND and OR exist between three or more matching conditions at the same time, there may be ambiguity. If brackets are used to clarify the logical relationship, the complexity of the matching items will be further increased, the difficulty of parsing will be increased, and the parsing efficiency will be reduced.

[0082] In addition, a complex match item that includes two logical relationships at the same time can also be decomposed into multiple simple match items that only include one logical relationship. That is, a complex filtering rule in which one match item includes multiple logical relationships can be implemented through multiple simple filtering rules with the same storage class.

[0083] Furthermore, since the above embodiment does not limit the number of filter rules contained in the filter policy, and the specific matching conditions contained in the matching items of each filter rule. Therefore, in actual applications, a scenario may occur in which the metadata information of a source object can hit multiple filter rules with different matching items at the same time. At this time, if the storage classes contained in these hit filter rules are the same, it is easy to solve. However, if the storage classes in these filter rules are different, how to determine which storage pool the source object is specifically stored in? This embodiment provides a possible implementation plan for this:

[0084] Each filtering rule has a different priority. When the metadata information of the source object satisfies the matching items in multiple filtering rules at the same time, the filtering rule with the highest priority is hit.

[0085] That is, when configuring the filtering rules, this embodiment also assigns a corresponding priority to each filtering rule, and each filtering rule has a different priority, so that when the source object satisfies multiple arbitrary filtering rules, there is only one filtering rule with the highest priority, thereby determining that the only filtering rule indicates the storage of the source object in the target site.

[0086] In addition, regarding the configuration of priorities. It can be seen from the solution provided in the above embodiment that each filtering rule uniquely corresponds to a priority. That is, in terms of the corresponding relationship, a filtering rule corresponds to a match item, a storage class and a priority. Therefore, when configuring the filtering policy, a corresponding match item, a storage class and a priority can be configured as a group, that is, the configuration of a filtering rule is completed. If the filtering policy exists in the form of a configuration file, the filtering rule is a group of specific configuration items in the configuration file that includes a match item, a storage class and a priority.

[0087] In addition, based on the priorities configured for the filtering rules in the above embodiment, it can be achieved that when the source object satisfies the hit conditions of multiple filtering rules at the same time, the source object can still be accurately stored in a specified storage pool. However, this embodiment does not limit how the source object specifically hits the only filtering rule in step S12. In a possible implementation scheme, all filtering rules can be matched with the metadata information of the source object. If multiple filtering rules are matched, the filtering rule with the highest priority is selected as the filtering rule hit in this step S12.

[0088] However, in addition to the above implementation scheme, this embodiment also provides another filtering rule hit scheme of step S12, and the above step S12 specifically includes:

[0089] According to the priority of each filtering rule, the metadata information of the source object is matched in descending order. When any filtering rule is matched successfully, the subsequent matching is stopped and the process goes to step S13.

[0090] Among them, different filtering rules have different priorities.

[0091] That is, in this embodiment, all filtering rules are not directly matched with the metadata information of the source object. Instead, the filtering rules are sorted according to the priority level, and then matched with the metadata information of the source object in sequence. At this time, if the source object meets the matching items of one of the filtering rules during the matching process, the source object directly hits this filtering rule and no longer matches other subsequent filtering rules with lower priorities, even if there may be matching items that the source object can meet in these filtering rules.

[0092] Based on the above, the matching scheme provided in this embodiment can also ensure that each time the source object is matched, the filter rule with the highest priority among the filter rules that the source object can meet the matching condition is hit. In addition, such matching does not require matching all filter rules every time. When one filter rule meets the condition, it is hit, and no subsequent matching is required, thereby further improving the matching efficiency.

[0093] On the other hand, the above embodiments are all further implementation plans provided around the filtering rules. In addition, this embodiment also provides another possible implementation plan. Before step S11 of the above method, Figure 2 As shown, it also includes:

[0094] S10: Determine whether the state of the source object storage class configuration item is enabled; if not, go to step S11; if so, go to step S14.

[0095] The source object storage class configuration item is a configurable status bit, including two states: on and off.

[0096] S14: Obtain the object policy of the source site, and store the source object in a designated storage pool in the target site according to the object policy of the source site.

[0097] That is, this embodiment also provides a data synchronization solution for application scenarios where the hard disks of multiple sites are configured the same. When the hard disks of multiple sites are configured the same, it can meet the needs of users to synchronize the data in a certain site to other sites in exactly the same way according to the storage distribution method of the data. It should be noted that when the hard disk configurations between sites are exactly the same and the user needs to synchronize data in the same storage distribution method, the filtering method provided in the above steps S11~S13 is no longer required. As shown in step S14, the object policy of the source site is directly obtained, because the hard disk configurations between the sites are the same. Therefore, the storage pool associated with each storage class in the object policy of the source site also exists in the target site. Therefore, based on the object policy of the source site, the source object can be stored in the same storage pool in the target site to meet the user's more diverse data synchronization needs.

[0098] In addition, in this solution, whether to enable full synchronization between sites ("full" means that the storage pools storing the object data are also completely the same) is controlled by a configuration item. This configuration item is equivalent to a switch, and whether to enable full synchronization can be controlled by changing the switch state. When full synchronization is enabled, the method implements full synchronization of the source object at the target site through step S14. When full synchronization is not enabled, the method implements filtered synchronization of the source object at the target site through steps S11 to S13.

[0099] It should be noted that whether to enable the full synchronization function depends not only on user needs, but also on whether the hard disk configurations of the source and target sites are the same. If the hard disk configurations of the source and target sites are different, the full synchronization function cannot be enabled.

[0100] Furthermore, this embodiment also provides another possible implementation scheme:

[0101] When cross-site data synchronization with filtering rules is performed based on steps S11 to 13, if the filtering policy obtained in step S11 is an empty set (no valid filtering policy is obtained, or the target site is not configured with a filtering policy), or the source object in step S12 does not hit any filtering rule, such as Figure 2 As shown, the method also includes:

[0102] S151: Obtain a default storage class.

[0103] The default storage class is included in all sites. The default storage class is automatically generated when the object policy with the same name is created in each site, and no additional manual settings are required.

[0104] S152: According to the default storage class, the source object is stored in a designated storage pool in the target site.

[0105] From the above, it can be seen that this embodiment provides a possible implementation plan for how to ensure that the source object can be smoothly synchronized to the target site when the filtering policy fails (whether due to acquisition failure, failure due to non-configuration, or failure due to matching). The default storage class generated by default when creating an object policy with the same name in a multi-site scenario is utilized. The default storage class is a storage class included in the object policy with the same name of each site. That is, when the storage class is specified for the storage of the source object, a specific storage pool can be determined in any site to implement the storage of the object data. This embodiment protects a data synchronization solution in extreme scenarios, ensuring that even if the filtering policy fails, cross-site synchronization of object data can still be achieved, thereby ensuring the robustness of data synchronization.

[0106] On the other hand, the above embodiment does not limit where the filtering policy is specifically configured so that each site can synchronize data according to the distributed object storage system multi-site data synchronization method provided by the above embodiment when object data from other sites is uploaded.

[0107] In a possible implementation, the filtering policy can be configured in the metadata of the bucket. At this time, if there is a master-slave relationship between multiple sites, the master site has the need to automatically synchronize data to each slave site. Therefore, when the corresponding filtering policy is set for the bucket of the master site, the master site will record this part of the configuration in the metadata synchronization log. Later, when each slave site synchronizes the bucket of the master site to the slave site according to the metadata synchronization log, the filtering policy of the master site will overwrite the filtering policy in the slave site, resulting in filtering failure.

[0108] To solve this problem, this embodiment provides a filtering policy to prohibit synchronization solution. Specifically, this solution is mainly divided into a common part for each site and a characteristic part for the slave site. First, for the common part applied to each site, this method also includes:

[0109] S21: When the filtering policy is received, the bucket metadata of the current site cluster is obtained.

[0110] S22: Parse the bucket metadata and add the filtering policy to the bucket metadata.

[0111] S23: rewrite the bucket metadata after adding the filtering policy into the current site cluster.

[0112] For the feature part of the method applied to the slave site, when the slave site needs to synchronize the bucket from the master site, the method further includes:

[0113] S31: Obtain the metadata synchronization log of the primary site, and parse to obtain the bucket metadata corresponding to the primary site as the first metadata.

[0114] S32: Erasing the filtering policy in the first metadata to obtain second metadata.

[0115] S33: Load the bucket metadata of the current site cluster into the memory as the third metadata.

[0116] S34: Parse the third metadata and obtain the metadata portion of the filtering policy corresponding to the current site as the fourth metadata.

[0117] S35: Write the fourth metadata into the second metadata to obtain fifth metadata.

[0118] S36: Write the fifth metadata into the current site cluster to complete the bucket synchronization.

[0119] Specifically, the above steps S21 to S23 are steps for any site to make the filtering policy effective after receiving the filtering policy entered by the user or operation and maintenance personnel. The filtering policy is written into the bucket metadata and written into the cluster of the current site to make it effective. When this site subsequently receives object data sent by other sites, the cluster can filter the object data according to the filtering policy and store it in the specified storage pool.

[0120] Steps S31 to S36 are specific solutions for avoiding synchronizing the filtering policy of the master site to the current site when the slave site synchronizes the bucket from the master site. When the metadata synchronization log of the master site is obtained, it is first parsed to obtain the first metadata. The first metadata includes both the bucket metadata required for synchronizing the bucket of the master site and the filtering policy of the master site. Therefore, in step S32, the filtering policy of the first metadata is deleted. Since the filtering policy format corresponding to both the master site and the slave site is the same, it is not difficult to parse and determine from the first metadata which part of the metadata the filtering policy of the master site specifically corresponds to. After erasing the filtering policy of the master site in the first metadata, the remaining metadata is the second metadata, that is, it only includes the metadata required for synchronizing the bucket of the master site. If the second metadata is synchronized to the cluster of the current site, bucket synchronization can be achieved. However, the second metadata does not include the filtering policy. If only the second metadata is synchronized, the current site loses the filtering policy, resulting in the failure of the multi-site data synchronization method of the distributed object storage system provided in the above embodiment at the current site. Therefore, step S33 in this method loads the bucket metadata in the cluster of the current site into the memory. Based on the same analysis as step S32, the filtering policy corresponding to the current site is determined, and step S34 extracts part of the metadata corresponding to the filtering policy of the current site, that is, the fourth metadata is obtained. After that, the fourth metadata is written into the second metadata, that is, the bucket metadata of the main site bucket metadata + the current site filtering policy is obtained, that is, the fifth metadata. Finally, based on step S36, the fifth metadata is written into the current site cluster, and the bucket synchronization of the master and slave sites can be completed, and only the filtering policy corresponding to the current site is retained in the slave site.

[0121] From the above, it can be seen that when the configured filtering policy takes effect, the filtering policy needs to be written into the bucket metadata of the current site cluster. However, when there is a master-slave relationship among multiple sites, when the slave site synchronizes the bucket of the master site, the filtering policy of the slave site will be overwritten by the filtering policy of the master site, resulting in the implementation of this method. Based on this, this embodiment provides a solution for erasing the filtering policy of the master site during bucket synchronization and rewriting the filtering policy of this site, ensuring that the filtering policy of the slave site will not be overwritten by the master site under the premise that the slave site can synchronize the storage bucket of the master site normally. That is to ensure the normal application of this method in multi-site scenarios with a master-slave relationship.

[0122] In addition to the embodiment of a distributed object storage system multi-site data synchronization method provided in the above embodiment, the present invention also provides an embodiment corresponding to a computer program product. A computer program product includes a computer program / instruction, and when the computer program / instruction is executed by a processor, the steps of the distributed object storage system multi-site data synchronization method described in any of the above embodiments can be implemented.

[0123] Since the embodiments of the computer program product part correspond to the embodiments of the method part, please refer to the description of the embodiments of the method part for the embodiments of the computer program product part, which will not be repeated here.

[0124] In the above embodiment, a distributed object storage system multi-site data synchronization method is described in detail, and the present invention also provides a distributed object storage system multi-site data synchronization device corresponding embodiment. It should be noted that the present invention describes the embodiment of the device part from two perspectives, one is based on the functional module perspective, and the other is based on the hardware perspective.

[0125] Based on the perspective of functional modules, this embodiment provides a multi-site data synchronization device for a distributed object storage system, such as Figure 3 As shown, including:

[0126] The acquisition module 11 is used to obtain the metadata information of the source object and the filtering policy corresponding to the target site when the source object on the source site needs to be synchronized to the target site; the filtering policy includes multiple filtering rules, each filtering rule includes: a set of corresponding matching items and storage classes; the matching items include one or more metadata information.

[0127] The matching module 12 is used to match the metadata information of the source object with the matching items in each filtering rule.

[0128] The synchronization module 13 is used to store the source object in a designated storage pool in the target site according to the storage class in the filtering rule matched by the source object.

[0129] Since the embodiments of the apparatus part correspond to the embodiments of the method part, please refer to the description of the embodiments of the method part for the embodiments of the apparatus part, which will not be repeated here.

[0130] Figure 4 A structural diagram of a multi-site data synchronization device for a distributed object storage system provided by another embodiment of the present invention is shown in FIG. Figure 4 As shown, a distributed object storage system multi-site data synchronization device includes: a memory 20, for storing computer programs;

[0131] The processor 21 is configured to implement the steps of a multi-site data synchronization method for a distributed object storage system as described in the above embodiment when executing a computer program.

[0132] The distributed object storage system multi-site data synchronization device provided in this embodiment may include but is not limited to a mobile terminal, a personal computer, a workstation, etc.

[0133] Among them, the processor 21 may include one or more processing cores, such as a 4-core processor, an 8-core processor, etc. The processor 21 can be implemented in at least one hardware form of a digital signal processor (DSP), a field-programmable gate array (FPGA), and a programmable logic array (PLA). The processor 21 may also include a main processor and a coprocessor. The main processor is a processor for processing data in the awake state, also known as a central processing unit (CPU); the coprocessor is a low-power processor for processing data in the standby state. In some embodiments, the processor 21 may be integrated with a graphics processing unit (GPU), which is responsible for rendering and drawing the content to be displayed on the display screen. In some embodiments, the processor 21 may also include an artificial intelligence (AI) processor, which is used to process computing operations related to machine learning.

[0134] The memory 20 may include one or more computer-readable storage media, which may be non-transitory. The memory 20 may also include a high-speed random access memory, and a non-volatile memory, such as one or more disk storage devices, flash memory storage devices. In this embodiment, the memory 20 is at least used to store the following computer program 201, wherein, after the computer program is loaded and executed by the processor 21, it can implement the relevant steps of a distributed object storage system multi-site data synchronization method disclosed in any of the aforementioned embodiments. In addition, the resources stored in the memory 20 may also include an operating system 202 and data 203, etc., and the storage method may be temporary storage or permanent storage. Among them, the operating system 202 may include Windows, Unix, Linux, etc. Data 203 may include, but is not limited to, a distributed object storage system multi-site data synchronization method, etc.

[0135] In some embodiments, a distributed object storage system multi-site data synchronization device may also include a display screen 22 , an input and output interface 23 , a communication interface 24 , a power supply 25 , and a communication bus 26 .

[0136] Those skilled in the art will understand that Figure 4 The structure shown in the figure does not constitute a limitation on a multi-site data synchronization device for a distributed object storage system, and may include more or fewer components than those shown in the figure.

[0137] An embodiment of the present invention provides a distributed object storage system multi-site data synchronization device, including a memory and a processor. When the processor executes a program stored in the memory, it can implement the following method: a distributed object storage system multi-site data synchronization method.

[0138] Finally, the present invention also provides an embodiment corresponding to a non-volatile storage medium. The non-volatile storage medium stores a computer program, and when the computer program is executed by a processor, the steps recorded in the above method embodiment are implemented.

[0139] It is understandable that if the method in the above embodiment is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a non-volatile storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium to execute all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM), random access memory (RAM), disk or optical disk, etc. Various media that can store program codes.

[0140] The above is a detailed introduction to a distributed object storage system multi-site data synchronization method, device and medium provided by the present invention. The various embodiments in the specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same and similar parts between the embodiments can be referenced to each other. For the device disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant parts can be referred to the method part description. It should be pointed out that for ordinary technicians in this technical field, without departing from the principle of the present invention, several improvements and modifications can be made to the present invention, and these improvements and modifications also fall within the scope of protection of the present invention.

[0141] It should also be noted that, in this specification, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "comprise", "include" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the statement "comprises a ..." does not exclude the presence of other identical elements in the process, method, article or device including the element.

Claims

1. A distributed object storage system multi-site data synchronization method, characterized in that: include: When it is necessary to synchronize the source object on the source site to the target site, the metadata information of the source object and the filtering policy corresponding to the target site are obtained; wherein the filtering policy includes a plurality of filtering rules; each filtering rule includes: a set of matching items and storage classes corresponding to each other; the storage class is associated with any storage pool in the target site; the matching items include one or more metadata information; Matching the metadata information of the source object with the matching items in each of the filtering rules; According to the storage class in the filtering rule matched by the source object, the source object is stored in a designated storage pool in the target site.

2. The distributed object storage system multi-site data synchronization method according to claim 1, characterized in that: The matching of the metadata information of the source object with the matching items in each of the filtering rules includes: According to the priority of each filtering rule, matching is performed with the metadata information of the source object in descending order; wherein different filtering rules have different priorities; When any of the filtering rules is matched successfully, subsequent matching is stopped, and the process proceeds to the step of storing the source object in a designated storage pool in the target site according to the storage class in the filtering rule matched by the source object.

3. The distributed object storage system multi-site data synchronization method according to claim 1, characterized in that: The matching items include: one matching condition corresponding to any kind of metadata information, or multiple matching conditions corresponding to the same or different kinds of metadata information; The types of metadata information include: object prefix, object suffix, object size, object type, object tag, metadata, and object owner.

4. The distributed object storage system multi-site data synchronization method according to claim 3, characterized in that: When the matching item includes a plurality of matching conditions, the matching conditions further include: a logical relationship; Wherein, the logical relationship includes: and relationship, or relationship; If the logical relationship is an AND relationship, the source object is considered to successfully match the filtering rule only when the metadata information of the source object satisfies all the matching conditions in the matching items; If the logical relationship is an OR relationship, then when the metadata information of the source object satisfies any of the matching conditions in the matching items, it is considered that the source object successfully matches the filtering rule.

5. The distributed object storage system multi-site data synchronization method according to claim 1, characterized in that: Before obtaining the filtering policy corresponding to the target site, the method further includes: Determine whether the state of the source object storage class configuration item is an open state; wherein the source object storage class configuration item is a configurable state bit, including two states of open and closed; If not, proceed to the step of obtaining a filtering policy corresponding to the target site; If so, the object policy of the source site is obtained, and the source object is stored in a designated storage pool in the target site according to the object policy of the source site.

6. The distributed object storage system multi-site data synchronization method according to claim 1, characterized in that: If the filtering policy obtained in the step of obtaining the filtering policy corresponding to the target site is an empty set, or if any of the filtering rules is not matched in the step of matching the metadata information of the source object with the matching items in each of the filtering rules, the method further includes: Obtain a default storage class; wherein the default storage class is a storage class included in each site; According to the default storage class, the source object is stored in a designated storage pool in the target site.

7. The multi-site data synchronization method of a distributed object storage system according to any one of claims 1 to 6, characterized in that: Applied to any site, the method further comprises: When receiving the filtering policy, obtaining bucket metadata of the current site cluster; Parsing the bucket metadata and adding the filtering policy to the bucket metadata; Rewrite the bucket metadata after adding the filtering policy into the current site cluster; The distributed object storage system multi-site data synchronization method according to any one of claims 1 to 6 is applied to a slave site, and when the slave site needs to synchronize a bucket from a master site, the method further includes: Obtaining the metadata synchronization log of the primary site, and parsing to obtain the bucket metadata corresponding to the primary site as the first metadata; erasing the filtering policy in the first metadata to obtain second metadata; Loading the bucket metadata of the current site cluster into the memory as the third metadata; Parsing the third metadata to obtain a metadata portion of the filtering policy corresponding to the current site as fourth metadata; writing the fourth metadata into the second metadata to obtain fifth metadata; The fifth metadata is written into the current site cluster to complete the bucket synchronization.

8. A distributed object storage system multi-site data synchronization device, characterized in that: include: An acquisition module, used for acquiring metadata information of the source object and a filtering policy corresponding to the target site when a source object on the source site needs to be synchronized to the target site; The filtering strategy includes a plurality of filtering rules, each of which includes: a set of matching items and storage classes corresponding to each other; the matching items include one or more metadata information; A matching module, used for matching the metadata information of the source object with the matching items in each of the filtering rules; A synchronization module is used to store the source object in a designated storage pool in the target site according to the storage class in the filtering rule hit by the source object.

9. A distributed object storage system multi-site data synchronization device, characterized in that: include: Memory for storing computer programs; A processor, configured to implement the steps of the distributed object storage system multi-site data synchronization method as described in any one of claims 1 to 7 when executing the computer program.

10. A non-volatile storage medium, characterized in that: The non-volatile storage medium stores a computer program, which, when executed by a processor, implements the steps of the multi-site data synchronization method of a distributed object storage system as described in any one of claims 1 to 7.

Images