A cluster data storage method, device, equipment and medium

Abstract

The application discloses a cluster data storage method and device, equipment and medium, and relates to the technical field of massive distributed storage automation. The application is applied to a distributed storage system, and comprises the following steps: creating a target bucket in a master cluster of the distributed storage system, and creating a target namespace in a slave cluster; uploading a target object to be synchronized into the target bucket, and setting an access control list of the target object as private based on a first permission state; creating a corresponding target file snapshot for the target bucket; establishing a synchronization relationship between the target object and a directory of the target namespace, and triggering a first data synchronization operation, so as to synchronize the target object in the target bucket to the target namespace of the slave cluster; and after the first data synchronization operation ends, performing a rollback operation on the target bucket of the master cluster by using the target file snapshot, so as to detect whether the rollback function is normal. Through the technical scheme of the application, the creation of a file snapshot in a bucket and the synchronization of object permissions of master-slave clusters in a fusion interworking scene are supported.

Description

Technical Field

[0001] This invention relates to the field of automated technology for massive distributed storage, and in particular to a cluster data storage method, apparatus, equipment and medium. Background Technology

[0002] Automated operations for massive storage have been widely used in IT (Internet Technology) companies, cloud computing, big data, virtualization and other fields. At the same time, these fields have increasingly higher requirements for the integration of file scenarios and object scenarios.

[0003] Currently, automated platforms for massive storage distributed systems cannot support multi-cluster synchronization in converged scenarios, cannot synchronize ACL (Access Control List) permission information for objects within buckets, and cannot perform related operations to control objects within buckets through file snapshots. This will affect the robustness, usability, and functionality of the system, and severely impact the competitiveness of massive storage automation platforms.

[0004] Therefore, how to provide a solution to the above-mentioned technical problems is a problem that needs to be solved by those skilled in the art. Summary of the Invention

[0005] In view of this, the purpose of this invention is to provide a cluster data storage method, apparatus, device, and medium that can support the creation of in-bucket file snapshots and the synchronization of object permissions in master-slave clusters under converged and interoperable scenarios. The specific solution is as follows:

[0006] In a first aspect, this application discloses a clustered data storage method applied to a distributed storage system, comprising:

[0007] Create a target bucket in the master cluster of the distributed storage system and a target namespace in the slave cluster of the distributed storage system;

[0008] The target object to be synchronized is uploaded to the target bucket of the main cluster, and the access control list corresponding to the target object is set to a private list based on the first permission state; the first permission state is a permission state that represents full control.

[0009] Check whether the permission status of the access control list corresponding to the current main cluster user has been configured to the first permission status. If so, create a corresponding target file snapshot for the target bucket in the main cluster.

[0010] establish a synchronization relationship between the target object of the master cluster and a directory of the target namespace of the slave cluster, and trigger a first data synchronization operation based on the synchronization relationship to synchronize the target object in the target bucket to the target namespace of the slave cluster;

[0011] perform a rollback operation on the target bucket of the master cluster using the target file snapshot after the first data synchronization operation ends to detect whether a rollback function is normal.

[0012] Optionally, the creating of the target bucket in the master cluster of the distributed storage system and the creating of the target namespace in the slave cluster of the distributed storage system comprises:

[0013] establishing a synchronization connection of the master cluster and the slave cluster in the distributed storage system;

[0014] creating a corresponding authentication user in the master cluster and the slave cluster respectively based on a state of the synchronization connection to obtain a master cluster authentication user and a slave cluster authentication user;

[0015] creating a target bucket in the master cluster by a master cluster user associated with the master cluster authentication user and creating a target namespace in the slave cluster by the slave cluster authentication user.

[0016] Optionally, the establishing of the synchronization connection of the master cluster and the slave cluster in the distributed storage system comprises:

[0017] configuring a corresponding gigabit network virtual internet protocol for the master cluster and the slave cluster in the distributed storage system respectively;

[0018] generating a synchronization key pair between the master cluster and the slave cluster using a preset encryption algorithm;

[0019] establishing the synchronization connection of the master cluster and the slave cluster based on the gigabit network virtual internet protocol and the synchronization key pair.

[0020] Optionally, after the establishing of the synchronization connection of the master cluster and the slave cluster in the distributed storage system, the method further comprises:

[0021] checking a state of the synchronization connection of the slave cluster through the master cluster, and exiting the program with an error if the state of the synchronization connection of the slave cluster is abnormal;

[0022] checking a state of the synchronization connection of the master cluster through the slave cluster, and exiting the program with an error if the state of the synchronization connection of the master cluster is abnormal;

[0023] Correspondingly, based on the state of the synchronization connection, a corresponding authentication user is created in the master cluster and the slave cluster respectively to obtain a master cluster authentication user and a slave cluster authentication user, including:

[0024] When the state of the synchronization connection of the slave cluster and the state of the synchronization connection of the master cluster are both normal, a corresponding authentication user is created in the master cluster and the slave cluster respectively to obtain a master cluster authentication user and a slave cluster authentication user.

[0025] Optionally, the first data synchronization operation is triggered based on the synchronization relationship to synchronize the target object in the target bucket to the target namespace of the slave cluster, including:

[0026] The first data synchronization operation is triggered based on the synchronization relationship, and a data synchronization state flag in the master cluster is detected;

[0027] When the data synchronization state flag is a state representing completion, the target object in the target bucket is synchronized to the target namespace of the slave cluster.

[0028] Optionally, the first data synchronization operation is triggered based on the synchronization relationship to synchronize the target object in the target bucket to the target namespace of the slave cluster, including:

[0029] It is detected whether the target object synchronized from the target bucket of the master cluster exists in the target namespace of the slave cluster;

[0030] If the target object does not exist in the target namespace, the program returns a first identifier and reports an error; if the target object exists in the target namespace, the program returns a second identifier and detects whether the permission state of an access control list corresponding to the target object is the first permission state;

[0031] If the permission state of the access control list corresponding to the target object is not the first permission state, the program returns a first identifier and reports an error; if the permission state of the access control list corresponding to the target object is the first permission state, the program returns a second identifier and modifies the permission state of the access control list corresponding to the target object in the master cluster to a second permission state, and then automatically triggers the synchronization relationship to synchronize the second permission state of the access control list corresponding to the target object to the target namespace of the slave cluster;

[0032] It is detected whether the permission state of the access control list corresponding to the target object synchronized in the target namespace of the slave cluster is the second permission state;

[0033] If the permission state of the access control list corresponding to the target object is not the second permission state, the program returns a first identifier and reports an error; if the permission state of the access control list corresponding to the target object is the second permission state, the program returns a second identifier without reporting an error.

[0034] Optionally, the rollback operation on the target bucket of the master cluster by using the target file snapshot after the first data synchronization operation ends to detect whether the rollback function is normal, comprises:

[0035] The rollback operation on the target bucket of the master cluster by using the target file snapshot after the first data synchronization operation ends to roll back the second permission state of the access control list corresponding to the target object in the master cluster to the first permission state;

[0036] Trigger a second data synchronization operation based on the synchronization relationship to synchronize the target object in the current target bucket to the target namespace of the slave cluster;

[0037] Detect whether the permission state of the access control list corresponding to the target object synchronized in the target namespace of the slave cluster is the first permission state;

[0038] If the permission state of the access control list corresponding to the target object synchronized in the target namespace of the slave cluster is not the first permission state, the program returns a first identifier and reports an error; if the permission state of the access control list corresponding to the target object synchronized in the target namespace of the slave cluster is the first permission state, determine a first information digest algorithm value corresponding to the current target object and compare it with a second information digest algorithm value; wherein the second information digest algorithm value is an information digest algorithm value calculated for the target object after the target object to be synchronized is uploaded to the target bucket of the master cluster;

[0039] If the first information digest algorithm value is equal to the second information digest algorithm value, the program returns a second identifier without reporting an error; if the first information digest algorithm value is not equal to the second information digest algorithm value, the program returns a first identifier and reports an error.

[0040] In a second aspect, the present application discloses a cluster data storage device applied to a distributed storage system, comprising:

[0041] A bucket creation module is configured to create a target bucket in a master cluster of the distributed storage system.

[0042] A namespace creation module is configured to create a target namespace in a slave cluster of the distributed storage system.

[0043] an object uploading module configured to upload a target object to be synchronized into the target bucket of the master cluster;

[0044] a list setting module configured to set an access control list corresponding to the target object as a private list based on a first permission state; the first permission state is a permission state representing complete control;

[0045] a state detecting module configured to detect whether a permission state of the access control list corresponding to a current master cluster user has been configured as the first permission state, and if so, create a corresponding target file snapshot for the target bucket in the master cluster;

[0046] a synchronizing module configured to establish a synchronization relationship between the target object of the master cluster and a directory of the target namespace of the slave cluster, and trigger a first data synchronization operation based on the synchronization relationship to synchronize the target object in the target bucket to the target namespace of the slave cluster;

[0047] a rollback module configured to perform a rollback operation on the target bucket of the master cluster using the target file snapshot after the first data synchronization operation ends to detect whether a rollback function is normal.

[0048] In a third aspect, the present application discloses an electronic device, comprising a processor and a memory; wherein the memory is configured to store a computer program, the computer program is loaded and executed by the processor to implement the cluster data storage method as described above.

[0049] In a fourth aspect, the present application discloses a computer readable storage medium for storing a computer program; wherein the computer program is executed by a processor to implement the cluster data storage method as described above.

[0050] The application provides a cluster data storage method, applied to a distributed storage system, comprising the following steps: creating a target bucket in a master cluster of the distributed storage system and creating a target namespace in a slave cluster of the distributed storage system; uploading a target object to be synchronized into the target bucket in the master cluster, and setting an access control list corresponding to the target object as a private list based on a first permission state; the first permission state is a permission state representing complete control; detecting whether the permission state of the access control list corresponding to a current master cluster user has been configured as the first permission state, and if yes, creating a corresponding target file snapshot for the target bucket in the master cluster; establishing a synchronization relationship between the target object of the master cluster and a directory of the target namespace of the slave cluster, and triggering a first data synchronization operation based on the synchronization relationship to synchronize the target object in the target bucket to the target namespace of the slave cluster; and performing a rollback operation on the target bucket of the master cluster by using the target file snapshot after the first data synchronization operation is completed to detect whether the rollback function is normal.

[0051] The application has the beneficial technical effects that the creation of a file snapshot in a bucket and the synchronization of object permissions of master and slave clusters are supported in a fusion interworking scenario, the data information between the buckets and the namespaces of the clusters in the fusion scenario can be synchronized, and the object permission information uploaded by a master cluster user can be synchronized between multiple fusion scenario clusters and subjected to the control of a file snapshot.

[0052] In addition, the application provides a cluster data storage device, equipment and storage medium, which correspond to the cluster data storage method and have the same effects. BRIEF DESCRIPTION OF DRAWINGS

[0053] In order to more clearly illustrate the technical solutions in the embodiments of the application or the prior art, the following will briefly introduce the drawings needed to be used in the embodiments or the prior art description. Obviously, the drawings in the following description only belong to the embodiments of the application, and for those skilled in the art, other drawings can also be obtained without creative labor on the basis of the provided drawings.

[0054] Figure 1 A cluster data storage method flow chart disclosed by the application;

[0055] Figure 2 A master-slave cluster connection establishment schematic diagram disclosed by the application;

[0056] Figure 3 A master-slave cluster tenant creation schematic diagram disclosed by the application through a tenant module;

[0057] Figure 4A schematic diagram of an authentication user group and an authentication user created by the present application is disclosed;

[0058] Figure 5 A schematic diagram of an S3 user created by the present application is disclosed;

[0059] Figure 6 A schematic diagram of a target bucket and a target namespace created by the present application is disclosed;

[0060] Figure 7 A schematic diagram of a snapshot of a file in a bucket created by the present application is disclosed;

[0061] Figure 8 A schematic diagram of cluster synchronization in a fusion scenario disclosed by the present application is disclosed;

[0062] Figure 9 A schematic diagram of cluster rollback synchronization in a fusion scenario disclosed by the present application is disclosed;

[0063] Figure 10 A schematic diagram of an implementation module of a cluster data storage method disclosed by the present application is disclosed;

[0064] Figure 11 A schematic diagram of a cluster data storage device structure disclosed by the present application is disclosed;

[0065] Figure 12 A schematic diagram of an electronic device structure disclosed by the present application is disclosed. DETAILED DESCRIPTION

[0066] The technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the accompanying drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor are within the scope of protection of the present application.

[0067] Currently, for the fusion scenario of file scenarios and object scenarios, multi-cluster remote synchronization cannot be performed, and the object permissions in the bucket cannot be synchronized, which cannot meet the use of users. Since the current automatic platform of the mass storage distributed system cannot support multi-cluster synchronization in the fusion scenario, the permission information of the access control list of the object in the bucket cannot be synchronized, and the related operations of controlling the object in the bucket through the file snapshot cannot be performed, which will affect the robustness, ease of use and richness of functions of the system, and seriously affect the competitiveness of the mass storage automatic platform.

[0068] To this end, the application provides a cluster data storage scheme, which can synchronize data information between buckets and namespaces between clusters in a fusion scene, synchronize object permission information uploaded by a user between multiple fusion scene clusters, and accept control of a file snapshot.

[0069] The embodiment of the application discloses a cluster data storage method, which is applied to a distributed storage system and includes the following steps: Figure 1

[0070] Step S11: creating a target bucket in a master cluster of the distributed storage system and creating a target namespace in a slave cluster of the distributed storage system.

[0071] In the embodiment of the application, the target bucket created in the master cluster of the distributed storage system is created by a master cluster user (S3 user, Simple Storage Service), and the target namespace created in the slave cluster of the distributed storage system is created by a slave cluster authentication user. Specifically, a synchronous connection of the master cluster and the slave cluster is established in the distributed storage system, and a corresponding authentication user is created in the master cluster and the slave cluster based on a state of the synchronous connection, to obtain a master cluster authentication user and a slave cluster authentication user. A target bucket is created in the master cluster by a master cluster user associated with the master cluster authentication user, and a target namespace is created in the slave cluster by the slave cluster authentication user.

[0072] It should be noted that the master cluster user and the slave cluster authentication user are both created on the basis of the establishment of the synchronous connection of the master cluster and the slave cluster. Therefore, a corresponding gigabit network virtual Internet Protocol (IP) is configured for the master cluster and the slave cluster in the distributed storage system. The gigabit network virtual IP of the master cluster is $MAIN_CTDB_VIR_IP_INFO, and the gigabit network virtual IP of the slave cluster is $CONG_CTDB_VIR_IP_INFO. The gigabit network virtual IP of the master cluster and the slave cluster is obtained by get_ctdb_vir_ip_list(), a self-checking program, to ensure that the cluster has the gigabit network virtual IP. If any cluster does not have the gigabit network virtual IP configured, an error is detected and the program exits.

[0073] ​Secondly, a preset encryption algorithm is used to generate a synchronization key pair between the master cluster and the slave cluster. In the embodiment of the application, the master cluster uses an encryption algorithm, such as a national encryption algorithm, to generate an encrypted cluster synchronization key pair $SYNC_KEY_PAIR. Finally, based on the virtual Internet Protocol of the 10-gigabit network and the synchronization key pair, a synchronization connection of the master cluster and the slave cluster is established. Specifically, the master cluster uses the $SYNC_KEY_PAIR and the $CONG_CTDB_VIR_IP_INFO to establish a synchronization connection from the master cluster to the slave cluster. Figure 2 That is, a schematic diagram of establishing a connection between the master and slave clusters. The master cluster establishes a synchronization connection to the slave cluster through create_remote_connect($SYNC_KEY_PAIR, $CONG_CTDB_VIR_IP_INFO).

[0074] Further, after the synchronization connection between the master cluster and the slave cluster is established, the state of the synchronization connection needs to be detected. Specifically, the master cluster checks the state of the synchronization connection of the slave cluster ($REMOTE_STATUS_CONG), and if the state of the synchronization connection of the slave cluster is abnormal, the program exits with an error; the slave cluster checks the state of the synchronization connection of the master cluster ($REMOTE_STATUS_MAIN), and if the state of the synchronization connection of the master cluster is abnormal, the program exits with an error. For example, as shown in Figure 2 After the synchronization connection between the master and slave clusters is established, the master cluster checks the synchronization connection state of the slave cluster through get_assign_remote_cluster_info($CONG_CTDB_VIR_IP_INFO), and if the state is abnormal, the program returns 1 and reports an error, otherwise, 0.

[0075] In the embodiment of the application, after the preliminary work is completed, the synchronization connection established based on the master and slave clusters is used to create a user. First, a tenant is created, as shown in Figure 3 The corresponding schematic diagram of creating a master and slave cluster tenant through a tenant module is shown in Figure 4The illustrated authentication user group / authentication user module is a creation process schematic diagram. The main cluster authentication user group $AUTH_GROUP_MAIN and the authentication user $AUTH_USER_MAIN are created by create_auth_user_group($AUTH_GROUP_MAIN, $AUTH_USER_MAIN); the slave cluster authentication user group $AUTH_GROUP_CONG and the authentication user $AUTH_USER_CONG are created by create_auth_user_group($AUTH_GROUP_CONG, $AUTH_USER_CONG); further, as shown in the following, the S3 user associated with the authentication user in the master-slave cluster is created: the $S3_USER_MAIN associated with the $AUTH_USER_MAIN is created in the master cluster by create_user_s3_nas_mix($AUTH_USER_MAIN, $S3_USER_MAIN); the $S3_USER_CONG associated with the $AUTH_USER_CONG is created in the slave cluster by create_user_s3_nas_mix($AUTH_USER_CONG, $S3_USER_CONG). Finally, the target bucket $BUCKET_MAIN is created in the master cluster by the $S3_USER_MAIN; the namespace $NAMESPACE_CONG is created in the slave cluster by the $USER_AUTH_CONG. Figure 5 As shown in the following, the S3 user associated with the authentication user in the master-slave cluster is created: the $S3_USER_MAIN associated with the $AUTH_USER_MAIN is created in the master cluster by create_user_s3_nas_mix($AUTH_USER_MAIN, $S3_USER_MAIN); the $S3_USER_CONG associated with the $AUTH_USER_CONG is created in the slave cluster by create_user_s3_nas_mix($AUTH_USER_CONG, $S3_USER_CONG). Finally, the target bucket $BUCKET_MAIN is created in the master cluster by the $S3_USER_MAIN; the namespace $NAMESPACE_CONG is created in the slave cluster by the $USER_AUTH_CONG.

[0076] Step S12: uploading the target object to be synchronized into the target bucket of the master cluster, and setting the access control list corresponding to the target object as a private list based on a first permission state; the first permission state is a permission state representing full control.

[0077] In the embodiment of the present application, after the target bucket is created in the master cluster, the target object $OBEJCT_1 to be synchronized is uploaded into the target bucket $BUCKET_MAIN, and the access control list (Access Control List, ACL) of the object $OBJECT_1 in the target bucket is set as a private list. It should be noted that the object $OBJECT_1 in the target bucket needs to calculate the md5 (Message-Digest Algorithm 5, Message-Digest Algorithm 5) value to obtain $MD5_1; the private list is set based on the first permission state, and the first permission state is a permission state representing full control (FULL CONTROL).

[0078] As shown in the following, the S3 user associated with the authentication user in the master-slave cluster is created: the $S3_USER_MAIN associated with the $AUTH_USER_MAIN is created in the master cluster by create_user_s3_nas_mix($AUTH_USER_MAIN, $S3_USER_MAIN); the $S3_USER_CONG associated with the $AUTH_USER_CONG is created in the slave cluster by create_user_s3_nas_mix($AUTH_USER_CONG, $S3_USER_CONG). Finally, the target bucket $BUCKET_MAIN is created in the master cluster by the $S3_USER_MAIN; the namespace $NAMESPACE_CONG is created in the slave cluster by the $USER_AUTH_CONG. Figure 6The diagram shows the process of creating target bucket in the master cluster and target namespace in the slave cluster by the data module. The target bucket $BUCKET_MAIN is created in the master cluster by s3_create_bucket($BUCKET_MAIN); the target namespace $NAMESPACE_CONG is created in the slave cluster by create_namespace($NAMESPACE_CONG). Further, the target object $OBEJCT_1 is uploaded into the target bucket by s3_put_object($BUCKET_MAIN, $OBJECT_1) and the md5 value $MD5_1 of $OBJECT_1 is calculated.

[0079] Step S13: detecting whether the permission state of the access control list corresponding to the current master cluster user has been configured as the first permission state, if yes, creating a corresponding target file snapshot for the target bucket in the master cluster.

[0080] In the embodiment of the present application, the master cluster is confirmed. Since the ACL corresponding to the target object is set based on the first permission state, the post-ACL setting permission state of $S3_USER_MAIN should be "FULL CONTROL"; if the state is not "FULL CONTROL", the program returns 1 and exits with an error; otherwise, the program returns 0, and then a corresponding target file snapshot $SNAPSHOT_MAIN is created for the target bucket $BUCKET_MAIN in the master cluster.

[0081] As shown in Figure 7 The diagram includes the process of setting the access control list corresponding to the target object as a private list and the process of confirming the master cluster and creating a file snapshot in the bucket. According to Figure 7 It can be seen that first, the ACL of the target object $OBJECT_1 in the target bucket $BUCKET_MAIN is set as private by s3_set_object_acl($BUCKET_MAIN, $OBJECT_1, "PV"); then the master cluster is confirmed: the post-ACL setting permission state of $S3_USER_MAIN should be "FULL CONTROL", if the state is not "FULL CONTROL", the program returns 1 and exits with an error; a file snapshot $SNAPSHOT_MAIN is created for the bucket $BUCKET_MAIN in the master cluster by create_directory_snapshot($BUCKET_MAIN, $SNAPSHOT_MAIN).

[0082] Step S14: establishing a synchronization relationship between the target object of the master cluster and the directory of the target namespace of the slave cluster, and triggering a first data synchronization operation based on the synchronization relationship to synchronize the target object in the target bucket to the target namespace of the slave cluster.

[0083] In the embodiments of the present application, the cluster in the fusion scenario can realize data information synchronization between the bucket and the namespace. It can be understood that only files and namespaces are supported for operation in the file scenario, and only objects and buckets are supported for operation in the object scenario. After the synchronization relationship between the target object of the master cluster and the directory of the target namespace of the slave cluster is established, the first data synchronization operation can be triggered based on the synchronization relationship to synchronize the target object in the target bucket to the target namespace of the slave cluster. In this way, the object permission information uploaded by the S3 user can be synchronized between the multiple fusion scenario clusters.

[0084] In the embodiments of the present application, the master cluster $BUCKET_MAIN, the slave cluster $NAMESPACE_CONG, and the cluster $PUBLIC_IP gigabit network are used to establish the synchronization relationship $SYNC_PAIR of the directory / bucket; it should be noted that $PUBLIC_IP is the gigabit network virtual Internet Protocol configured by the master and slave clusters. Further, it is detected whether the data synchronization starts and completes after the synchronization relationship is established. Specifically, the first data synchronization operation is triggered based on the synchronization relationship, and the data synchronization state flag in the master cluster is detected; when the data synchronization state flag is a state representing completion, the target object in the target bucket is synchronized to the target namespace of the slave cluster. That is, if the master cluster $REMOTE_DATA_STATE state is “complete”, the program returns 0 without error, and the target object in the target bucket is synchronized to the target namespace of the slave cluster; otherwise, 1 is returned and an error is reported.

[0085] Step S15: after the first data synchronization operation ends, using the target file snapshot to perform a rollback operation on the target bucket of the master cluster to detect whether the rollback function is normal.

[0086] In the embodiments of the present application, the control of the file snapshot is accepted. After the target object in the target bucket is synchronized to the target namespace of the slave cluster, a rollback operation is performed on the target bucket of the master cluster using the target file snapshot to detect whether the rollback function is normal. The rollback function detected includes whether the file snapshot rollback is abnormal, that is, whether the file snapshot rollback can be realized; it also includes whether the file synchronization content is abnormal, that is, whether the data after rollback is synchronized.

[0087] The application provides a cluster data storage method applied to a distributed storage system, including: creating a target bucket in a master cluster of the distributed storage system and creating a target namespace in a slave cluster of the distributed storage system; uploading a target object to be synchronized into the target bucket of the master cluster, and setting an access control list corresponding to the target object as a private list based on a first permission state; the first permission state is a permission state representing complete control; detecting whether the permission state of the access control list corresponding to a current master cluster user has been configured as the first permission state, and if yes, creating a corresponding target file snapshot for the target bucket in the master cluster; establishing a synchronization relationship between the target object of the master cluster and a directory of the target namespace of the slave cluster, and triggering a first data synchronization operation based on the synchronization relationship to synchronize the target object in the target bucket to the target namespace of the slave cluster; and performing a rollback operation on the target bucket of the master cluster by using the target file snapshot after the first data synchronization operation ends to detect whether the rollback function is normal.

[0088] The application has the beneficial technical effects that the creation of a file snapshot in a bucket and the synchronization of object permissions of master and slave clusters are supported in a fusion interworking scenario, data information between buckets and namespaces between clusters in a fusion scenario can be synchronized, and object permission information uploaded by a master cluster user can be synchronized between multiple fusion scenario clusters and subjected to the control of a file snapshot.

[0089] It should be noted that, based on the above embodiment, the application embodiment also provides a corresponding improvement scheme. In subsequent embodiments, the steps involved in the above embodiment or between the corresponding steps can be mutually referred, and the corresponding beneficial effects can also be mutually referred. In the following improved embodiments, they will not be described one by one.

[0090] In a specific embodiment of the application, step S14, the first data synchronization operation is triggered based on the synchronization relationship to synchronize the target object in the target bucket to the target namespace of the slave cluster, which can include the following steps:

[0091] Step one: detecting whether the target object synchronized from the target bucket of the master cluster exists in the target namespace of the slave cluster;

[0092] Step two: if the target object does not exist in the target namespace, the program returns a first identifier and reports an error; if the target object exists in the target namespace, the program returns a second identifier and detects whether the permission state of the access control list corresponding to the target object is the first permission state;

[0093] Step three: if the permission state of the access control list corresponding to the target object is not the first permission state, the program returns the first identification and reports an error; if the permission state of the access control list corresponding to the target object is the first permission state, the program returns the second identification and modifies the permission state of the access control list corresponding to the target object in the master cluster to the second permission state, and then automatically triggers the synchronization relationship to synchronize the second permission state of the access control list corresponding to the target object to the target namespace in the slave cluster;

[0094] Step four: detecting whether the permission state of the access control list corresponding to the target object synchronized in the target namespace of the slave cluster is the second permission state;

[0095] Step five: if the permission state of the access control list corresponding to the target object is not the second permission state, the program returns the first identification and reports an error; if the permission state of the access control list corresponding to the target object is the second permission state, the program returns the second identification and does not report an error.

[0096] For convenience of description, the above steps can be combined for description.

[0097] In the embodiment of the application, in the process of triggering the first data synchronization operation based on the synchronization relationship, the program detects whether $OBJECT_1 synchronized from the master cluster exists in $NAMESPACE_CONG created by $AUTH_USER_CONG associated with slave cluster user $S3_USER_CONG, if not, the program returns the first identification (returns 1) and reports an error; otherwise, 0 is returned to continue detecting whether the ACL of $OBJECT_1 in the slave cluster is "FULL_CONTROL", if not, 1 is returned and the program reports an error; then the master cluster modifies the object ACL permission state to "WRITE"; then automatically triggers the synchronization relationship $SYNC_PAIR to synchronize the new data again, at this time, the new data is the target object whose permission state of the access control list is modified to the second permission state (WRITE); at this time, the slave cluster checks whether the ACL permission state of the object synchronized at present is synchronized to "WRITE".

[0098] As Figure 8 Fig. 1 shows a schematic diagram of cluster synchronization in a fusion scenario.

[0099] The sync relationship $SYNC_PAIR between the directories / buckets of the master-slave cluster is established by sync_remote_copy_pair($BUCKET_MAIN, $NAMESPACE_CONG, $PUBLIC_IP, $SYNC_PAIR); whether the data synchronization starts and completes after the sync relationship is established is detected by verify_remote_copy_pair_sync_complete(); if the state of the master cluster $REMOTE_DATA_STATE is "complete", the program returns 0 (the second identifier) without error, otherwise, 1 is returned and an error is reported.

[0100] Then, whether the $OBJECT_1 synchronized from the master cluster exists in the $NAMESPACE_CONG created by the slave cluster authentication user $AUTH_USER_CONG associated with the $S3_USER_CONG is detected by s3_query_object($S3_USER_CONG, $AUTH_USER_CONG, $NAMESPACE_CONG, $OBJECT_1, "FULL_CONTROL") in the slave cluster; if not, 1 is returned and an error is reported; at the same time, whether the ACL of the $OBJECT_1 in the slave cluster is "FULL_CONTROL" is detected, if not, 1 is returned and the program reports an error; further, the object ACL permission state is modified to "WRITE" in the master cluster by s3_set_object_acl($BUCKET_MAIN, $OBJECT_1, "PRW"), which automatically triggers the use of the sync relationship $SYNC_PAIR to synchronize the new data again; whether the ACL of the $OBJECT_1 in the slave cluster is synchronized to "WRITE" is detected by s3_query_object($S3_USER_CONG, $AUTH_USER_CONG, $NAMESPACE_CONG, $OBJECT_1, "WRITE") in the slave cluster; if not synchronized to "WRITE", the program returns 1 and reports an error.

[0101] In one specific embodiment of the application, after the first data synchronization operation ends, the target file snapshot is used to perform a rollback operation on the target bucket of the master cluster to detect whether the rollback function is normal, which can include the following steps:

[0102] Step one: after the first data synchronization operation ends, the target file snapshot is used to perform a rollback operation on the target bucket of the master cluster, so as to roll back the second permission state of the access control list corresponding to the target object in the master cluster to the first permission state;

[0103] Step two: triggering a second data synchronization operation based on the synchronization relationship to synchronize the target object in the target bucket to the target namespace of the slave cluster;

[0104] Step three: detecting whether the permission state of the access control list corresponding to the target object synchronized in the target namespace of the slave cluster is the first permission state;

[0105] Step four: if the permission state of the access control list corresponding to the target object synchronized in the target namespace of the slave cluster is not the first permission state, the program returns the first identification and reports an error; if the permission state of the access control list corresponding to the target object synchronized in the target namespace of the slave cluster is the first permission state, the first information digest algorithm value corresponding to the target object is determined and compared with the second information digest algorithm value; wherein the second information digest algorithm value is the information digest algorithm value calculated for the target object after the target object to be synchronized is uploaded to the target bucket of the master cluster;

[0106] Step five: if the first information digest algorithm value is equal to the second information digest algorithm value, the program returns the second identification and does not report an error; if the first information digest algorithm value is not equal to the second information digest algorithm value, the program returns the first identification and reports an error.

[0107] For the convenience of description, the above steps can be combined for description.

[0108] In the embodiment of the application, the file snapshot is accepted. After the target object in the target bucket is synchronized to the target namespace of the slave cluster, the target file snapshot is used to perform a rollback operation on the target bucket of the master cluster to detect whether the rollback function is normal.

[0109] The master cluster rolls back the data and metadata in the target bucket $BUCKET_MAIN through the rollback snapshot $SNAPSHOT_MAIN, and the ACL permission state of the target object $OBJECT_1 in the master cluster is rolled back to "FULL_CONTROL"; then the automatic triggering of the synchronization relationship $SYNC_PAIR is triggered again to synchronize the new data, at this time the new data is the target object whose permission state of the access control list is rolled back to the first permission state (FULL_CONTROL); then the automatic verification of the ACL permission state is performed from the cluster, and the ACL permission state is synchronized to "FULL_CONTROL" after the rollback; if it is not "FULL_CONTROL" at this time, the program returns 1 (the first identifier) and an error "file snapshot rollback exception" is reported; then the object $OBJECT_1 in $NAMESPACE_CONG is automatically downloaded from the cluster, the md5 value $MD5_2 at this time is verified, and if $MD5_1 is not equal to $MD5_2, the program reports an error prompt "file synchronization content exception".

[0110] As Figure 9 shown is a schematic diagram of implementing the rollback function and synchronization through the rollback synchronization module.

[0111] The data and metadata in $BUCKET_MAIN are rolled back in the master cluster through the rollback snapshot $SNAPSHOT_MAIN by recovery_directory_snapshot($SNAPSHOT_MAIN, $BUCKET_MAIN); the ACL permission state of the target object $OBJECT_1 in the master cluster is rolled back to "FULL_CONTROL"; the automatic triggering of the synchronization relationship $SYNC_PAIR is triggered again to synchronize the new data; whether the ACL of the object $OBJECT_1 in the slave cluster is synchronized to "FULL_CONTROL" is detected in the slave cluster by s3_query_object($S3_USER_CONG, $AUTH_USER_CONG, $NAMESPACE_CONG, $OBJECT_1, "FULL CONTROL"); if it is not synchronized to "FULL_CONTROL", the program returns 1 and an error "file snapshot rollback exception" is reported; the object $OBJECT_1 in $NAMESPACE_CONG is automatically downloaded from the cluster by s3_download_object($NAMESPACE_CONG, $OBJECT_1) to compare and verify the md5 value $MD5_2 at this time, and if $MD5_1 is not equal to $MD5_2, the program reports an error prompt "file synchronization content exception".

[0112] In summary, as Figure 10As shown, in combination with the above embodiments, the method for supporting the creation of a file snapshot in a bucket and synchronizing the object ACL permission in a fusion interworking scenario mainly comprises eight modules, i.e., a master-slave cluster synchronization connection module, a tenant module, an authenticated user / user group module, an S3 user module, a data module, a permission and file snapshot module, a fusion file scenario cluster synchronization module, and a rollback synchronization module. The robustness, ease of use, and richness of functions of the system can be improved, and the competitiveness of the mass storage automation platform can be improved.

[0113] Correspondingly, the embodiment of the application further discloses a cluster data storage device applied to a distributed storage system, referring to Figure 11 As shown, the device comprises:

[0114] a bucket creation module 11, configured to create a target bucket in a master cluster of the distributed storage system;

[0115] a namespace creation module 12, configured to create a target namespace in a slave cluster of the distributed storage system;

[0116] an object uploading module 13, configured to upload a target object to be synchronized into the target bucket of the master cluster;

[0117] a list setting module 14, configured to set an access control list corresponding to the target object as a private list based on a first permission state; the first permission state is a permission state representing complete control;

[0118] a state detection module 15, configured to detect whether a permission state of the access control list corresponding to a current master cluster user has been configured as the first permission state, and if yes, create a corresponding target file snapshot for the target bucket in the master cluster;

[0119] a synchronization module 16, configured to establish a synchronization relationship between the target object of the master cluster and a directory of the target namespace of the slave cluster, and trigger a first data synchronization operation based on the synchronization relationship, so as to synchronize the target object in the target bucket to the target namespace of the slave cluster;

[0120] a rollback module 17, configured to perform a rollback operation on the target bucket of the master cluster by using the target file snapshot after the first data synchronization operation ends, so as to detect whether the rollback function is normal.

[0121] Wherein, the more specific working process of each module can refer to the corresponding content disclosed in the foregoing embodiments, which will not be repeated here.

[0122] As can be seen, through the above scheme of the embodiment, a computer program is stored; wherein the computer program is executed by a processor to implement the cluster data storage method as described above.

[0123] The application provides a cluster data storage method applied to a distributed storage system, including: creating a target bucket in a master cluster of the distributed storage system and creating a target namespace in a slave cluster of the distributed storage system; uploading a target object to be synchronized into the target bucket of the master cluster, and setting an access control list corresponding to the target object as a private list based on a first permission state; the first permission state is a permission state representing complete control; detecting whether the permission state of the access control list corresponding to a current master cluster user has been configured as the first permission state, and if yes, creating a corresponding target file snapshot for the target bucket in the master cluster; establishing a synchronization relationship between the target object of the master cluster and a directory of the target namespace of the slave cluster, and triggering a first data synchronization operation based on the synchronization relationship to synchronize the target object in the target bucket to the target namespace of the slave cluster; and performing a rollback operation on the target bucket of the master cluster by using the target file snapshot after the first data synchronization operation ends to detect whether the rollback function is normal.

[0124] The application has the beneficial technical effects that the creation of a file snapshot in a bucket in a fusion interworking scene and the synchronization of object permissions of master and slave clusters are supported, and the data information between the buckets and the namespaces between the clusters in the fusion scene can be synchronized, so that the object permission information uploaded by the master cluster user can be synchronized between the clusters in the multi-fusion scene and the control of the file snapshot is accepted.

[0125] Further, the application embodiment further discloses an electronic device, Figure 12 The electronic device 20 is shown in the structure diagram according to an exemplary embodiment, and the contents in the diagram cannot be considered as any limitation on the use range of the application.

[0126] Figure 12 The electronic device 20 provided by the application embodiment is shown in the structure diagram. The electronic device 20 specifically can include: at least one processor 21, at least one memory 22, a power supply 23, a communication interface 24, an input / output interface 25 and a communication bus 26. The memory 22 is used to store a computer program, the computer program is loaded and executed by the processor 21 to realize the related steps in the cluster data storage method disclosed in any of the preceding embodiments.

[0127] In this embodiment, the power supply 23 is configured to provide operating voltage for each hardware device on the electronic device 20; the communication interface 24 is configured to create a data transmission channel between the electronic device 20 and external devices, and the communication protocol followed by the communication interface 24 can be any communication protocol applicable to the technical solution of the present application, which will not be specifically limited herein; the input / output interface 25 is configured to obtain external input data or output data to the outside, and the specific interface type can be selected according to the specific application needs, which will not be specifically limited herein.

[0128] In addition, the memory 22 as a carrier of resource storage can be a read-only memory, a random access memory, a magnetic disk or an optical disk, etc., and the resources stored thereon can include an operating system 221, a computer program 222 and data 223, etc., and the data 223 can include various data. The storage mode can be temporary storage or permanent storage.

[0129] The operating system 221 is configured to manage and control each hardware device on the electronic device 20 and the computer program 222, and can be Windows Server, Netware, Unix, Linux, etc. In addition to the computer program capable of completing the cluster data storage method executed by the electronic device 20 disclosed in any of the foregoing embodiments, the computer program 222 can further include a computer program capable of completing other specific work.

[0130] Further, the present application also discloses a computer readable storage medium, which includes a random access memory (RAM), a memory, a read-only memory (ROM), an electrically programmable ROM, an electrically erasable programmable ROM, a register, a hard disk, a magnetic disk or an optical disk, or any other form of storage medium known in the technical field. The computer program is executed by the processor to implement the foregoing cluster data storage method. For the specific steps of the method, reference can be made to the corresponding content disclosed in the foregoing embodiments, which will not be described here again.

[0131] In the specification, each embodiment is described in a progressive manner, and each embodiment focuses on the difference from other embodiments. For the same or similar parts between each embodiment, reference can be made to each other. For the device disclosed in the embodiments, since it corresponds to the method disclosed in the embodiments, the description is relatively simple, and the relevant part can be referred to the method part.

[0132] The steps of the cluster data storage method or algorithm described in connection with the embodiments disclosed herein can be embodied directly in hardware, in a software module executed by a processor, or in a combination of the two. A software module can reside in random access memory (RAM), memory, flash memory, read-only memory (ROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), registers, hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art.

[0133] Finally, it should be noted that, in the present document, relational terms such as first and second, and the like can be used solely to distinguish one entity or action from another entity or action without necessarily requiring or implying any actual such relationship or order between such entities or actions. Moreover, the terms "comprises", "comprising", or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but can include other elements not expressly listed or inherent to such process, method, article, or apparatus. The terms "comprises", "comprising", or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but can include other elements not expressly listed or inherent to such process, method, article, or apparatus.

[0134] The above provides a cluster data storage method, device, equipment and medium provided by the present application, and the principle and implementation mode of the present application are described in the present document by applying specific examples. The above description of the embodiments is only used to help understand the method of the present application and its core idea; meanwhile, for those skilled in the art, according to the idea of the present application, the specific implementation mode and application range can be changed, and the above description of the present application should not be understood as a limitation of the present application.

Claims

1. A clustered data storage method, characterized in that, Applications in distributed storage systems include: Create a target bucket in the master cluster of the distributed storage system and a target namespace in the slave cluster of the distributed storage system; The target object to be synchronized is uploaded to the target bucket of the main cluster, and the access control list corresponding to the target object is set to a private list based on the first permission state; the first permission state is a permission state that represents full control. Check whether the permission status of the access control list corresponding to the current master cluster user has been configured to the first permission status. If so, create a corresponding target file snapshot for the target bucket in the master cluster. Establish a synchronization relationship between the target object in the master cluster and the directory of the target namespace in the slave cluster, and trigger a first data synchronization operation based on the synchronization relationship to synchronize the target object in the target bucket to the target namespace in the slave cluster; After the first data synchronization operation is completed, the target file snapshot is used to perform a rollback operation on the target bucket of the main cluster to check whether the rollback function is normal.

2. The cluster data storage method according to claim 1, characterized in that, The steps of creating the target bucket in the master cluster of the distributed storage system and creating the target namespace in the slave cluster of the distributed storage system include: Establish a synchronous connection between the master cluster and the slave cluster in the distributed storage system; Based on the state of the synchronization connection, corresponding authenticated users are created in the master cluster and the slave cluster respectively to obtain master cluster authenticated users and slave cluster authenticated users; In the primary cluster, a target bucket is created by the primary cluster user associated with the primary cluster authentication user, and in the secondary cluster, a target namespace is created by the secondary cluster authentication user.

3. The cluster data storage method according to claim 2, characterized in that, Establishing a synchronization connection between the master cluster and the slave cluster in the distributed storage system includes: Configure the corresponding 10 Gigabit network virtual Internet protocol for the master cluster and slave cluster in the distributed storage system; A synchronization key pair between the master cluster and the slave cluster is generated using a preset encryption algorithm; Based on the 10 Gigabit Network Virtual Internet Protocol and the synchronization key pair, a synchronization connection is established between the master cluster and the slave cluster.

4. The cluster data storage method according to claim 2, characterized in that, After establishing the synchronization connection between the master cluster and the slave cluster in the distributed storage system, the method further includes: The program checks the synchronization connection status of the slave cluster through the master cluster. If the synchronization connection status of the slave cluster is abnormal, the program will report an error and exit. The program checks the synchronization connection status of the master cluster from the slave cluster. If the synchronization connection status of the master cluster is abnormal, the program will report an error and exit. Accordingly, based on the state of the synchronization connection, corresponding authenticated users are created in the master cluster and the slave cluster respectively, to obtain master cluster authenticated users and slave cluster authenticated users, including: When the synchronization connection status of the slave cluster and the synchronization connection status of the master cluster are both normal, corresponding authentication users are created in the master cluster and the slave cluster respectively to obtain master cluster authentication users and slave cluster authentication users.

5. The cluster data storage method according to claim 1, characterized in that, The step of triggering a first data synchronization operation based on the synchronization relationship to synchronize the target object in the target bucket to the target namespace of the slave cluster includes: The first data synchronization operation is triggered based on the synchronization relationship, and the data synchronization status flag in the main cluster is detected. When the data synchronization status flag indicates a completed state, the target object in the target bucket is synchronized to the target namespace of the slave cluster.

6. The cluster data storage method according to any one of claims 1 to 5, characterized in that, The step of triggering a first data synchronization operation based on the synchronization relationship to synchronize the target object in the target bucket to the target namespace of the slave cluster includes: Detect whether the target object synchronized from the target bucket of the master cluster exists in the target namespace of the slave cluster; If the target object does not exist in the target namespace, the program returns a first identifier and reports an error; if the target object exists in the target namespace, the program returns a second identifier and checks whether the permission status of the access control list corresponding to the target object is the first permission status. If the permission status of the access control list corresponding to the target object is not the first permission status, the program returns a first identifier and reports an error; if the permission status of the access control list corresponding to the target object is the first permission status, the program returns a second identifier and modifies the permission status of the access control list corresponding to the target object to the second permission status in the master cluster, and then automatically triggers the synchronization relationship to synchronize the second permission status of the access control list corresponding to the target object to the target namespace of the slave cluster. Detect whether the permission status of the access control list corresponding to the target object synchronized from the target namespace of the cluster is the second permission status; If the permission status of the access control list corresponding to the target object is not the second permission status, the program returns the first identifier and reports an error; if the permission status of the access control list corresponding to the target object is the second permission status, the program returns the second identifier and does not report an error.

7. The cluster data storage method according to claim 6, characterized in that, The step of performing a rollback operation on the target bucket of the main cluster using the target file snapshot after the first data synchronization operation is completed to check whether the rollback function is normal includes: After the first data synchronization operation is completed, the target file snapshot is used to perform a rollback operation on the target bucket of the main cluster, so as to roll back the second permission status of the access control list corresponding to the target object in the main cluster to the first permission status. A second data synchronization operation is triggered based on the synchronization relationship to synchronize the target object in the current target bucket to the target namespace of the slave cluster; Detect whether the permission status of the access control list corresponding to the current target object, which is synchronized from the target namespace of the cluster, is the first permission status; If the permission status of the access control list corresponding to the current target object synchronized from the target namespace in the cluster is not the first permission status, the program returns a first identifier and reports an error; if the permission status of the access control list corresponding to the current target object synchronized from the target namespace in the cluster is the first permission status, the program determines the first message digest algorithm value corresponding to the current target object and compares it with the second message digest algorithm value; wherein, the second message digest algorithm value is the message digest algorithm value calculated for the target object after uploading the target object to be synchronized to the target bucket of the main cluster; If the first message digest algorithm value is equal to the second message digest algorithm value, the program returns a second identifier without reporting an error; if the first message digest algorithm value is not equal to the second message digest algorithm value, the program returns a first identifier and reports an error.

8. A clustered data storage device, characterized in that, Applications in distributed storage systems include: The bucket creation module is used to create target buckets in the main cluster of the distributed storage system. The namespace creation module is used to create a target namespace from the slave cluster of the distributed storage system. The object upload module is used to upload the target object to be synchronized to the target bucket of the main cluster; The list setting module is used to set the access control list corresponding to the target object as a private list based on the first permission state; the first permission state is a permission state that represents full control; The status detection module is used to detect whether the permission status of the access control list corresponding to the current main cluster user has been configured to the first permission status. If so, a corresponding target file snapshot is created for the target bucket in the main cluster. The synchronization module is used to establish a synchronization relationship between the target object in the master cluster and the directory of the target namespace in the slave cluster, and to trigger a first data synchronization operation based on the synchronization relationship to synchronize the target object in the target bucket to the target namespace in the slave cluster. The rollback module is used to perform a rollback operation on the target bucket of the main cluster using the target file snapshot after the first data synchronization operation is completed, in order to detect whether the rollback function is normal.

9. An electronic device, characterized in that, The electronic device includes a processor and a memory; wherein the memory is used to store a computer program, which is loaded and executed by the processor to implement the cluster data storage method as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, Used for storing computer programs; wherein the computer programs, when executed by a processor, implement the cluster data storage method as described in any one of claims 1 to 7.

Images