Data synchronization method and device, server, storage medium and program product

Abstract

The invention relates to the technical field of data management, and discloses a data synchronization method and device, a server, a storage medium and a program product. The method comprises the following steps: acquiring a source cluster and a target cluster to be subjected to data synchronization, and data configuration information; the source cluster and the target cluster are clusters starting different security modes; performing data security authentication on the source cluster and the target cluster based on the data configuration information, and determining whether the source cluster and the target cluster pass the data security authentication; and if the source cluster and the target cluster pass the data security authentication, reading the to-be-synchronized data from the source cluster, and writing the to-be-synchronized data into the target cluster. By implementing the technical scheme of the invention, data synchronization between clusters in different security modes can be realized.

Description

Technical Field

[0001] The present invention relates to the technical field of data management, and in particular to a data synchronization method, device, server, storage medium and program product. Background Art

[0002] To ensure cluster data security, security management measures are usually enabled. However, after enabling the security mode, although data security is enhanced, it also increases the complexity of data synchronization between clusters with different security modes. Different security modes have unique security management mechanisms, which makes it impossible for them to communicate directly and synchronize data.

[0003] Currently, in order to achieve data synchronization between different clusters, users usually need to write programs to extract data from the source cluster, temporarily store it in a temporary cluster that does not enable security mode, and then write the data to the destination cluster. This will cause users to write complex business programs by themselves, and the operation is relatively complicated. At the same time, it is impossible to detect data integrity in real time, and manual verification is required during the synchronization process. Summary of the invention

[0004] In view of this, the present invention provides a data synchronization method, device, server, storage medium and program product to solve the problem of difficulty in data synchronization between clusters in different security modes.

[0005] In a first aspect, the present invention provides a data synchronization method, comprising: obtaining a source cluster and a destination cluster for data synchronization, and data configuration information; the source cluster and the destination cluster are clusters with different security modes enabled; performing data security authentication on the source cluster and the destination cluster based on the data configuration information, and determining whether the source cluster and the destination cluster pass the data security authentication; if the source cluster and the destination cluster pass the data security authentication, reading the data to be synchronized from the source cluster, and writing the data to be synchronized to the destination cluster.

[0006] The data synchronization method provided by the embodiment of the present invention can handle data synchronization between a source cluster and a destination cluster with different security modes enabled, that is, it can adapt to data transmission requirements under various security settings. Data security authentication is performed on the source cluster and the destination cluster based on the data configuration information, which ensures security during the data transmission process and ensures that only authenticated source and destination clusters can perform data synchronization operations, thereby preventing unauthorized access or data leakage. Once the security authentication is passed, the method can efficiently read the data to be synchronized from the source cluster and write it to the destination cluster. This effective data synchronization operation can help ensure data consistency and integrity.

[0007] In an optional implementation, obtaining data configuration information includes: obtaining a first security mode corresponding to a source cluster, and a second security mode corresponding to a destination cluster; obtaining a first authentication credential corresponding to the first security mode, and a second authentication credential corresponding to the second security mode; performing data configuration on the source cluster based on the first security mode and the first authentication credential to obtain data configuration information corresponding to the source cluster; performing data configuration on the destination cluster based on the second security mode and the second authentication credential to obtain data configuration information corresponding to the destination cluster.

[0008] The data synchronization method provided by the embodiment of the present invention can be flexibly configured according to specific security requirements by obtaining different security modes corresponding to the source cluster and the destination cluster, so as to meet the security requirements in different environments. In the process of obtaining authentication credentials, it is ensured that each cluster has its own exclusive security verification mechanism, which further improves the security of the data synchronization process and reduces the risk of illegal access to data.

[0009] In an optional implementation, access control information corresponding to the source cluster is obtained; the access control information is configured into the data configuration information corresponding to the source cluster, so as to determine the index data to be synchronized in the source cluster based on the access control information and the first security mode, and write the index data into the destination cluster.

[0010] The data synchronization method provided by the embodiment of the present invention realizes access control of cluster data, allows customization of index data to be synchronized, can effectively avoid synchronization of unauthorized data, and ensure that core sensitive data is not leaked. The index data to be synchronized is determined based on access control information and security mode, which improves the accuracy and relevance of data synchronization. It avoids unnecessary data transmission, optimizes the use of bandwidth and storage resources, and improves overall system performance.

[0011] In an optional implementation, data transmission bandwidth information between the source cluster and the destination cluster is obtained; and the data transmission bandwidth information is configured into data configuration information corresponding to the source cluster and data configuration information corresponding to the destination cluster.

[0012] The data synchronization method provided by the embodiment of the present invention can effectively adjust the data transmission strategy by determining the bandwidth information between the source cluster and the destination cluster, ensure data synchronization under the best conditions, and improve transmission efficiency. It can relieve the pressure on the cluster and effectively reduce the impact on the original business of the cluster during data synchronization by customizing the data transmission bandwidth.

[0013] In an optional implementation, data security authentication is performed on the source cluster and the destination cluster based on the data configuration information to determine whether the source cluster and the destination cluster have passed the data security authentication, including: determining whether the source cluster and the destination cluster have enabled security mode based on the data configuration information; when it is determined that the source cluster has enabled security mode, performing data security authentication on the source cluster based on the authentication credentials corresponding to the security mode in the data configuration information to obtain an authentication result; when it is determined that the destination cluster has enabled security mode, performing data security authentication on the destination cluster based on the authentication credentials corresponding to the security mode in the data configuration information to obtain an authentication result; the authentication result is used to characterize whether the source cluster and the destination cluster have passed the security authentication.

[0014] The data synchronization method provided by the embodiment of the present invention dynamically determines the security mode based on the data configuration information, so that the system can flexibly adjust the authentication mechanism according to specific needs and adapt to different security requirements. The authentication result indicates the validity of both parties of the data transmission, ensuring that only verified clusters can interact with data, thereby maintaining the integrity of the data. Combining the authentication process with the data configuration information can automatically execute the authentication process, reduce manual intervention, and improve efficiency.

[0015] In an optional implementation, the security mode includes a third security mode and a fourth security mode; the authentication credentials include a security file credential corresponding to the third security mode, and an identity authentication credential corresponding to the fourth security mode; when it is determined that the source cluster turns on the security mode, data security authentication is performed on the source cluster based on the authentication credential corresponding to the security mode in the data configuration information to obtain an authentication result, including: when it is determined that the source cluster turns on the third security mode, data security authentication is performed on the source cluster according to the security file credential to obtain an authentication result; when it is determined that the source cluster turns on the fourth security mode, data security authentication is performed on the source cluster according to the identity authentication credential to obtain an authentication result.

[0016] The data synchronization method provided by the embodiment of the present invention can improve the authentication security of the source cluster when the security mode is turned on through different authentication credentials, and ensure the security and confidentiality of the data transmission process. Performing corresponding authentication operations according to specific credential types helps to reduce misoperation and improve authentication efficiency.

[0017] In an optional implementation, when it is determined that the source cluster does not have security mode enabled, it is directly determined that the source cluster security authentication has passed; when it is determined that the destination cluster does not have security mode enabled, it is directly determined that the destination cluster data security authentication has passed.

[0018] The data synchronization method provided by the embodiment of the present invention avoids unnecessary authentication processes, can speed up data transmission, improve the overall efficiency of the system, and significantly reduce delays, especially in low-risk environments. For data transmission that does not involve sensitive data or is in a trusted environment, it is possible to flexibly choose not to enable the security mode, thereby quickly responding to business needs. Avoiding unnecessary authentication processes can speed up data transmission, improve the overall efficiency of the system, and significantly reduce delays, especially in low-risk environments.

[0019] In an optional implementation, the data to be synchronized is read from the source cluster and written to the destination cluster, including: reading data from the source cluster in batches, and performing consistency check on the data read in each batch; for each batch of data read, if the data read in the batch passes the consistency check, writing the data read in the batch to the destination cluster.

[0020] The data synchronization method provided by the embodiment of the present invention can effectively reduce the memory consumption of a single read by reading data in batches, adapt to the needs of large data volume transmission, and improve performance and stability. If a batch of data fails the consistency check, retries or error processing can be performed only on the batch without affecting the entire data synchronization process, thereby improving fault tolerance. At the same time, the consistency of the synchronized data is guaranteed, which can effectively avoid data loss and data corruption problems that occur when synchronizing data.

[0021] In an optional implementation, for each batch of data read, if the data read in the batch fails the consistency check, the data that fails the consistency check is repeatedly reread, and a consistency check is performed on the reread data until the reread data passes the consistency check or the number of rereads reaches a preset number of times; if the number of rereads reaches a preset number of times and the consistency check of the reread data fails each time, the data written in the destination cluster is cleared, and a prompt indicating data synchronization failure is given.

[0022] The data synchronization method provided by the embodiment of the present invention ensures that the data finally written is reliable through repeated reading and consistency verification, thereby avoiding problems caused by data errors. In the event of a consistency verification failure, multiple rereads are allowed to enhance the fault tolerance of the system. A reread mechanism with a preset number of times is set to reasonably control the use of resources and avoid waste of system resources caused by unlimited retries. Once the preset number of rereads is reached and it still fails, the written data is cleared in time to prevent the erroneous data from causing a greater impact in the cluster, and the information on synchronization failure is fed back in time to facilitate the troubleshooting and handling of the problem.

[0023] In a second aspect, the present invention provides a data synchronization device, comprising: an acquisition module, used to acquire a source cluster and a destination cluster for data synchronization, and data configuration information; the source cluster and the destination cluster are clusters with different security modes enabled; an authentication module, used to perform data security authentication on the source cluster and the destination cluster based on the data configuration information, and determine whether the source cluster and the destination cluster pass the data security authentication; a synchronization module, used to read the data to be synchronized from the source cluster and write the data to be synchronized to the destination cluster if the source cluster and the destination cluster pass the data security authentication.

[0024] In a third aspect, the present invention provides a server comprising: a memory and a processor, the memory and the processor being communicatively connected to each other, the memory storing computer instructions, and the processor executing the data synchronization method of the first aspect or any corresponding embodiment thereof by executing the computer instructions.

[0025] In a fourth aspect, the present invention provides a computer-readable storage medium having computer instructions stored thereon, the computer instructions being used to enable a computer to execute the data synchronization method of the first aspect or any corresponding embodiment thereof.

[0026] In a fifth aspect, the present invention provides a computer program product, comprising computer instructions for causing a computer to execute the data synchronization method of the first aspect or any corresponding embodiment thereof. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] In order to more clearly illustrate the specific implementation methods of the present invention or the technical solutions in the prior art, the drawings required for use in the specific implementation methods or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some implementation methods of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0028] Figure 1 is a flow chart of a data synchronization method according to an embodiment of the present invention;

[0029] Figure 2 is a flow chart of another data synchronization method according to an embodiment of the present invention;

[0030] Figure 3 is a flowchart of another data synchronization method according to an embodiment of the present invention;

[0031] Figure 4 is a structural block diagram of a data synchronization device according to an embodiment of the present invention;

[0032] Figure 5 It is a schematic diagram of the hardware structure of the server according to an embodiment of the present invention. DETAILED DESCRIPTION

[0033] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present invention.

[0034] Elasticsearch is a powerful full-text search system that is widely used in various industries. To ensure the security of data in the cluster, the security management function is usually enabled. There are currently two mainstream security modes: one is the security management mode based on X-Pack, and the other is the security management mode based on Kerberos. These two modes have their own characteristics, adapt to different security requirements, and can effectively protect the security and privacy of data.

[0035] After enabling the security mode of the Elasticsearch cluster, although data security can be effectively guaranteed, it also brings challenges to data synchronization between clusters with different security modes. This is because the differences in security management mechanisms make it impossible for clusters to communicate directly. The current solution is usually for users to write programs by themselves to extract data from the source cluster to a temporary storage system (such as an Elasticsearch cluster without security mode enabled), and then write the data to the destination cluster. However, this method has obvious disadvantages: users need to develop business programs by themselves, which is more complicated to operate; the integrity cannot be detected in real time during data synchronization and manual verification is required; in addition, cluster load and network bandwidth are often ignored, resulting in the synchronization process increasing the burden on the cluster, thereby affecting normal business operations.

[0036] In view of this, the technical solution of the present invention designs an Elasticsearch inter-cluster data synchronization tool (Elasticsearch Data Synchronization Tool, EDST) that supports different security modes, which can directly realize data synchronization between two clusters. The tool allows users to specify the Elasticsearch index data that needs to be synchronized. Through this data access control, the leakage of core sensitive data can be effectively prevented. During the synchronization process, the tool will perform data verification to ensure data consistency. Users only need to implement data synchronization between clusters with different security modes according to the configuration. At the same time, they can flexibly set the data transmission bandwidth to avoid affecting the normal business of the cluster, and specify the synchronization data at the index level to prevent the leakage of unauthorized data.

[0037] According to an embodiment of the present invention, a data synchronization method embodiment is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0038] In this embodiment, a data synchronization method is provided, which can be used in a server, wherein a data synchronization tool is configured in the server. Figure 1 is a flow chart of a data synchronization method according to an embodiment of the present invention. Figure 1 As shown, the process includes the following steps:

[0039] Step S101, obtaining a source cluster and a destination cluster for data synchronization, as well as data configuration information; the source cluster and the destination cluster are clusters with different security modes enabled.

[0040] The source cluster refers to the Elasticsearch cluster that stores the data to be synchronized. The data in this cluster will be read and synchronized to the destination cluster. The destination cluster refers to the Elasticsearch cluster that receives the synchronized data. The synchronized data will be stored in this cluster. Specifically, users can select the source cluster and the destination cluster from a series of available clusters through the command line interface (CLI) or remote management tools (such as SSH, remote desktop, etc.), and transmit the selected source cluster and destination cluster information to the server.

[0041] Data configuration information refers to the configuration information entered by the user during the initial setup, including key information required for data synchronization between the source cluster and the destination cluster, such as network address information, security mode type information, and authentication credential information of the source cluster and the destination cluster. Among them, the security mode refers to the security measures implemented in the cluster, such as: no security mode, Kerberos security mode, and X-Pack security mode. Specifically, the user can enter the data configuration information of the source cluster and the destination cluster in the command line interface or remote management tool, such as entering the IP address, selecting the security mode, entering the authentication information, etc. After the user fills in the information, click Submit, and the server receives the input and stores it in the memory for subsequent use.

[0042] Step S102: Perform data security authentication on the source cluster and the destination cluster based on the data configuration information to determine whether the source cluster and the destination cluster pass the data security authentication.

[0043] Data security authentication is performed on the source cluster and the destination cluster according to the data configuration information configured by the user to ensure that only authorized clusters can access and synchronize data, and to ensure the security and integrity of data transmission. Specifically, the corresponding security mode information of the source cluster and the destination cluster is extracted from the data configuration information, and the authentication mechanism used to perform data security authentication on the source cluster and the destination cluster is determined according to the security mode information. After the data security authentication of the source cluster and the destination cluster is performed through the determined authentication mechanism, the authentication result is obtained, that is, whether the source cluster and the destination cluster have passed the data security authentication.

[0044] Step S103: If the source cluster and the destination cluster pass the data security authentication, the data to be synchronized is read from the source cluster, and the data to be synchronized is written to the destination cluster.

[0045] Data to be synchronized refers to the data selected from the source cluster that needs to be copied or synchronized to the destination cluster. Specifically, when it is determined that the source cluster and the destination cluster have passed the data security authentication, the index and data range to be synchronized in the source cluster are determined according to the data configuration information, a request is initiated, the data to be synchronized is read from the source cluster, and the data read from the source cluster is written to the destination cluster.

[0046] The data synchronization method provided by the embodiment of the present invention can handle data synchronization between a source cluster and a destination cluster with different security modes enabled, that is, it can adapt to data transmission requirements under various security settings. Data security authentication is performed on the source cluster and the destination cluster based on the data configuration information, which ensures security during the data transmission process and ensures that only authenticated source and destination clusters can perform data synchronization operations, thereby preventing unauthorized access or data leakage. Once the security authentication is passed, the method can efficiently read the data to be synchronized from the source cluster and write it to the destination cluster. This effective data synchronization operation can help ensure data consistency and integrity.

[0047] In this embodiment, a data synchronization method is provided, which can be used for a server. Figure 2 is a flow chart of a data synchronization method according to an embodiment of the present invention. Figure 2 As shown, the process includes the following steps:

[0048] Step S201, obtaining a source cluster and a destination cluster for data synchronization, as well as data configuration information; the source cluster and the destination cluster are clusters with different security modes enabled.

[0049] Specifically, the above step S201 includes:

[0050] Step S2011: Get the source cluster and the destination cluster for data synchronization. Figure 1 Step S101 of the illustrated embodiment will not be described in detail here.

[0051] Step S2012: Acquire a first security mode corresponding to the source cluster and a second security mode corresponding to the destination cluster.

[0052] The first security mode refers to the security mode configured by the user for the source cluster, and the second security mode is the security mode setting configured for the destination cluster, such as no security mode, Kerberos security mode, and X-Pack security mode. Specifically, the user can select the security mode of the source cluster and the destination cluster in the command line interface or remote management tool, and transmit the selected security mode information to the server.

[0053] Step S2013, obtaining a first authentication credential corresponding to the first security mode and a second authentication credential corresponding to the second security mode.

[0054] The first authentication credential refers to the authentication information corresponding to the first security mode, and the second authentication credential refers to the authentication information corresponding to the second security mode, such as a user name and password, a certificate, or other forms of credentials. Specifically, after determining the first security mode corresponding to the source cluster and the second security mode corresponding to the destination cluster, the user may apply for authorization from the cluster administrator to obtain the first authentication credential corresponding to the first security mode and the second authentication credential corresponding to the second security mode, and the user may transmit the first authentication credential and the second authentication credential to the server through a command line interface or a remote management tool.

[0055] Step S2014: perform data configuration on the source cluster based on the first security mode and the first authentication credential to obtain data configuration information corresponding to the source cluster.

[0056] Based on the first security mode and the first authentication credential, a specific configuration of the source cluster is prepared for data synchronization. Specifically, based on the first security mode and the first authentication credential, the source cluster is configured, for example, including index settings, data mapping, and permission control, and specific data configuration information of the source cluster is output as a basis for subsequent data synchronization operations.

[0057] Step S2015: Perform data configuration on the destination cluster based on the second security mode and the second authentication credential to obtain data configuration information corresponding to the destination cluster.

[0058] Based on the second security mode and the second authentication credential, a specific configuration of the destination cluster is prepared for data synchronization. Specifically, the destination cluster is configured based on the second security mode and the second authentication credential, and specific data configuration information of the destination cluster is output, such as index configuration and security settings of received data.

[0059] The data synchronization method provided by the embodiment of the present invention can be flexibly configured according to specific security requirements by obtaining different security modes corresponding to the source cluster and the destination cluster, so as to meet the security requirements in different environments. In the process of obtaining authentication credentials, it is ensured that each cluster has its own exclusive security verification mechanism, which further improves the security of the data synchronization process and reduces the risk of illegal access to data.

[0060] Step S2016, obtaining access control information corresponding to the source cluster.

[0061] Access control information refers to the permissions and restrictions related to the index data in the source cluster. This information is used to determine the specific data sets that can be accessed and synchronized, such as a list of index names that need to be synchronized, which can be the entire cluster or a specific set of indexes. Specifically, the user enters the access control information in the command line interface or remote management tool. This information can include, for example, user-defined access rules, roles, and indexes that are allowed to be accessed, and transmits the access control information to the server. In this way, the user-specified access control information can be obtained and stored for subsequent processing.

[0062] Step S2017: configure the access control information into the data configuration information corresponding to the source cluster, determine the index data to be synchronized in the source cluster based on the access control information and the first security mode, and write the index data into the destination cluster.

[0063] Index data refers to the specific data in the source cluster that needs to be synchronized to the target cluster. Specifically, the access control information entered by the user in the command line interface or remote management tool is integrated into the data configuration information of the source cluster. During the data synchronization process, the integrated data configuration information is parsed to identify which indexes comply with the user-defined access rules. The first security mode involves a security assessment of the index data to ensure that only data that meets specific security standards is allowed to be synchronized. When determining the index to be synchronized, both the access control information and the first security mode are considered to ensure security compliance. For example, if an index is marked as highly sensitive and is not in the access list, the data of the index will be excluded from synchronization. Once the index to be synchronized is determined, the corresponding data can be extracted from the source cluster using the Elasticsearch query application programming interface (API). The filtered data is written to the target cluster using Elasticsearch's write.

[0064] If it is determined that no access control information is configured, all index data in the source cluster is synchronized to the destination cluster by default, and the data transmission bandwidth is not restricted during the data transmission process.

[0065] The data synchronization method provided by the embodiment of the present invention realizes access control of cluster data, allows customization of index data to be synchronized, can effectively avoid synchronization of unauthorized data, and ensure that core sensitive data is not leaked. The index data to be synchronized is determined based on access control information and security mode, which improves the accuracy and relevance of data synchronization. It avoids unnecessary data transmission, optimizes the use of bandwidth and storage resources, and improves overall system performance.

[0066] Step S2018: Obtain data transmission bandwidth information between the source cluster and the destination cluster.

[0067] Data transmission bandwidth information refers to the available bandwidth limit of the network connection between the source cluster and the destination cluster, for example, it can be the maximum data transmission rate available in data transmission. Specifically, the user can use a drop-down menu in the command line interface or remote management tool to allow the user to select common bandwidth options (such as 10Mbps, 100Mbps, 1Gbps, etc.), or provide a custom input box for the user to enter a specific value, which is not limited here. The determined data transmission bandwidth information is transmitted to the server.

[0068] Step S2019: configure the data transmission bandwidth information into the data configuration information corresponding to the source cluster and the data configuration information corresponding to the destination cluster.

[0069] Specifically, the acquired data transmission bandwidth information from the source cluster to the destination cluster is input or edited into a corresponding field or option in the data configuration information, thereby optimizing the performance and stability of data transmission.

[0070] The data synchronization method provided by the embodiment of the present invention can effectively adjust the data transmission strategy by determining the bandwidth information between the source cluster and the destination cluster, ensure data synchronization under the best conditions, and improve transmission efficiency. It can relieve the pressure on the cluster and effectively reduce the impact on the original business of the cluster during data synchronization by customizing the data transmission bandwidth.

[0071] Step S202: Perform data security authentication on the source cluster and the destination cluster based on the data configuration information to determine whether the source cluster and the destination cluster have passed the data security authentication. Figure 1 Step S102 of the illustrated embodiment will not be described in detail here.

[0072] Step S203: If the source cluster and the destination cluster pass the data security authentication, the data to be synchronized is read from the source cluster and written to the destination cluster. Figure 1 Step S103 of the illustrated embodiment will not be described in detail here.

[0073] In this embodiment, a data synchronization method is provided, which can be used for a server. Figure 3is a flow chart of a data synchronization method according to an embodiment of the present invention. Figure 3 As shown, the process includes the following steps:

[0074] Step S301: Obtain the source cluster and target cluster for data synchronization, as well as data configuration information; the source cluster and target cluster are clusters with different security modes enabled. Figure 2 Step S201 of the illustrated embodiment will not be described in detail here.

[0075] Step S302: Perform data security authentication on the source cluster and the destination cluster based on the data configuration information to determine whether the source cluster and the destination cluster pass the data security authentication.

[0076] Specifically, the above step S302 includes:

[0077] Step S3021: Determine whether the source cluster and the destination cluster are in safety mode based on the data configuration information.

[0078] Check whether the source cluster and the destination cluster have security mode enabled according to the configured data configuration information. Specifically, if the securityMode attribute of the source cluster or the destination cluster is None, it means that the security mode is not enabled. If the securityMode attribute is Kerberos, X-Pack, or other security mode, it means that the security mode is enabled and subsequent authentication is required.

[0079] Step S3022: When it is determined that the source cluster starts the security mode, data security authentication is performed on the source cluster based on the authentication credentials corresponding to the security mode in the data configuration information to obtain an authentication result.

[0080] Authentication credentials are security information used to verify the identity of a user or system. For example, authentication credentials can include a Keytab file for Kerberos security mode, which contains keys and related information to allow security authentication; they can also include a user name and password for X-Pack security mode, which is used to authenticate identity by entering valid credentials. Specifically, if the source cluster has security mode enabled, use the corresponding authentication credentials for data security authentication.

[0081] In some optional implementations, the security mode includes a third security mode and a fourth security mode; the authentication credential includes a security file credential corresponding to the third security mode and an identity authentication credential corresponding to the fourth security mode; the above step S3022 includes:

[0082] Step a1: when it is determined that the source cluster starts the third security mode, data security authentication is performed on the source cluster according to the security file credential to obtain an authentication result.

[0083] Specifically, the third security mode may be a Kerberos security mode, and the security file credential may be a Keytab file. If the source cluster has Kerberos security mode enabled, authentication is performed based on the Keytab file, for example, by calling a kerberosCheck() method to verify the validity of the credential.

[0084] Step a2: when it is determined that the source cluster starts the fourth security mode, data security authentication is performed on the source cluster according to the identity authentication credential to obtain an authentication result.

[0085] Specifically, the fourth security mode may be an X-Pack security mode, and the identity authentication credential may be a user name and password. If the source cluster has the X-Pack security mode enabled, authentication is performed based on the user name and password, for example, by calling the xpackCheck() method to check whether the credential is correct.

[0086] In the above implementation, different authentication credentials can be used to improve the authentication security of the source cluster when the security mode is turned on, ensuring the security and confidentiality of the data transmission process. Performing corresponding authentication operations according to specific credential types helps to reduce misoperations and improve authentication efficiency.

[0087] Step S3023, when it is determined that the destination cluster has enabled the security mode, data security authentication is performed on the destination cluster based on the authentication credentials corresponding to the security mode in the data configuration information to obtain an authentication result; the authentication result is used to indicate whether the source cluster and the destination cluster have passed the security authentication.

[0088] If the destination cluster also has security mode enabled, data security authentication is performed based on the corresponding authentication credentials. Specifically, when it is determined that the destination cluster has Kerberos security mode enabled (third security mode), authentication is performed based on the Keytab file. When it is determined that the destination cluster has X-Pack security mode enabled (fourth security mode), authentication is performed based on the user name and password.

[0089] Step S3024, when it is determined that the source cluster does not have the security mode enabled, directly determine that the source cluster security authentication has passed; when it is determined that the destination cluster does not have the security mode enabled, directly determine that the destination cluster data security authentication has passed.

[0090] When the securityMode attribute of the source cluster is detected as None, it means that the source cluster has not enabled any security measures, that is, the source cluster has not enabled security mode. Specifically, when it is determined that the source cluster has not enabled security mode, the source cluster is considered to have passed the security authentication directly. During the data synchronization process, the source cluster does not need to perform any form of identity authentication or credential check and directly allows data transmission.

[0091] Similarly, when the securityMode attribute of the destination cluster is detected as None, it means that the destination cluster has not enabled any security measures, that is, the destination cluster has not enabled security mode. Specifically, when it is determined that the destination cluster has not enabled security mode, the destination cluster is considered to have passed security authentication directly. During the data synchronization process, the destination cluster does not need to perform any form of identity authentication or credential check and directly allows data transmission.

[0092] The data synchronization method provided by the embodiment of the present invention dynamically determines the security mode based on the data configuration information, so that the system can flexibly adjust the authentication mechanism according to specific needs to adapt to different security requirements. The authentication result shows the effectiveness of both parties of the data transmission, ensuring that only verified clusters can interact with data, thereby maintaining the integrity of the data. Combining the authentication process with the data configuration information can automatically execute the authentication process, reduce manual intervention, and improve efficiency. Avoiding unnecessary authentication processes can speed up data transmission and improve the overall efficiency of the system, especially in low-risk environments, significantly reducing delays. For data transmission that does not involve sensitive data or in a trusted environment, it is possible to flexibly choose not to enable the security mode, thereby quickly responding to business needs. Avoiding unnecessary authentication processes can speed up data transmission and improve the overall efficiency of the system, especially in low-risk environments, significantly reducing delays.

[0093] Step S303: If the source cluster and the destination cluster pass the data security authentication, the data to be synchronized is read from the source cluster, and the data to be synchronized is written to the destination cluster.

[0094] Specifically, the above step S303 includes:

[0095] Step S3031: If the source cluster and the destination cluster pass the data security authentication, data is read from the source cluster in batches, and consistency check is performed on the data read in each batch.

[0096] The batch size is the amount of data to be read each time, which can be adjusted based on server performance and data size. Batch processing is used to read data to optimize performance and reduce memory usage. Specifically, the batch processing conditions pre-set by the user are determined, such as time range, data type, etc., and data is read from the source cluster in batches according to the batch processing conditions.

[0097] Consistency checking is a mechanism to ensure that the data read from the source cluster maintains integrity and accuracy during transmission. Specifically, when reading data, a checksum of the batch of data can be calculated, such as a fixed-length value generated by a hash algorithm. Before the data is transmitted to the destination cluster, the checksum is appended to the end of the data, and then verified in the destination cluster to ensure that the read data has not been modified or lost. By comparing the checksum, it is ensured that the data in the source cluster and the destination cluster are logically consistent, thereby avoiding data inconsistency problems caused by transmission errors.

[0098] Step S3032: For each batch of data read, if the batch of data read passes the consistency check, the batch of data read is written to the destination cluster.

[0099] For each batch of data read, determine whether the data read in the current batch passes the consistency check. If the check passes, the data read in the current batch is considered complete and accurate. Specifically, if the data in the current batch passes the consistency check, continue to write data.

[0100] Furthermore, after a batch of data is written, the source cluster will be connected to check whether the data synchronization is complete. If not, step S3031 will be continued; if the data synchronization is complete, the data synchronization task will be terminated and the user will be prompted that the data synchronization is successfully completed.

[0101] Step S3033, for each batch of data read, if the batch of data read fails the consistency check, the data that fails the consistency check is repeatedly reread, and the reread data is subjected to a consistency check until the reread data passes the consistency check or the number of rereads reaches a preset number.

[0102] The preset number of times is the maximum number of attempts set for the retry operation during the data extraction and consistency check process, for example, it can be 10 times. Specifically, for each batch of data read, if the data of the current batch fails the consistency check, the write operation will not be performed. Instead, the current batch of data will be repeatedly reread according to the preset number of retries, and the consistency check will be performed again on the reread data. If the consistency check is successfully passed during the retry process, step S3032 will continue to be executed and written to the destination cluster. If all retries fail, step S3034 will be triggered.

[0103] Step S3034: If the number of rereads reaches a preset number and the data consistency check fails each time, the written data in the destination cluster is cleared and a prompt indicating data synchronization failure is given.

[0104] Specifically, if the preset number of consistency checks fail, all data written in the current batch will be deleted to prevent inconsistent data from affecting subsequent operations or causing data errors. At the same time, an exception message is generated and returned to clearly indicate the reason for the data synchronization failure.

[0105] The data synchronization method provided by the embodiment of the present invention can effectively reduce the memory consumption of a single read by reading data in batches, adapt to the needs of large data volume transmission, and improve performance and stability. If a batch of data fails to pass the consistency check, retries or error processing can be performed only for the batch without affecting the entire data synchronization process, thereby improving fault tolerance. At the same time, the consistency guarantee of the synchronized data is achieved, which can effectively avoid data loss and data corruption problems when synchronizing data. By repeated reading and consistency checking, it is ensured that the data finally written is reliable, and problems caused by data errors are avoided. When the consistency check fails, multiple rereads are allowed to enhance the fault tolerance of the system. A reread mechanism with a preset number of times is set to reasonably control the use of resources and avoid waste of system resources caused by unlimited retries. Once the preset number of rereads is reached and it still fails, the written data is cleared in time to prevent the erroneous data from causing a greater impact in the cluster, and the information on synchronization failure is fed back in time to facilitate the troubleshooting and processing of the problem.

[0106] In this embodiment, a data synchronization device is also provided, which is used to implement the above-mentioned embodiments and preferred implementation modes, and the descriptions that have been made will not be repeated. As used below, the term "module" can be a combination of software and / or hardware that implements a predetermined function. Although the devices described in the following embodiments are preferably implemented in software, the implementation of hardware, or a combination of software and hardware, is also possible and conceivable.

[0107] This embodiment provides a data synchronization device, such as Figure 4 As shown, including:

[0108] The acquisition module 401 is used to acquire the source cluster and the destination cluster for data synchronization, as well as data configuration information; the source cluster and the destination cluster are clusters with different security modes enabled.

[0109] The authentication module 402 is used to perform data security authentication on the source cluster and the destination cluster based on the data configuration information, and determine whether the source cluster and the destination cluster pass the data security authentication.

[0110] The synchronization module 403 is used to read the data to be synchronized from the source cluster and write the data to be synchronized to the destination cluster if the source cluster and the destination cluster pass the data security authentication.

[0111] In some optional implementations, the acquisition module 401 includes:

[0112] A first acquisition submodule, used to acquire a first security mode corresponding to a source cluster and a second security mode corresponding to a destination cluster;

[0113] A second acquisition submodule, used to acquire a first authentication credential corresponding to the first security mode and a second authentication credential corresponding to the second security mode;

[0114] A first configuration submodule, configured to perform data configuration on the source cluster based on the first security mode and the first authentication credential, and obtain data configuration information corresponding to the source cluster;

[0115] The second configuration submodule is used to perform data configuration on the destination cluster based on the second security mode and the second authentication credential to obtain data configuration information corresponding to the destination cluster.

[0116] In some optional implementations, the acquisition module 401 further includes:

[0117] The third acquisition submodule is used to obtain the access control information corresponding to the source cluster;

[0118] The third configuration submodule is used to configure the access control information into the data configuration information corresponding to the source cluster, so as to determine the index data to be synchronized in the source cluster based on the access control information and the first security mode, and write the index data into the destination cluster.

[0119] In some optional implementations, the acquisition module 401 further includes:

[0120] The fourth acquisition submodule is used to obtain the data transmission bandwidth information between the source cluster and the destination cluster;

[0121] The fourth configuration submodule is used to configure the data transmission bandwidth information into the data configuration information corresponding to the source cluster and the data configuration information corresponding to the destination cluster.

[0122] In some optional implementations, the authentication module 402 includes:

[0123] A first determination submodule is used to determine whether the source cluster and the destination cluster are in a safe mode based on the data configuration information;

[0124] A first authentication submodule is used to perform data security authentication on the source cluster based on the authentication credentials corresponding to the security mode in the data configuration information when it is determined that the source cluster is in security mode, and obtain an authentication result;

[0125] The second authentication submodule is used to perform data security authentication on the destination cluster based on the authentication credentials corresponding to the security mode in the data configuration information when it is determined that the destination cluster has enabled the security mode, and obtain an authentication result; the authentication result is used to indicate whether the source cluster and the destination cluster have passed the security authentication.

[0126] In some optional implementations, the security mode includes a third security mode and a fourth security mode; the authentication credential includes a security file credential corresponding to the third security mode and an identity authentication credential corresponding to the fourth security mode; and the first authentication submodule includes:

[0127] A first authentication unit is used to perform data security authentication on the source cluster according to the security file credential to obtain an authentication result when it is determined that the source cluster starts the third security mode;

[0128] The second authentication unit is used to perform data security authentication on the source cluster according to the identity authentication credential to obtain an authentication result when it is determined that the source cluster has enabled the fourth security mode.

[0129] In some optional implementations, the authentication module 402 further includes:

[0130] The second determination submodule is used to directly determine that the source cluster security authentication has passed when it is determined that the source cluster has not enabled the security mode;

[0131] The third determination submodule is used to directly determine that the data security authentication of the destination cluster has passed when it is determined that the destination cluster has not enabled the security mode.

[0132] In some optional implementations, the synchronization module 403 includes:

[0133] The reading submodule is used to read data from the source cluster in batches if the source cluster and the destination cluster pass the data security authentication, and perform consistency check on the data read in each batch;

[0134] The synchronization submodule is used to write the data read in each batch to the destination cluster if the data read in the batch passes the consistency check.

[0135] In some optional implementations, the synchronization module 403 further includes:

[0136] The retry submodule is used to repeatedly reread the data that failed the consistency check for each batch of data read, and perform consistency check on the reread data until the reread data passes the consistency check or the number of rereads reaches a preset number;

[0137] The deletion submodule is used to clear the written data in the destination cluster and prompt data synchronization failure if the number of rereads reaches the preset number and the data consistency check fails each time.

[0138] The further functional description of each of the above modules and units is the same as that of the above corresponding embodiments and will not be repeated here.

[0139] The data synchronization device in this embodiment is presented in the form of a functional unit, where the unit refers to an ASIC (Application Specific Integrated Circuit) circuit, a processor and memory that executes one or more software or fixed programs, and / or other devices that can provide the above functions.

[0140] The data synchronization device provided by the embodiment of the present invention can handle data synchronization between a source cluster and a destination cluster with different security modes enabled, that is, it can adapt to data transmission requirements under various security settings. Data security authentication is performed on the source cluster and the destination cluster based on the data configuration information, which ensures security during the data transmission process and ensures that only authenticated source and destination clusters can perform data synchronization operations, thereby preventing unauthorized access or data leakage. Once the security authentication is passed, the method can efficiently read the data to be synchronized from the source cluster and write it to the destination cluster. This effective data synchronization operation can help ensure the consistency and integrity of the data.

[0141] The embodiment of the present invention also provides a server having the above Figure 4 The data synchronization device shown.

[0142] See also Figure 5 , Figure 5 is a schematic diagram of the structure of a server provided by an optional embodiment of the present invention, such as Figure 5 As shown, the server includes: one or more processors 10, a memory 20, and interfaces for connecting various components, including high-speed interfaces and low-speed interfaces. The various components are connected to each other using different buses for communication, and can be installed on a common mainboard or installed in other ways as needed. The processor can process instructions executed in the server, including instructions stored in or on the memory to display graphical information of the GUI on an external input / output device (such as a display device coupled to the interface). In some optional embodiments, if necessary, multiple processors and / or multiple buses can be used together with multiple memories and multiple memories. Similarly, multiple servers can be connected, and each device provides some necessary operations (for example, as a server array, a group of blade servers, or a multi-processor system). Figure 5 A processor 10 is taken as an example.

[0143] The processor 10 may be a central processing unit, a network processor or a combination thereof. The processor 10 may further include a hardware chip. The hardware chip may be a dedicated integrated circuit, a programmable logic device or a combination thereof. The programmable logic device may be a complex programmable logic device, a field programmable gate array, a general purpose array logic or any combination thereof.

[0144] The memory 20 stores instructions executable by at least one processor 10, so that at least one processor 10 executes the method shown in the above embodiment.

[0145] The memory 20 may include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required by at least one function; the data storage area may store data created according to the use of the server, etc. In addition, the memory 20 may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some optional embodiments, the memory 20 may optionally include a memory remotely arranged relative to the processor 10, and these remote memories may be connected to the server via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0146] The memory 20 may include a volatile memory, such as a random access memory; the memory may also include a non-volatile memory, such as a flash memory, a hard disk or a solid state drive; the memory 20 may also include a combination of the above types of memory.

[0147] The server further comprises a communication interface 30 for the server to communicate with other devices or a communication network.

[0148] The embodiment of the present invention also provides a computer-readable storage medium. The method according to the embodiment of the present invention can be implemented in hardware, firmware, or can be implemented as a computer code that can be recorded in a storage medium, or can be implemented as a computer code that is originally stored in a remote storage medium or a non-temporary machine-readable storage medium and will be stored in a local storage medium through a network download, so that the method described herein can be stored in such software processing on a storage medium using a general-purpose computer, a dedicated processor, or programmable or dedicated hardware. Among them, the storage medium can be a magnetic disk, an optical disk, a read-only storage memory, a random access memory, a flash memory, a hard disk or a solid-state hard disk, etc.; further, the storage medium can also include a combination of the above types of memories. It can be understood that a computer, a processor, a microprocessor controller, or programmable hardware includes a storage component that can store or receive software or computer code. When the software or computer code is accessed and executed by a computer, a processor, or hardware, the method shown in the above embodiment is implemented.

[0149] A part of the present invention may be applied as a computer program product, such as a computer program instruction, which, when executed by a computer, can call or provide the method and / or technical solution according to the present invention through the operation of the computer. Those skilled in the art should understand that the existence of the computer program instruction in a computer-readable medium includes, but is not limited to, a source file, an executable file, an installation package file, etc., and accordingly, the way in which the computer program instruction is executed by the computer includes, but is not limited to: the computer directly executes the instruction, or the computer compiles the instruction and then executes the corresponding compiled program, or the computer reads and executes the instruction, or the computer reads and installs the instruction and then executes the corresponding installed program. Here, the computer-readable medium may be any available computer-readable storage medium or communication medium accessible to the computer.

[0150] Although the embodiments of the present invention have been described in conjunction with the accompanying drawings, those skilled in the art may make various modifications and variations without departing from the spirit and scope of the present invention, and such modifications and variations are all within the scope defined by the appended claims.

Claims

1. A data synchronization method, characterized in that: The method comprises: Obtain a source cluster and a destination cluster for data synchronization, as well as data configuration information; the source cluster and the destination cluster are clusters with different security modes enabled; Performing data security authentication on the source cluster and the destination cluster based on the data configuration information, and determining whether the source cluster and the destination cluster pass the data security authentication; If the source cluster and the destination cluster pass the data security authentication, the data to be synchronized is read from the source cluster, and the data to be synchronized is written to the destination cluster.

2. The method according to claim 1, characterized in that Get data configuration information, including: Obtaining a first security mode corresponding to the source cluster and a second security mode corresponding to the destination cluster; Obtaining a first authentication credential corresponding to the first security mode and a second authentication credential corresponding to the second security mode; Performing data configuration on the source cluster based on the first security mode and the first authentication credential to obtain the data configuration information corresponding to the source cluster; Data configuration is performed on the destination cluster based on the second security mode and the second authentication credential to obtain the data configuration information corresponding to the destination cluster.

3. The method according to claim 2, characterized in that Also includes: Obtain access control information corresponding to the source cluster; The access control information is configured into data configuration information corresponding to the source cluster, so as to determine index data to be synchronized in the source cluster based on the access control information and the first security mode, and write the index data into the destination cluster.

4. The method according to claim 3, characterized in that Also includes: Obtaining data transmission bandwidth information between the source cluster and the destination cluster; The data transmission bandwidth information is configured into the data configuration information corresponding to the source cluster and the data configuration information corresponding to the destination cluster.

5. The method according to claim 1, characterized in that The performing data security authentication on the source cluster and the destination cluster based on the data configuration information, and determining whether the source cluster and the destination cluster pass the data security authentication, includes: Determine whether the source cluster and the destination cluster are in safe mode based on the data configuration information; When it is determined that the source cluster turns on the security mode, performing data security authentication on the source cluster based on the authentication credentials corresponding to the security mode in the data configuration information to obtain an authentication result; When it is determined that the destination cluster turns on the security mode, data security authentication is performed on the destination cluster based on the authentication credentials corresponding to the security mode in the data configuration information to obtain an authentication result; the authentication result is used to indicate whether the source cluster and the destination cluster have passed the security authentication.

6. The method according to claim 5, characterized in that The security mode includes a third security mode and a fourth security mode; the authentication credential includes a security file credential corresponding to the third security mode and an identity authentication credential corresponding to the fourth security mode; When it is determined that the source cluster turns on the security mode, data security authentication is performed on the source cluster based on the authentication credentials corresponding to the security mode in the data configuration information to obtain an authentication result, including: When it is determined that the source cluster turns on the third security mode, performing data security authentication on the source cluster according to the security file credential to obtain the authentication result; When it is determined that the source cluster starts the fourth security mode, data security authentication is performed on the source cluster according to the identity authentication credential to obtain the authentication result.

7. The method according to claim 5, characterized in that Also includes: When it is determined that the source cluster does not enable the security mode, directly determining that the source cluster security authentication passes; When it is determined that the target cluster does not enable the security mode, it is directly determined that the data security authentication of the target cluster passes.

8. The method according to claim 1, characterized in that The reading the to-be-synchronized data from the source cluster and writing the to-be-synchronized data into the destination cluster includes: Read data from the source cluster in batches, and perform consistency check on the data read in each batch; For each batch of data read, if the data read in the batch passes the consistency check, the data read in the batch is written to the destination cluster.

9. The method according to claim 8, characterized in that Also includes: For each batch of data read, if the data read in the batch fails the consistency check, the data that fails the consistency check is repeatedly reread, and the reread data is subjected to a consistency check until the reread data passes the consistency check or the number of rereads reaches a preset number; If the number of rereads reaches the preset number and the data consistency check fails each time, the written data in the destination cluster is cleared and a prompt indicating data synchronization failure is given.

10. A data synchronization device, characterized in that: The device comprises: An acquisition module, used to acquire a source cluster and a destination cluster for data synchronization, as well as data configuration information; the source cluster and the destination cluster are clusters with different security modes enabled; An authentication module, configured to perform data security authentication on the source cluster and the destination cluster based on the data configuration information, and determine whether the source cluster and the destination cluster pass the data security authentication; A synchronization module is used to read the data to be synchronized from the source cluster and write the data to be synchronized into the destination cluster if the source cluster and the destination cluster pass the data security authentication.

11. A server, characterized in that: include: A memory and a processor, wherein the memory and the processor are communicatively connected to each other, the memory stores computer instructions, and the processor executes the data synchronization method according to any one of claims 1 to 9 by executing the computer instructions.

12. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a computer to execute the data synchronization method according to any one of claims 1 to 9.

13. A computer program product, characterized in that The method comprises computer instructions, wherein the computer instructions are used to cause a computer to execute the data synchronization method according to any one of claims 1 to 9.

Images