Data synchronization method and data synchronization device

Abstract

The invention discloses a data synchronization method and device and a computer readable storage medium, and the method comprises the steps: generating multiple index layers based on data in KEY black and white lists corresponding to a source ETCD cluster and a target ETCD cluster, generating an index table corresponding to the index layers according to a preset rule, and determining a to-be-matched index based on the index layers and the index table; based on to-be-matched data of the source ETCD cluster and the to-be-matched index, determining a to-be-matched index layer in multiple index layers; based on data in a to-be-matched index layer, performing data screening operation on the to-be-matched data to obtain first matched data; and determining target data based on a KEY white list and the first matching data, and synchronizing the target data to the target ETCD cluster. According to the method, the to-be-matched index is determined through the customized KEY black list and the customized KEY white list, the to-be-matched data is screened according to the to-be-matched index, the target data needing to be synchronized is obtained, and customized data synchronization in a cross-level group scene is achieved.

Description

Technical Field

[0001] The present application relates to the field of data processing technology, and in particular to a data synchronization method and a data synchronization device. Background Art

[0002] As enterprises deepen their digital transformation, containerization technology has become an important part of IT infrastructure. As an open source platform for container orchestration, Kubernetes (K8S) implements application deployment, expansion, and management in an automated manner, greatly improving resource utilization efficiency and application delivery speed. In the Kubernetes cluster, ETCD, as a core storage component, undertakes the important task of storing and managing cluster status data, configuration data, and metadata. Its stability and reliability directly affect the normal operation of the entire Kubernetes system.

[0003] However, as the business of enterprises expands, the limitations of a single ETCD cluster in data management and recovery gradually become apparent, especially in scenarios such as multi-cluster management, cross-cluster data synchronization, and remote disaster recovery. Traditional ETCD data management methods include snapshot backup and mirror synchronization. Snapshot backup saves the entire database state of ETCD in the form of a snapshot, and mirror synchronization (such as mirror-maker) synchronizes ETCD data through mirroring technology. However, since both snapshot backup and mirror synchronization are full synchronization, the synchronization process is time-consuming and resource-intensive, and is inefficient in cross-cluster recovery or remote multi-active scenarios.

[0004] Therefore, how to achieve customized data synchronization in cross-level group scenarios is a problem that needs to be solved urgently. Summary of the invention

[0005] The main purpose of the present application is to provide a data synchronization method, device and computer-readable storage medium, aiming to solve the technical problem of how to achieve customized data synchronization in a cross-level group scenario.

[0006] To achieve the above object, the present application provides a data synchronization method, which comprises the following steps:

[0007] Generate a multi-layer index layer based on the data in the KEY black and white lists corresponding to the source ETCD cluster and the target ETCD cluster, generate an index table corresponding to the index layer according to preset rules, and determine the index to be matched based on the index layer and the index table;

[0008] Based on the to-be-matched data of the source ETCD cluster and the to-be-matched index, determine the to-be-matched index layer in the multi-layer index layer;

[0009] Based on the data in the to-be-matched index layer, a data screening operation is performed on the to-be-matched data to obtain first matching data;

[0010] The target data is determined based on the KEY whitelist and the first matching data, and the target data is synchronized to the target ETCD cluster.

[0011] Furthermore, the step of generating a multi-layer index layer based on the data in the KEY black and white lists corresponding to the source ETCD cluster and the target ETCD cluster, generating an index table corresponding to the index layer according to a preset rule, and determining the index to be matched based on the index layer and the index table includes:

[0012] Performing character combination on the data in the KEY blacklist and whitelist to obtain combined data, and generating a multi-layer index layer based on the character length of each data in the combined data;

[0013] Based on the preset rules and each data in the multi-layer index layer, generate an index table corresponding to the multi-layer index layer;

[0014] The index data corresponding to each index in the index table are compared with each data in the multi-layer index layer to obtain the number of matches corresponding to each index data, and the index to be matched is determined based on the number of matches.

[0015] Furthermore, the step of determining the index layer to be matched in the multi-layer index layer based on the data to be matched of the source ETCD cluster and the index to be matched includes:

[0016] Obtain the to-be-matched KEY corresponding to the to-be-matched data;

[0017] Matching the KEY to be matched with the data corresponding to each of the indexes to be matched, so as to determine the target index corresponding to the KEY to be matched in each of the indexes to be matched;

[0018] The index layer corresponding to the target index in the multiple index layers is used as the index layer to be matched.

[0019] Furthermore, the step of performing a data screening operation on the data to be matched based on the data in the to-be-matched index layer to obtain the first matching data includes:

[0020] Match and verify the data in the to-be-matched index layer with each piece of data in the to-be-matched data, respectively, to obtain second matching data corresponding to the data in the to-be-matched index layer;

[0021] The data other than the second matching data in the to-be-matched data is used as the first matching data.

[0022] Furthermore, the step of determining target data based on the KEY whitelist and the first matching data, and synchronizing the target data to the target ETCD cluster includes:

[0023] Through a preset optimization algorithm, the data in the first matching data are verified one by one to determine whether each data in the first matching data belongs to a category in the KEY black and white list, and the data in the first matching data that belongs to the KEY white list is used as the target data, and the target data is synchronized to the target ETCD cluster.

[0024] Further, after the step of synchronizing the target data to the target ETCD cluster, the data synchronization method further includes:

[0025] Based on the KEY black and white lists, a continuous monitoring operation is performed on the source ETCD cluster to determine the data change KEY in the KEY of the source ETCD cluster;

[0026] Determine whether there is a KEY to be synchronized that matches the KEY blacklist and whitelist in the data change KEY;

[0027] If the KEY to be synchronized exists, the change data corresponding to the KEY to be synchronized is obtained, and the change data is synchronized to the target ETCD cluster.

[0028] Furthermore, after the step of obtaining the changed data corresponding to the KEY to be synchronized and synchronizing the changed data to the target ETCD cluster, the data synchronization method further includes:

[0029] In the process of synchronizing the changed data to the target ETCD cluster, monitor the synchronization operation in real time to see if there is any abnormality;

[0030] If there is an exception in the synchronization operation, restart the data synchronization service and perform the data synchronization operation based on the source ETCD cluster after the most recent full synchronization.

[0031] Furthermore, the data synchronization method further includes:

[0032] When receiving the ETCD certificate file corresponding to the cluster adding operation, perform a validity check operation on the ETCD certificate file;

[0033] If the ETCD certificate file is a valid certificate, a format conversion operation is performed on the ETCD certificate file, and the ETCD certificate file after the format conversion is stored.

[0034] Furthermore, the data synchronization method further includes:

[0035] Based on the ETCD cluster list, display each ETCD certificate file and the corresponding expiration countdown of each ETCD certificate file;

[0036] If there is a target expiration countdown that is about to expire in each of the expiration countdowns, then obtain the ETCD certificate file to be updated corresponding to the target expiration countdown;

[0037] If there is automatic update information in the configuration information corresponding to the ETCD certificate file to be updated, a new ETCD certificate file corresponding to the ETCD certificate file to be updated is generated;

[0038] In the database, the ETCD certificate file to be updated is updated based on the new ETCD certificate file, and the ETCD cluster list is updated;

[0039] If there is no automatic update information in the configuration information corresponding to the ETCD certificate file to be updated, an expiration alarm message is sent to the user terminal corresponding to the ETCD certificate file to be updated.

[0040] In addition, to achieve the above-mentioned purpose, the present application also provides a data synchronization device, which includes: a memory, a processor, and a data synchronization program stored on the memory and executable on the processor, and the data synchronization program implements the steps of the aforementioned data synchronization method when executed by the processor.

[0041] The present application generates a multi-layer index layer based on the data in the KEY black and white lists corresponding to the source ETCD cluster and the target ETCD cluster, generates an index table corresponding to the index layer according to preset rules, and determines the index to be matched based on the index layer and the index table; then, based on the data to be matched of the source ETCD cluster and the index to be matched, determines the index layer to be matched in the multi-layer index layer; then, based on the data in the index layer to be matched, determines the first matching data corresponding to the KEY blacklist and the second matching data corresponding to the KEY whitelist in the data to be matched; then, based on the association relationship between the KEY blacklist and the KEY whitelist, the first matching data and the second matching data, determines the target data, and synchronizes the target data to the target ETCD cluster, determines the index to be matched through the customized KEY black and white lists, filters the matching data according to the index to be matched, and obtains the target data to be synchronized, thereby realizing customized data synchronization in cross-level group scenarios, reducing the amount of data processing, and improving the efficiency of backup and recovery. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0043] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0044] Figure 1 A flowchart of the first embodiment of the data synchronization method of the present application is provided;

[0045] Figure 2 This is a schematic diagram of the module structure of the data synchronization device according to an embodiment of the present application.

[0046] The realization of the purpose, functional features and advantages of this application will be further explained in conjunction with embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION

[0047] It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0048] In order to better understand the technical solution of the present application, a detailed description will be given below in conjunction with the accompanying drawings and specific implementation methods.

[0049] The main solution of the present application is: based on the data in the KEY black and white lists corresponding to the source ETCD cluster and the target ETCD cluster, generate a multi-layer index layer, generate an index table corresponding to the index layer according to preset rules, and determine the index to be matched based on the index layer and the index table; based on the data to be matched of the source ETCD cluster and the index to be matched, determine the index layer to be matched in the multi-layer index layer; based on the data in the index layer to be matched, perform data screening operations on the data to be matched to obtain first matching data; determine the target data based on the KEY white list and the first matching data, and synchronize the target data to the target ETCD cluster.

[0050] At present, with the deepening of enterprise digital transformation, containerization technology has become an important part of IT infrastructure. As an open source platform for container orchestration, Kubernetes (K8S) realizes application deployment, expansion and management in an automated way, greatly improving resource utilization efficiency and application delivery speed. In the Kubernetes cluster, ETCD, as the core storage component, undertakes the important task of storing and managing cluster status data, configuration data and metadata. Its stability and reliability directly affect the normal operation of the entire Kubernetes system.

[0051] However, with the expansion of enterprise business, the limitations of a single ETCD cluster in data management and recovery have gradually emerged, especially in scenarios such as multi-cluster management, cross-cluster data synchronization, and remote disaster recovery. Traditional ETCD data management methods include snapshot backup and mirror synchronization. Snapshot backup saves the entire database state of ETCD in the form of a snapshot, and mirror synchronization (such as mirror-maker) synchronizes ETCD data through mirroring technology. However, since both snapshot backup and mirror synchronization are full synchronization, the synchronization process is time-consuming and resource-intensive, and is inefficient in cross-cluster recovery or remote multi-active scenarios. Therefore, how to achieve custom data synchronization in cross-level cluster scenarios is a problem that needs to be solved urgently.

[0052] This application determines the to-be-matched index through a customized KEY black and white list, filters the to-be-matched data according to the to-be-matched index, obtains the target data that needs to be synchronized, realizes customized data synchronization in cross-level group scenarios, reduces the amount of data processing, and improves the efficiency of backup and recovery.

[0053] It should be noted that the execution subject of this embodiment can be a data synchronization device, or a computing service device with data processing, network communication and program running functions, such as a tablet computer, a personal computer, a mobile phone, etc., or a data synchronization device capable of realizing the above functions, etc., and this embodiment does not specifically limit this. The following takes the data synchronization device as the execution subject as an example to illustrate this embodiment and the following embodiments.

[0054] Based on this, this application proposes a data synchronization method of the first embodiment, please refer to Figure 1 , the data synchronization method includes steps S101 to S104:

[0055] Step S101, based on the data in the KEY black and white lists corresponding to the source ETCD cluster and the target ETCD cluster, a multi-layer index layer is generated, an index table corresponding to the index layer is generated according to a preset rule, and an index to be matched is determined based on the index layer and the index table;

[0056] Step S102, based on the to-be-matched data of the source ETCD cluster and the to-be-matched index, determining the to-be-matched index layer in the multi-layer index layer;

[0057] Step S103, based on the data in the to-be-matched index layer, performing a data screening operation on the to-be-matched data to obtain first matching data;

[0058] Step S104: determine target data based on the KEY whitelist and the first matching data, and synchronize the target data to the target ETCD cluster.

[0059] The data synchronization method of the embodiment of the present application can be applied to an integrated management platform, which covers functions such as certificate management, data management, data backup and data synchronization of the ETCD cluster. The management platform can not only solve the complexity of ETCD multi-cluster management, but also realize efficient data recovery in cross-cluster and remote multi-active scenarios through a custom synchronization mechanism.

[0060] The management platform supports a complex multi-user management system. Different users can perform different operations according to their roles and permissions, thus ensuring the security of data management and the controllability of operations. After logging into the system, each user will be authenticated through a multi-factor authentication mechanism. Multi-factor authentication includes username, password, OTP (one-time password) and hardware-based authentication, such as smart cards or biometric systems.

[0061] Specifically, when a user login operation is received, the user name, user password and OTP information corresponding to the user login operation are obtained; based on the user name, user password and OTP information, when the user identity authentication is passed, the login information is returned and redirected to the user information page, and the operation authority corresponding to the user name is determined.

[0062] When the user enters the username, password and OTP information through the web page, the auth component performs user authentication based on the username, password and OTP information. If the user authentication is successful, the auth component returns the login information and redirects to the user information page. After the auth component identifies the user group of the logged-in user, it will grant the corresponding platform operation permissions to the current user to complete the login authentication and identity authentication process. If the user authentication fails, the auth component refuses to log in and records the current IP information and the number of errors. If the number of errors exceeds the preset number of errors, the login attempt will be refused again.

[0063] In addition, the system's built-in permission management module allows administrators to set refined permission policies for different user groups. For example, ordinary users may only be able to perform data query and viewing operations, while administrators have the permission to perform data modification, deletion, backup and recovery, etc. Through a multi-level permission control mechanism, unauthorized operations can be effectively prevented to protect the data security of the ETCD cluster.

[0064] The management platform supports unified management of multiple ETCD clusters, allowing administrators to operate and monitor all clusters through a control interface, simplifying the management process and improving the consistency and efficiency of operations. Through the cluster management interface, users can view the operating status, node health status, and data synchronization of each ETCD cluster in real time. At the same time, the management platform supports batch modification of cluster configurations, reducing the complexity of manual operations. For example, administrators can convert the certificate formats of multiple clusters at one time, or configure backup policies in batches, thereby greatly improving management efficiency. The management platform also supports the conversion and management of cluster certificate formats to ensure that ETCD clusters with different deployment methods can communicate and collaborate smoothly.

[0065] The management platform supports administrators to add, delete, modify and query data in the ETCD cluster through an intuitive UI interface. Administrators can accurately manage data in the ETCD cluster through a simple operation interface without having to directly access the underlying command line interface, reducing the risk of misoperation and improving operational efficiency.

[0066] The management platform also supports batch operation functions. For example, administrators can delete or modify multiple keys at the same time, which is suitable for large-scale data change scenarios. The system also provides data version management functions. Each modification will automatically generate a new data version and retain historical versions to facilitate data rollback or audit when necessary.

[0067] When the user performs a change operation on the ETCD data in the UI on the WEB side, the user information, KEY change data, and change time are passed to the etcd-datatreating module through the API. etcd-datatreating completes the actual change operation on the ETCD data. The etcd-audit service continuously monitors the API of the data processing module etcd-datatreating, writes the monitored content into the database for persistence, and completes the recording and auditing of the entire change declaration cycle. Specifically, when a change operation instruction is received, the user information and the change data of the KEY to be changed corresponding to the change operation instruction are obtained; a backup operation is performed on the KEY to be changed based on the change data, and a change operation is performed on the KEY to be changed based on the user information and the change data.

[0068] In ETCD, KEY stands for key, which is a unique identifier for data storage and retrieval. Each key is associated with a specific data value and ensures data consistency and uniqueness in a distributed system.

[0069] The management platform supports two modes: full synchronization and custom synchronization. You can choose the most appropriate synchronization strategy according to specific needs. Full synchronization is suitable for the multi-active high-availability architecture of the ETCD cluster itself, and ensures data consistency of all cluster nodes through complete data replication.

[0070] The management platform also supports incremental synchronization. By comparing the differences between the source cluster and the target cluster and watching the ETCT key, only the changed data is synchronized, which greatly reduces the time and resource consumption of data synchronization. This is especially important for cross-region synchronization of large-scale clusters. In addition, the system also integrates a real-time monitoring function, which can track the synchronization progress in real time and automatically issue an alarm when an abnormality occurs, ensuring the stability and security of the data synchronization process.

[0071] The management platform provides two modes: scheduled backup and manual backup. Users can choose the most appropriate backup strategy according to their needs. Scheduled backup allows users to configure the backup cycle and time. The system will automatically perform the backup task according to the configuration and back up the data of the ETCD cluster to the specified remote storage or local directory.

[0072] The manual backup function is suitable for emergency or special situations. Users can trigger the backup operation at any time to ensure the immediacy and integrity of data. The management platform also provides a backup strategy management function. Users can set different backup strategies for different clusters or data types to optimize backup efficiency and storage resource utilization.

[0073] The management platform supports multiple backup formats and storage targets, such as OSS remote storage, local disks, and third-party cloud storage services. Users can choose the most appropriate backup method based on actual needs. The management platform also integrates a snapshot recovery function, allowing users to quickly restore data to the state at the time of backup when a cluster fails, greatly reducing system downtime.

[0074] The management platform supports AES256-bit encryption of backup data to ensure the security of backup data during transmission and storage. The management platform also supports multi-version backup management. Users can retain multiple historical versions of backups as needed and roll back or compare data when needed, which improves the flexibility of data management.

[0075] In the custom synchronization mode, the blacklist and whitelist of KEYs that need to be monitored in the ETCD cluster are customized through the front-end UI interface. After receiving the request, the server verifies the content of the KEY and stores it in the database for persistence. At the same time, the source ETCD cluster and target ETCD cluster that need to be synchronized are configured.

[0076] Next, start the etcd-mirror service. The etcd-mirror service performs health checks on the source and target ETCD clusters. If the health check fails, the etcd-mirror service stops and outputs an error message. If the health check passes, it enters the ready state. When etcd-mirror enters the synchronization ready state, it performs a full data synchronization based on the defined blacklist and whitelist.

[0077] In this embodiment, after the health check of the source ETCD cluster and the target ETCD cluster passes, the KEY black and white lists corresponding to the source ETCD cluster and the target ETCD cluster are generated, wherein the KEY black and white lists include a KEY black list and a KEY white list. Then, a multi-layer index layer is generated according to the KEY black and white lists, an index table corresponding to the index layer is generated according to a preset rule, and an index to be matched is determined based on the index layer and the index table. Further, in a feasible implementation manner, step S101 may include steps S1011 to S1013:

[0078] Step S1011, performing character combination on the data in the KEY black and white list to obtain combined data, and generating a multi-layer index layer based on the character length of each data in the combined data;

[0079] Step S1012, generating an index table corresponding to the multi-layer index layer based on the preset rule and each data in the multi-layer index layer;

[0080] Step S1013, comparing the index data corresponding to each index in the index table with each data in the multi-layer index layer, obtaining the number of matches corresponding to each index data, and determining the index to be matched based on the number of matches.

[0081] In this embodiment, after obtaining the KEY black and white list, the data in the KEY black and white list are combined to obtain the combined data, that is, the data in the KEY white list and the data in the KEY black list are combined. For example, the data in the KEY white list includes A and B, and the data in the KEY black list includes C and D, then the combined data includes A, B, C, D, AB, AC, AD, BC, BD, CD, ABC, ABD, ACD, BCD and ABCD, and there is no logical error in the combined data. Then, based on the character length of each data in the combined data, a multi-layer index layer is generated, wherein each index layer corresponds to a character length range, so that data belonging to different character length ranges are allocated to different index layers, so that the distribution of data of different categories is optimized using a hierarchical indexing method.

[0082] After obtaining the multi-layer index layer, an index table corresponding to the multi-layer index layer is generated based on preset rules and the various data in the multi-layer index layer, wherein the index in the index table is the index of each data in the index layer, wherein different data structures can be used to generate the index in the index table, such as Bloom Filter, hash table, etc.

[0083] After obtaining the index table, the index data corresponding to each index in the index table is compared with each data in the index layer to obtain the number of matches corresponding to each index data, and the index to be matched is determined based on the number of matches. Specifically, the index with the least number of matches is selected as the index to be matched to reduce repeated queries and calculation of invalid data.

[0084] After the to-be-matched index is obtained, the to-be-matched index layer is determined in the multi-layer index layer based on the to-be-matched data of the source ETCD cluster and the to-be-matched index; further, in a feasible implementation manner, step S102 may include steps S1021 to S1023:

[0085] Step S1021, obtaining the to-be-matched KEY corresponding to the to-be-matched data;

[0086] Step S1022, matching the to-be-matched KEY with data corresponding to each to-be-matched index, so as to determine a target index corresponding to the to-be-matched KEY in each to-be-matched index;

[0087] Step S1023: taking the index layer corresponding to the target index in the multi-layer index layer as the index layer to be matched.

[0088] In this embodiment, after obtaining the index to be matched, the data to be matched of the source ETCD cluster, i.e., the data to be synchronized, is obtained, and the KEY to be matched corresponding to the data to be matched is obtained, i.e., the KEY of each data in the data to be matched is obtained.

[0089] After obtaining the KEY to be matched, the KEY to be matched is matched with the data corresponding to each index to be matched, and the data matching each KEY to be matched is determined in the data corresponding to each index to be matched, and the data matching each KEY to be matched is used as the target index corresponding to the KEY to be matched. After obtaining the target index, the index layer corresponding to the target index in the multi-layer index layer is used as the index layer to be matched.

[0090] In this embodiment, after the to-be-matched index layer is obtained, a data screening operation is performed on the to-be-matched data based on the data in the to-be-matched index layer to obtain the first matching data; further, in a feasible implementation manner, step S103 may include steps S1031 to S1032:

[0091] Step S1032, performing matching verification on the data in the to-be-matched index layer and each data in the to-be-matched data, respectively, to obtain second matching data corresponding to the data in the to-be-matched index layer;

[0092] Step S1033: taking the data other than the second matching data in the to-be-matched data as the first matching data.

[0093] In this embodiment, after obtaining the index layer to be matched, the data in the index layer to be matched are respectively matched and verified with each data in the data to be matched. Specifically, the data in the index layer to be matched are compared and matched with the KEY to be matched, and the KEY to be matched that matches the data in the index layer to be matched is obtained, and the data in the data to be matched that corresponds to the KEY that matches the data in the index layer to be matched is used as the second matching data.

[0094] After the second matching data is obtained, the data other than the second matching data in the to-be-matched data is used as the first matching data, thereby quickly eliminating most of the unmatched candidate results and obtaining the first matching data.

[0095] In this embodiment, after obtaining the first matching data, the target data is determined based on the KEY whitelist and the first matching data. Specifically, the remaining first matching data are verified one by one through a preset optimization algorithm to finally determine whether the data conforms to a category in the blacklist or whitelist. The data in the first matching data that conforms to the KEY whitelist is used as the target data, and the target data is synchronized to the target ETCD cluster.

[0096] The custom synchronization mode in this embodiment is more flexible and is particularly suitable for the multi-active high-availability architecture in the Kubernetes (K8S) environment. Administrators can configure black and white lists to filter data that does not need to be synchronized to ensure that only critical business data is synchronized, thereby avoiding K8S cluster startup failures caused by cluster information conflicts during cross-cluster synchronization.

[0097] The data synchronization method proposed in this embodiment generates a multi-layer index layer based on the data in the KEY black and white lists corresponding to the source ETCD cluster and the target ETCD cluster, generates an index table corresponding to the index layer according to preset rules, and determines the index to be matched based on the index layer and the index table; then, based on the data to be matched of the source ETCD cluster and the index to be matched, determines the index layer to be matched in the multi-layer index layer; then, based on the data in the index layer to be matched, determines the first matching data corresponding to the KEY blacklist and the second matching data corresponding to the KEY whitelist in the data to be matched; then, based on the association relationship between the KEY blacklist and the KEY whitelist, the first matching data and the second matching data, determines the target data, and synchronizes the target data to the target ETCD cluster, determines the index to be matched through the customized KEY black and white lists, filters the matching data according to the index to be matched, and obtains the target data to be synchronized, thereby realizing customized data synchronization in cross-level group scenarios, reducing the amount of data processing, and improving the efficiency of backup and recovery.

[0098] In a feasible implementation manner, after step S104, the data processing method may further include steps S201 to S203:

[0099] Step S201, performing a continuous monitoring operation on the source ETCD cluster based on the KEY black and white list to determine the data change KEY in the KEY of the source ETCD cluster;

[0100] Step S202, determining whether there is a KEY to be synchronized that matches the KEY blacklist and whitelist in the data change KEY;

[0101] Step S203: If there is a KEY to be synchronized, obtain the change data corresponding to the KEY to be synchronized, and synchronize the change data to the target ETCD cluster.

[0102] In this embodiment, after completing the full data synchronization, the etcd-mirror service enters the listening state and continues to monitor the source ETCD cluster according to the black and white lists. When it is detected that the source ETCD cluster has a KEY that matches the black and white lists, the changed content will be synchronized to the target ETCD cluster in real time.

[0103] Specifically, the source ETCD cluster is continuously monitored based on the KEY black and white list to determine the data change KEY in the KEY of the source ETCD cluster. Then, it is determined in the data change KEY whether there is a KEY to be synchronized that matches the KEY black and white list.

[0104] If there is a key to be synchronized, the change data corresponding to the key to be synchronized is obtained, and the change data is synchronized to the target ETCD cluster to achieve incremental data synchronization across clusters. By comparing the differences between the source cluster and the target cluster and watching the ETCT key, the changed data is synchronized, which greatly reduces the time and resource consumption of data synchronization. After each watch and synchronization of the etcd-mirror service, a log record will be generated to the database.

[0105] In a feasible implementation manner, after step S203, the data processing method may further include steps S204 to S204:

[0106] Step S204, during the process of synchronizing the changed data to the target ETCD cluster, monitor the synchronization operation in real time to see if there is any abnormality;

[0107] Step S205: If there is an exception in the synchronization operation, restart the data synchronization service and perform the data synchronization operation based on the source ETCD cluster after the most recent full synchronization.

[0108] In this embodiment, in the process of synchronizing the changed data to the target ETCD cluster, the etcd-mirror-monitor service monitors each thread of the etcd-mirror service and records the synchronization status of each time to monitor in real time whether there are any abnormalities in the synchronization operation.

[0109] If there is an exception in the synchronization operation, restart the data synchronization service and perform data synchronization based on the source ETCD cluster after the most recent full synchronization to synchronize the changed data and the data after the data. That is, the etcd-mirror-monitor service restarts the etcd-mirror service, and the restarted etcd-mirror service continues to monitor based on the most recent full synchronization.

[0110] It should be noted that when the data synchronization service is restarted, the restart times are accumulated. After the etcd-mirror-monitor service restarts the etcd-mirror service three times in a row, the etcd-mirror service cannot enter the ready state (there are clusters that fail the health check in the source ETCD cluster and the target ETCD cluster), the etcd-mirror service is terminated, and an alarm prompt is triggered to display on the management UI page and push an alarm message by email.

[0111] The data synchronization method proposed in this embodiment continuously monitors the source ETCD cluster based on the KEY black and white list to determine the data change KEY in the KEY of the source ETCD cluster; then determines whether there is a KEY to be synchronized that matches the KEY black and white list in the data change KEY; and then if there is a KEY to be synchronized, obtains the change data corresponding to the KEY to be synchronized, and synchronizes the change data to the target ETCD cluster to achieve incremental data synchronization across clusters. By synchronizing the changed data, the time and resource consumption of data synchronization are greatly reduced.

[0112] In a feasible implementation manner, the data processing method may further include steps S301 to S302:

[0113] Step S301, when receiving the ETCD certificate file corresponding to the cluster adding operation, performing a validity check operation on the ETCD certificate file;

[0114] Step S302: If the ETCD certificate file is a valid certificate, a format conversion operation is performed on the ETCD certificate file, and the ETCD certificate file after the format conversion is stored.

[0115] In this embodiment, the management platform has the function of converting the certificate format. The certificate formats generated by the ETCD cluster in different deployment modes may be different, such as PKCS1, PKCS8, PEM format, etc. The differences between these formats may cause cross-cluster communication failures. This embodiment automatically converts the certificate format and converts certificates of various formats into PKCS8 format, making communication between different clusters smoother and safer.

[0116] The management platform can also perform detailed attribute verification and legality checks on certificates. The management platform automatically identifies the validity period, issuing authority, and encryption strength of the certificate, and issues warnings when potential risks are found. This mechanism effectively avoids communication interruptions caused by certificate expiration or incompatibility, providing strong protection for enterprise-level applications.

[0117] Specifically, when the administrator adds a cluster, the certificate file of the ETCD cluster is uploaded. When the ETCD certificate file corresponding to the cluster addition operation is received, the validity check operation is performed on the ETCD certificate file; specifically, the Cert module will verify the content of the ETCD certificate file to determine whether it is a valid certificate. If the ETCD certificate file is not a valid certificate, the user will be prompted and required to re-upload the ETCD certificate file in the correct format.

[0118] If the ETCD certificate file is a valid certificate, the format conversion operation is performed on the ETCD certificate file, and the ETCD certificate file after the format conversion is stored, that is, the certificate format of the ETCD certificate file is converted into the PKCS8 format, and the ETCD certificate file converted into the PKCS8 format is encrypted using MD5 and stored in the database.

[0119] In a feasible implementation manner, the data processing method may further include steps S303 to S307:

[0120] Step S303, displaying each ETCD certificate file and the corresponding expiration countdown of each ETCD certificate file based on the ETCD cluster list;

[0121] Step S304: If there is a target expiration countdown that is about to expire among the expiration countdowns, then obtain the ETCD certificate file to be updated corresponding to the target expiration countdown;

[0122] Step S305: If there is automatic update information in the configuration information corresponding to the ETCD certificate file to be updated, a new ETCD certificate file corresponding to the ETCD certificate file to be updated is generated;

[0123] Step S306, updating the ETCD certificate file to be updated in the database based on the new ETCD certificate file, and updating the ETCD cluster list;

[0124] Step S307: If there is no automatic update information in the configuration information corresponding to the ETCD certificate file to be updated, an expiration alarm message is sent to the user terminal corresponding to the ETCD certificate file to be updated.

[0125] In this embodiment, the management platform also includes a certificate management module, which is used to manage the certificate lifecycle of the entire cluster, including the issuance, renewal, revocation and update of certificates. Through the built-in certificate management tool, administrators can easily manage the certificates of each ETCD cluster to avoid communication failures due to expired certificates. The system can automatically monitor the validity period of the certificate and send reminder notifications through the UI interface and email when the certificate is about to expire.

[0126] The certificate management module supports automatic renewal. When a certificate is about to expire, the system will automatically generate a new certificate and update it according to preset conditions to ensure that cluster communication is not interrupted, further improving the security of the cluster.

[0127] Specifically, each ETCD certificate file and the expiration countdown of each ETCD certificate file are obtained, and each ETCD certificate file and the corresponding expiration countdown of each ETCD certificate file are displayed through the ETCD cluster list, wherein the countdown display is updated every day, that is, the expiration countdown of each ETCD certificate file in the ETCD cluster list is updated once a day.

[0128] Next, the validity period of the certificate is scanned and checked according to the configured time schedule to determine whether there is a target expiration countdown that is about to expire among each expiration countdown. If there is a target expiration countdown that is about to expire among each expiration countdown, the ETCD certificate file to be updated corresponding to the target expiration countdown is obtained; wherein, when the expiration countdown of the ETCD certificate file is less than the corresponding preset expiration time, the expiration countdown of the ETCD certificate file is determined to be the target expiration countdown.

[0129] When it is determined that there is an ETCD certificate file to be updated, the configuration information of the ETCD certificate file to be updated is obtained. If there is automatic update information in the configuration information, a new ETCD certificate file corresponding to the ETCD certificate file to be updated is generated; the ETCD certificate file to be updated is updated in the database based on the new ETCD certificate file, and the ETCD cluster list is updated, that is, the new ETCD certificate file corresponding to the ETCD certificate file to be updated is automatically generated, the ETCD certificate file to be updated in the database is replaced with the new ETCD certificate file, the ETCD certificate file to be updated in the ETCD cluster list is replaced with the new ETCD certificate file, the expiration countdown of the ETCD certificate file to be updated is replaced with the expiration countdown of the new ETCD certificate file, and the download button corresponding to the download address is displayed in the ETCD cluster list for the user to download the new ETCD certificate file.

[0130] It should be noted that if there is no automatic update information in the configuration information, an alarm prompt message is sent to remind the user to pay attention to the validity period of the certificate. If the ETCD certificate file to be updated fails to be updated, a certificate update prompt message is sent to remind the user.

[0131] The data synchronization method proposed in this embodiment performs a validity check operation on the ETCD certificate file when the ETCD certificate file corresponding to the cluster adding operation is received; then, if the ETCD certificate file is a valid certificate, a format conversion operation is performed on the ETCD certificate file, and the ETCD certificate file after the format conversion is stored. By uniformly converting ETCD certificate files of various different formats into ETCD certificate files of the same format, communication between different clusters becomes smoother and more secure.

[0132] The data synchronization device provided in the embodiment of the present application adopts the data synchronization method in the above embodiment, which can solve the technical problem of how to realize customized data synchronization in a cross-level group scenario. Compared with the prior art, the beneficial effects of the data synchronization device provided in the embodiment of the present application are the same as the beneficial effects of the data synchronization method provided in the above embodiment, and the other technical features in the data synchronization device are the same as the features disclosed in the above embodiment method, which will not be repeated here.

[0133] The present application provides a data synchronization device, which includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the data synchronization method in the above-mentioned embodiment one.

[0134] Reference below Figure 2 , which shows a schematic diagram of the structure of a data synchronization device suitable for implementing the embodiment of the present application. The data synchronization device in the embodiment of the present application may include but is not limited to mobile terminals such as mobile phones, laptop computers, digital broadcast receivers, PDAs (Personal Digital Assistants), PADs (Portable Application Descriptions), PMPs (Portable Media Players), vehicle-mounted terminals (such as vehicle-mounted navigation terminals), etc., and fixed terminals such as digital TVs, desktop computers, etc. Figure 2 The data synchronization device shown is merely an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.

[0135] like Figure 2As shown, the data synchronization device may include a processing device 1001 (e.g., a central processing unit, a graphics processor, etc.), which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM: Read Only Memory) 1002 or a program loaded from a storage device 1003 to a random access memory (RAM: Random Access Memory) 1004. In RAM1004, various programs and data required for the operation of the data synchronization device are also stored. The processing device 1001, ROM1002, and RAM1004 are connected to each other through a bus 1005. An input / output (I / O) interface 1006 is also connected to the bus. Generally, the following systems can be connected to the I / O interface 1006: an input device 1007 including, for example, a touch screen, a touch pad, a keyboard, a mouse, an image sensor, a microphone, an accelerometer, a gyroscope, etc.; an output device 1008 including, for example, a liquid crystal display (LCD: Liquid Crystal Display), a speaker, a vibrator, etc.; a storage device 1003 including, for example, a magnetic tape, a hard disk, etc.; and a communication device 1009. The communication device 1009 can allow the data synchronization device to communicate with other devices wirelessly or by wire to exchange data. Although the data synchronization device with various systems is shown in the figure, it should be understood that it is not required to implement or have all the systems shown. More or fewer systems can be implemented or have alternatively.

[0136] In particular, according to the embodiments disclosed in the present application, the process described above with reference to the flowchart can be implemented as a computer software program. For example, the embodiments disclosed in the present application include a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network through a communication device, or installed from a storage device 1003, or installed from a ROM 1002. When the computer program is executed by the processing device 1001, the above-mentioned functions defined in the method of the embodiment disclosed in the present application are executed.

[0137] The data synchronization device provided by the present application adopts the data synchronization method in the above embodiment, which can solve the technical problem of how to realize customized data synchronization in a cross-level group scenario. Compared with the prior art, the beneficial effects of the data synchronization device provided by the present application are the same as the beneficial effects of the data synchronization method provided by the above embodiment, and the other technical features in the data synchronization device are the same as the features disclosed in the method of the previous embodiment, which will not be repeated here.

[0138] It should be understood that the various parts disclosed in this application can be implemented by hardware, software, firmware or a combination thereof. In the description of the above embodiments, specific features, structures, materials or characteristics can be combined in any one or more embodiments or examples in a suitable manner.

[0139] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art who is familiar with the present technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

[0140] The present application provides a computer-readable storage medium having computer-readable program instructions (ie, computer programs) stored thereon, wherein the computer-readable program instructions are used to execute the data synchronization method in the above-mentioned embodiment.

[0141] The computer-readable storage medium provided in the present application may be, for example, a USB flash drive, but is not limited to electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, systems or devices, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In this embodiment, the computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system, system or device. The program code contained on the computer-readable storage medium may be transmitted using any appropriate medium, including but not limited to: wires, optical cables, RF (Radio Frequency), etc., or any suitable combination of the above.

[0142] The computer-readable storage medium may be included in the data synchronization device; or may exist independently without being assembled into the data synchronization device.

[0143] The above-mentioned computer-readable storage medium carries one or more programs. When the above-mentioned one or more programs are executed by the data synchronization device, the data synchronization device: generates a multi-layer index layer based on the data in the KEY black and white lists corresponding to the source ETCD cluster and the target ETCD cluster, generates an index table corresponding to the index layer according to preset rules, and determines the index to be matched based on the index layer and the index table; determines the index layer to be matched in the multi-layer index layer based on the data to be matched of the source ETCD cluster and the index to be matched; performs data screening operations on the data to be matched based on the data in the index layer to be matched to obtain first matching data; determines the target data based on the KEY white list and the first matching data, and synchronizes the target data to the target ETCD cluster.

[0144] Computer program code for performing the operations of the present application may be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, C++, and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0145] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present application. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0146] The modules involved in the embodiments described in this application may be implemented by software or hardware, wherein the name of the module does not constitute a limitation on the unit itself in some cases.

[0147] The readable storage medium provided in this application is a computer-readable storage medium, which stores computer-readable program instructions (i.e., computer programs) for executing the above-mentioned data synchronization method, and can solve the technical problem of how to achieve customized data synchronization in a cross-level group scenario. Compared with the prior art, the beneficial effects of the computer-readable storage medium provided in this application are the same as the beneficial effects of the data synchronization method provided in the above-mentioned embodiment, and will not be repeated here.

[0148] An embodiment of the present application provides a computer program product, including a computer program, which implements the steps of the above-mentioned data synchronization method when executed by a processor.

[0149] The computer program product provided in this application can solve the technical problem of how to achieve customized data synchronization in a cross-level group scenario. Compared with the prior art, the beneficial effects of the computer program product provided in the embodiment of this application are the same as the beneficial effects of the data synchronization method provided in the above embodiment, which will not be repeated here.

[0150] The above are only preferred embodiments of the present application, and are not intended to limit the patent scope of the present application. Any equivalent structure or equivalent process transformation made using the contents of the present application specification and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present application.

Claims

1. A data synchronization method, characterized in that: The data synchronization method comprises the following steps: Generate a multi-layer index layer based on the data in the KEY black and white lists corresponding to the source ETCD cluster and the target ETCD cluster, generate an index table corresponding to the index layer according to preset rules, and determine the index to be matched based on the index layer and the index table; Based on the to-be-matched data of the source ETCD cluster and the to-be-matched index, determine the to-be-matched index layer in the multi-layer index layer; Based on the data in the to-be-matched index layer, a data screening operation is performed on the to-be-matched data to obtain first matching data; The target data is determined based on the KEY whitelist and the first matching data, and the target data is synchronized to the target ETCD cluster.

2. The data synchronization method according to claim 1, characterized in that: The step of generating a multi-layer index layer based on the data in the KEY black and white lists corresponding to the source ETCD cluster and the target ETCD cluster, generating an index table corresponding to the index layer according to a preset rule, and determining the index to be matched based on the index layer and the index table includes: Performing character combination on the data in the KEY blacklist and whitelist to obtain combined data, and generating a multi-layer index layer based on the character length of each data in the combined data; Based on the preset rules and each data in the multi-layer index layer, generate an index table corresponding to the multi-layer index layer; The index data corresponding to each index in the index table are compared with each data in the multi-layer index layer to obtain the number of matches corresponding to each index data, and the index to be matched is determined based on the number of matches.

3. The data synchronization method according to claim 1, characterized in that: The step of determining the index layer to be matched in the multi-layer index layer based on the data to be matched and the index to be matched of the source ETCD cluster includes: Obtain the to-be-matched KEY corresponding to the to-be-matched data; Matching the KEY to be matched with the data corresponding to each of the indexes to be matched, so as to determine the target index corresponding to the KEY to be matched in each of the indexes to be matched; The index layer corresponding to the target index in the multiple index layers is used as the index layer to be matched.

4. The data synchronization method according to claim 1, characterized in that: The step of performing a data screening operation on the data to be matched based on the data in the to-be-matched index layer to obtain the first matching data comprises: Match and verify the data in the to-be-matched index layer with each piece of data in the to-be-matched data, respectively, to obtain second matching data corresponding to the data in the to-be-matched index layer; The data other than the second matching data in the to-be-matched data is used as the first matching data.

5. The data synchronization method according to claim 1, characterized in that: The step of determining target data based on the KEY whitelist and the first matching data, and synchronizing the target data to the target ETCD cluster includes: Through a preset optimization algorithm, the data in the first matching data are verified one by one to determine whether each data in the first matching data belongs to a category in the KEY black and white list, and the data in the first matching data that belongs to the KEY white list is used as the target data, and the target data is synchronized to the target ETCD cluster.

6. The data synchronization method according to claim 1, characterized in that: After the step of synchronizing the target data to the target ETCD cluster, the data synchronization method further includes: Based on the KEY black and white lists, a continuous monitoring operation is performed on the source ETCD cluster to determine the data change KEY in the KEY of the source ETCD cluster; Determine whether there is a KEY to be synchronized matching the KEY blacklist and whitelist in the data change KEY; If the KEY to be synchronized exists, the change data corresponding to the KEY to be synchronized is obtained, and the change data is synchronized to the target ETCD cluster.

7. The data synchronization method according to claim 6, characterized in that: After the step of obtaining the changed data corresponding to the KEY to be synchronized and synchronizing the changed data to the target ETCD cluster, the data synchronization method further includes: In the process of synchronizing the changed data to the target ETCD cluster, monitor the synchronization operation in real time to see if there is any abnormality; If there is an exception in the synchronization operation, restart the data synchronization service and perform the data synchronization operation based on the source ETCD cluster after the most recent full synchronization.

8. The data synchronization method according to any one of claims 1 to 7, characterized in that: The data synchronization method further includes: When receiving the ETCD certificate file corresponding to the cluster adding operation, perform a validity check operation on the ETCD certificate file; If the ETCD certificate file is a valid certificate, a format conversion operation is performed on the ETCD certificate file, and the ETCD certificate file after the format conversion is stored.

9. The data synchronization method according to claim 8, characterized in that: The data synchronization method further includes: Based on the ETCD cluster list, display each ETCD certificate file and the corresponding expiration countdown of each ETCD certificate file; If there is a target expiration countdown that is about to expire in each of the expiration countdowns, then obtain the ETCD certificate file to be updated corresponding to the target expiration countdown; If there is automatic update information in the configuration information corresponding to the ETCD certificate file to be updated, a new ETCD certificate file corresponding to the ETCD certificate file to be updated is generated; In the database, the ETCD certificate file to be updated is updated based on the new ETCD certificate file, and the ETCD cluster list is updated; If there is no automatic update information in the configuration information corresponding to the ETCD certificate file to be updated, an expiration alarm message is sent to the user terminal corresponding to the ETCD certificate file to be updated.

10. A data synchronization device, characterized in that: The data synchronization device includes: a memory, a processor, and a data synchronization program stored in the memory and executable on the processor, and when the data synchronization program is executed by the processor, the steps of the data synchronization method according to any one of claims 1 to 9 are implemented.

Images