Data synchronization and partitioning methods and federated clusters

Abstract

This invention provides a data synchronization and partitioning method and a federated cluster, relating to the field of big data technology. After the target data is synchronized, the first cluster provides verification information to the second cluster through synchronization logs, enabling the second cluster to avoid problems such as missing files, inconsistent file sizes, or incomplete data write-to-disk through consistency checks. After the partition maintenance operation is completed, the second cluster also verifies whether the target data can be read normally, ensuring normal data availability. Furthermore, synchronization logs and partition logs are used for state management during the execution of tasks to be executed, ensuring the traceability of the execution process.

Description

Technical Field

[0001] This invention relates to the field of big data technology, and more specifically, to a data synchronization and partitioning method and a federated cluster. Background Technology

[0002] With the deepening of 5G, IoT, and digital transformation across various industries, telecom operators, large internet companies, and the financial sector have generally built multiple heterogeneous big data clusters. These include offline data warehouse clusters based on the Hadoop ecosystem, real-time computing clusters based on Flink / Kafka, integrated lake-warehouse clusters based on Spark + Delta Lake, and GPU (Graphics Processing Unit) accelerated clusters for AI (Artificial Intelligence) training. Under this multi-cluster architecture, data needs to flow frequently between different computing paradigms and storage engines. Typical scenarios include: synchronizing raw logs from edge collection clusters to the central offline cluster for ETL (Extract-Transform-Load) processing; synchronizing cleaned wide tables to the real-time cluster to support risk control model inference; or synchronizing feature samples to the AI ​​training cluster to support model iteration. Cross-cluster data synchronization and data warehouse partition management have become fundamental technical aspects of data production systems.

[0003] In existing technologies, data synchronization between the source cluster and the target cluster is usually achieved through data replication tools of distributed file systems (such as Hadoop-based data replication mechanisms), and after synchronization is completed, the corresponding partition loading or metadata maintenance operations are performed through the data warehouse system.

[0004] However, existing technologies cannot detect issues such as missing files, inconsistent file sizes, or incomplete data writes on the target end after data synchronization. Furthermore, synchronization and partitioning operations are often triggered by independent scheduling or task configurations, resulting in a low correlation between execution status and the specific execution process. This makes it difficult to effectively track, manage the status of, and locate anomalies in the execution process of a single task. Summary of the Invention

[0005] The purpose of this invention is to provide a data synchronization and partitioning method and a federated cluster to improve the problems existing in the prior art.

[0006] The embodiments of the present invention can be implemented as follows: In a first aspect, the present invention provides a data synchronization and partitioning method, comprising: When the first cluster determines that the preset timed synchronization task meets the scheduling rules, it retrieves the task to be executed from the task list in the task library; the task library includes the synchronization log and partition log associated with the task to be executed; the scheduling rules reflect the triggering conditions and billing cycle synchronization rules of the task to be executed. After the first cluster obtains the target data based on the requirements of the timed synchronization task and synchronizes the target data to the second cluster, it writes verification information into the synchronization log. When the second cluster receives the target data, it stores the target data based on the preset requirements of the timed partitioning task, and then obtains the verification information from the synchronization log to perform consistency verification on the target data. When the second cluster successfully completes the consistency check, it writes a synchronization success flag into the synchronization log. The second cluster performs partition maintenance operations based on the requirements of the scheduled partitioning task and the storage path of the target data to determine the target partition of the target data; The second cluster reads the storage path of the target partition. If the read is successful, a partition success flag is written to the partition log, and the task to be executed is updated to the executed status.

[0007] In an optional implementation, the federated cluster includes at least two clusters, the first cluster and the second cluster being part of the federated cluster, the federated cluster including a scheduling node; each cluster supports access to the task library; Before the step of retrieving task information of tasks to be executed from the task list in the task library when the first cluster triggers the synchronization condition, the method further includes: The scheduling node obtains the task plan table; the task plan table includes multiple plan table entries, which reflect the data tables, source cluster, target cluster, data range, and scheduling rules involved in a cross-cluster data synchronization and partitioning task; The scheduling node creates a scheduled task for each of the schedule entries, configures the requirements for the scheduled task, and sends the task schedule to each cluster. When each cluster determines that it is the source cluster of any of the aforementioned schedule entries, it performs synchronization task configuration based on the schedule entries; and when it determines that it is the target cluster of any of the aforementioned schedule entries, it performs partitioning task configuration based on the schedule entries. When the scheduling node satisfies any of the scheduling rules at the current time, it adds a task to be executed to the task list for each timed task corresponding to the scheduling rule; The scheduling node sets the newly added tasks to the pending execution state and initializes the synchronization logs and partition logs associated with the newly added tasks.

[0008] In an optional implementation, the plan table entries include data table ID, data table name, database name, rule ID, source path, target path, data range, source cluster ID, target cluster ID, synchronization node ID, and partition node ID; The steps of the scheduling node creating a scheduled task for each of the schedule entries and configuring the requirements information of the scheduled task include: For each of the plan entries, the scheduling node determines the triggering conditions and billing period synchronization rules associated with the rule ID in the plan entry from a preset rule mapping table, thereby obtaining the triggering conditions and billing period synchronization rules of the plan entry. The scheduling node creates a scheduled task based on the triggering conditions of the schedule entry; The scheduling node uses the data table name, database name, source cluster ID, target cluster ID, and billing cycle synchronization rule of the plan table item as the requirement information for the scheduled task.

[0009] In an optional implementation, the plan table entries include data table ID, data table name, database name, rule ID, source path, target path, data range, source cluster ID, target cluster ID, synchronization node ID, and partition node ID; The steps of configuring synchronization tasks based on the schedule entry when each cluster determines itself to be the source cluster of any of the schedule entries, and configuring partitioning tasks based on the schedule entry when it determines itself to be the target cluster of any of the schedule entries, include: For each of the plan entries, the cluster obtains the IP address of the source cluster, the IP address of the target cluster, the IP address of the synchronization node, and the IP address of the partition node from a preset object mapping table based on the plan entry. The cluster determines the triggering conditions and billing cycle synchronization rules associated with the rule IDs in the plan table entries based on a preset rule mapping table. When the cluster's external IP address is the same as the IP address of the source cluster, it notifies the corresponding synchronization node to create a scheduled synchronization task, and uses the triggering condition, the billing period synchronization rule, the IP address of the target cluster, the IP address of the partition node, the data table ID, data table name, database name, source path, and data range in the schedule table as the required information for the scheduled synchronization task. When the cluster's external IP address is the same as the target cluster's IP address, it notifies the corresponding partition node to create a scheduled partition task and saves the triggering condition, the billing cycle synchronization rule, the source cluster's IP address, the synchronization node's IP address, the data table ID, data table name, database name, target path, and data range in the schedule table as the requirements information for the scheduled partition task.

[0010] In an optional implementation, the first cluster and the second cluster respectively include a first synchronization node and a second partition node; The step of the first cluster obtaining target data based on the requirements of the scheduled synchronization task and synchronizing the target data to the second cluster, and then writing verification information into the synchronization log, includes: Based on the requirements of the scheduled synchronization task, the first synchronization node determines the billing period to be synchronized, the specified data table, the specified data range, the specified source path, the IP address of the second cluster, and the IP address of the second partition node. The first synchronization node reads the target data from the specified source path based on the specified data table, the specified data range, and the billing period to be synchronized; The first synchronization node encapsulates the target data into a synchronization message and sends it to the second partition node based on the task ID of the task to be executed, the IP address of the second cluster, and the IP address of the second partition node; After the synchronization message is sent, the first synchronization node writes the number of files containing the target data and the file size into the synchronization log.

[0011] In an optional implementation, the first cluster and the second cluster respectively include a first synchronization node and a second partition node; When the second cluster receives the target data, it stores the target data based on the preset requirements of the timed partitioning task, and then retrieves the verification information from the synchronization log to perform consistency verification on the target data. The steps include: When the second partition node receives the synchronization message sent by the first synchronization node, it parses the synchronization message to obtain the target data and the task ID of the task to be executed; The second partition node determines the account period to be synchronized, the specified data table, the specified data range, and the specified target path based on the requirements information of the scheduled partitioning task. The second partition node stores the target data into the specified target path; The second partition node reads the number of files and file size of the target data from the synchronization log associated with the task ID of the task to be executed; The second partition node verifies whether the number and size of the read files are consistent with the number and size of the target data under the specified target path; if yes, the consistency verification is confirmed to be successful; otherwise, the consistency verification is confirmed to be unsuccessful.

[0012] In an optional implementation, the second cluster performs a partition maintenance operation based on the requirements of the scheduled partitioning task and the storage path of the target data to determine the target partition of the target data, including: The second partition node generates at least one partition command corresponding to the task to be executed based on the billing period to be synchronized, the specified data table, the specified data range, and the specified target path; The second partition node uses a batch submission method to execute at least one partition command corresponding to the task to be executed, so as to add the target partition corresponding to the target data in the metadata of the specified data table.

[0013] In an optional implementation, the method further includes: When the consistency check fails, the second partition node clears the target data under the specified target path and writes a synchronization failure flag and failure information into the synchronization log so that the first synchronization node can resynchronize the data.

[0014] In an optional implementation, the step of the second cluster reading the storage path of the target partition includes: The second partition node reads the storage path of the target partition; The method further includes: If the read fails, the second partition node writes a partition failure flag and failure information into the partition log.

[0015] Secondly, the present invention provides a federated cluster, the federated cluster comprising a first cluster and a second cluster; The first cluster and the second cluster are used together to implement the data synchronization and partitioning method described in the first aspect above.

[0016] Compared with existing technologies, this invention provides a data synchronization and partitioning method and a federated cluster. First, when a first cluster determines that a preset scheduled synchronization task meets the scheduling rules, it retrieves the task to be executed from the task list in the task library. The task library includes synchronization logs and partition logs associated with the task to be executed. The scheduling rules reflect the triggering conditions and billing cycle synchronization patterns of the task to be executed. Then, the first cluster retrieves the target data based on the requirements of the scheduled synchronization task and synchronizes the target data to the second cluster, writing verification information into the synchronization log. When the second cluster receives the target data, it stores the target data based on the requirements of the preset scheduled partitioning task, retrieves verification information from the synchronization log to perform consistency verification on the target data, and writes a synchronization success flag into the synchronization log when the consistency verification is successful. Next, the second cluster performs partition maintenance operations based on the requirements of the scheduled partitioning task and the storage path of the target data to determine the target partition of the target data. Finally, the second cluster reads the storage path of the target partition. If it can be read normally, it writes a partition success flag into the partition log and updates the task to be executed to the executed state.

[0017] After the target data is synchronized, the first cluster provides verification information to the second cluster through the synchronization log, enabling the second cluster to avoid problems such as missing files, inconsistent file sizes, or incomplete data write-to-disk through consistency verification. After the partition maintenance operation is completed, the second cluster will also verify whether the target data can be read normally to ensure the normal availability of the data. Furthermore, the synchronization log and partition log are used for state management during the execution of tasks to be executed, ensuring the traceability of the execution process. Attached Figure Description

[0018] To more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings used in the embodiments will be briefly introduced below. It should be understood that the following drawings only show some embodiments of the present invention and should not be regarded as a limitation on the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0019] Figure 1 This is a schematic diagram of an application scenario provided by an embodiment of the present invention.

[0020] Figure 2 This is one of the flowcharts illustrating a data synchronization and partitioning method provided in an embodiment of the present invention.

[0021] Figure 3 This is a second schematic flowchart of a data synchronization and partitioning method provided in an embodiment of the present invention.

[0022] Figure 4 This is an example diagram of a synchronization log provided in an embodiment of the present invention.

[0023] Figure 5 This is an example diagram of a partition log provided in an embodiment of the present invention. Detailed Implementation

[0024] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. The components of the embodiments of the present invention described and shown in the accompanying drawings can generally be arranged and designed in various different configurations.

[0025] Therefore, the following detailed description of the embodiments of the invention provided in the accompanying drawings is not intended to limit the scope of the claimed invention, but merely to illustrate selected embodiments of the invention. All other embodiments obtained by those skilled in the art based on the embodiments of the invention without inventive effort are within the scope of protection of the invention.

[0026] It should be noted that similar labels and letters in the following figures indicate similar items. Therefore, once an item is defined in one figure, it does not need to be further defined and explained in subsequent figures.

[0027] Furthermore, the terms "first" and "second" are used only to distinguish descriptions and should not be interpreted as indicating or implying relative importance.

[0028] It should be noted that, where there is no conflict, the features in the embodiments of the present invention can be combined with each other.

[0029] Here, we will first introduce the keywords or key terms involved in this invention: 1. Data Partitioning: A physical database design technique aimed at reducing the total amount of data read and write operations during specific SQL operations, thereby reducing response time. Partitioning does not create new data tables, but rather distributes the table's data evenly across different hard drives, system disks, or different server storage media; it is essentially still a single table.

[0030] 2. Hive: In the field of big data, Hive is a commonly used data warehouse tool for managing and querying large-scale datasets.

[0031] 3. Hive Metadata: In Hive, metadata describes the attribute data of a data table, including its structure, partitioning information, and data storage path. Metadata is managed by Hive's Metastore, which tracks and manages the metadata of databases and tables. Through metadata, users can understand important information such as how data is organized, its storage location, and the relationships between tables.

[0032] In existing technologies, cross-cluster data synchronization is generally implemented using the following methods: By pre-configuring the synchronization path, a data replication tool is invoked after the task is triggered to copy data from a specified directory in the source cluster to the corresponding directory in the target cluster. Successful synchronization is typically determined by the return result of the replication command, with some implementations supplementing this with a simple check for the existence of a completion marker file (such as a success flag file). This approach is widely used in engineering practice, but it primarily focuses on whether the synchronization operation itself has been completed, without rigorously verifying the data consistency between the source and target. In complex network environments or large-scale file synchronization scenarios, even if the replication command returns successfully, issues such as missing files, inconsistent file sizes, or incomplete data writes may still occur, leading to inaccurate subsequent data processing results.

[0033] In addition, existing synchronization tasks are mostly driven by task tables or schedule tables, and the correlation between execution status and specific execution actions is weak. It is difficult to perform fine-grained tracking and auditing of a single synchronization process, and it is also not conducive to automated error correction and retry control in failure scenarios.

[0034] After data synchronization is complete, existing technologies typically perform partition maintenance operations through the data warehouse system, such as adding, deleting, or updating partition storage locations. A common implementation involves submitting one or more partition operation statements for each partition task and determining whether the partition creation was successful based on the execution return status. However, in actual production environments, the above partition execution method has the following problems: (1) The granularity of partitioning operations is fine and the execution efficiency is low: When a large number of partitions need to be processed at the same time, submitting partition operations one by one will generate a lot of session overhead and the overall execution efficiency is low. (2) Unreliable determination of successful partition execution: Existing solutions mostly use the success of statement execution as the sole criterion for judgment, lacking a verification mechanism for whether the data corresponding to the partition is real and usable, which can easily lead to situations where the partition metadata exists but the underlying data is empty or incomplete; (3) Execution failure due to uncertain partition predicate type: In scenarios where the partition field type is inconsistent or the partition value is dynamically generated, the existing solution usually relies on manual guarantee of the syntax correctness of the partition conditions, which can easily lead to partition query or verification failure due to mismatch of partition value type.

[0035] Based on the discovery of the aforementioned technical problems, the inventors, through creative labor, proposed the following technical solutions to solve or improve these problems. It should be noted that the deficiencies in the solutions of the prior art are all results derived by the inventors after practical experience and careful research. Therefore, the discovery process of the aforementioned problems and the solutions proposed in the embodiments of this application below should be considered contributions made by the inventors to this application during the inventive process, and should not be construed as technical content known to those skilled in the art.

[0036] Please see Figure 1 , Figure 1 This is a schematic diagram illustrating an application scenario provided by an embodiment of the present invention. For example... Figure 1 In a federated cluster, there are multiple clusters, and each cluster contains multiple nodes. A node can be, but is not limited to, a physical server (compute server or storage server), a personal laptop, a personal computer, or a virtual server.

[0037] In this invention, a scheduling master program is deployed on a node in the federated cluster, making that node a scheduling node. This scheduling node possesses a corresponding task library, and each cluster supports access to this task library. The scheduling node is used to flexibly and accurately schedule cross-cluster data synchronization and partitioning within the federated cluster based on the task library.

[0038] Therefore, when cross-cluster data synchronization and partitioning are required between two clusters, the scheduling node publishes the task in the task library. The synchronization node in the two clusters actively receives the task to synchronize the data, and the node being synchronized then performs the partition maintenance operation.

[0039] It should be noted that the first cluster and the second cluster below can be any two clusters in the federated cluster.

[0040] Please refer to Figure 2 , Figure 2 This is a flowchart illustrating a data synchronization and partitioning method provided in an embodiment of the present invention, as shown below. Figure 1 The method includes the following steps S201 to S206.

[0041] S201. When the first cluster determines that the preset timed synchronization task meets the scheduling rules, it retrieves the task to be executed from the task list in the task library.

[0042] The scheduling rules reflect the triggering conditions and billing cycle synchronization patterns of the tasks to be executed. Optionally, the triggering conditions can be timed triggering, file readiness triggering, etc., while the billing cycle synchronization patterns are used to define the data time dimension covered by this synchronization, such as daily or monthly synchronization.

[0043] The task library also includes synchronous logs and partition logs associated with the tasks to be executed. Both synchronous logs and partition logs support dynamic updates, which logically binds the log records to the tasks to be executed, avoiding the difficulty of tracing due to scattered states.

[0044] S202. After the first cluster obtains the target data based on the requirements of the timed synchronization task and synchronizes the target data to the second cluster, it writes verification information into the synchronization log.

[0045] In this embodiment, when the first cluster completes the synchronization operation, it does not directly and arbitrarily conclude that the synchronization is successful. Instead, it writes the number of files and the size of the target data on the source end (i.e., the first cluster) as verification information into the synchronization log associated with the task to be executed in the task library, so as to provide comparable benchmark data for subsequent consistency verification.

[0046] S203. When the second cluster receives the target data, it stores the target data based on the preset timed partition task requirements and then obtains the verification information from the synchronization log to perform consistency verification on the target data.

[0047] In this embodiment, the second cluster determines the designated target path based on the requirements of the timed partitioning task, and writes the received target data to the designated target path. Subsequently, the second cluster actively accesses the task library and reads the verification information written by the first cluster in the synchronization log associated with the same task to be executed, so as to determine the integrity and accuracy of the synchronization result through consistency verification.

[0048] S204. When the second cluster successfully verifies consistency, it writes a synchronization success flag into the synchronization log.

[0049] In this embodiment, the second cluster writes a synchronization success flag in the synchronization log, indicating that the first cluster has completed the synchronization operation and that the target data received by the second cluster is accurate and complete.

[0050] S205. The second cluster performs partition maintenance operations based on the requirements of the scheduled partitioning task and the storage path of the target data to determine the target partition of the target data.

[0051] In this embodiment, when performing partition maintenance operations, the second cluster generates partition commands adapted to the data warehouse syntax and executes them to map the physical storage path of the target data to the partition location at the metadata level, thereby creating a valid partition entry in the data table structure that points to the physical storage path.

[0052] S206. The second cluster reads the storage path of the target partition. If the reading is successful, a partition success flag is written to the partition log, and the task to be executed is updated to the executed status.

[0053] In this embodiment, the read operation here is a real verification of the partition's validity. That is, by actually initiating a lightweight data access request, it is confirmed that there is a parseable, compliant, and complete data file under the target partition path. Only when the access successfully returns a valid data stream is it considered that the partition has not only been successfully created at the metadata level, but also achieved the expected data availability level. At this time, the partition log records the partition success flag, and the status field of the task to be executed in the task library is updated to executed, indicating that a cross-cluster data synchronization and partitioning has been completed.

[0054] The data synchronization and partitioning method provided in this embodiment of the invention, after the target data is synchronized, the first cluster provides verification information to the second cluster through the synchronization log, so that the second cluster can avoid problems such as missing files, inconsistent file sizes, or incomplete data write-to-disk through consistency verification. After the partition maintenance operation is completed, the second cluster will also verify whether the target data can be read normally to ensure the normal availability of the data. Furthermore, the synchronization log and partition log are used to perform state management during the execution of the task to be executed, ensuring the traceability of the execution process.

[0055] The following section will first introduce the process of task planning and scheduling by the scheduling node based on the user's task configuration.

[0056] The scheduling node primarily achieves orderly scheduling of cross-cluster data flow by parsing task schedule tables, dispatching timed tasks, and uniformly allocating tasks to be executed. See also: Figure 3 The method may further include the following steps S101 to S105.

[0057] S101, The scheduling node obtains the task plan table.

[0058] In this embodiment, the task schedule table is configured by the engineer according to the actual business data migration requirement 2. It includes multiple schedule table items, which can reflect the data tables, source cluster, target cluster, data range and scheduling rules involved in a cross-cluster data synchronization and partitioning task.

[0059] Optionally, each item in the task schedule table may include, but is not limited to, fields such as: table ID, table name, database name, rule ID, source path, target path, data range, source cluster ID, target cluster ID, synchronization node ID, and partition node ID.

[0060] The data scope can reflect the business distribution characteristics based on the data. For example, assuming the data to be synchronized in this invention is obtained by analyzing signaling data from a large number of users, such as user residency data, travel trajectory data, and passenger flow data for high-speed rail / airplane / subway travel, then the data scope can be a regional range, such as specifying to synchronize residency data nationwide, or specifying to synchronize mainstream data from a specific province / city / administrative region. It should be noted that this example is merely illustrative and is not intended to be limiting.

[0061] S102, the scheduling node creates a scheduled task for each plan entry, configures the requirements for the scheduled task, and sends the task plan to each cluster.

[0062] In this embodiment, the scheduling node sends the task schedule to each cluster so that the corresponding nodes in each cluster are aware that they need to perform synchronization or partitioning operations periodically. Furthermore, the scheduling node maps each schedule entry to an independent, triggerable scheduled task and configures its requirements.

[0063] Optionally, the implementation process of step S102 may include: S1021. For each plan item, the scheduling node determines the triggering conditions and billing period synchronization rules associated with the rule ID in the plan item from the preset rule mapping table, thereby obtaining the triggering conditions and billing period synchronization rules of the plan item.

[0064] In this embodiment, the rule mapping table includes multiple scheduling rules uniquely identified by rule IDs. Each scheduling rule is used to determine the triggering conditions and the billing cycle synchronization pattern.

[0065] S1022. The scheduling node creates a scheduled task based on the triggering conditions of the schedule entry.

[0066] S1023. The scheduling node uses the data table name, database name, source cluster ID, target cluster ID, and billing cycle synchronization rules of the plan table entries as the requirement information for the scheduled task.

[0067] In this embodiment, the scheduling node stores the requirement information for each scheduled task, mainly so that the task can be released on time when each scheduled task is triggered.

[0068] For example, taking timed triggering as an example, assume that the task schedule table and rule mapping table are shown in Table 1 and Table 2 below, respectively. In Table 1, the first column is the Chinese name of each field and the English name in parentheses, and the second and third columns are the two schedule table items.

[0069] Table 1 Task Plan

[0070] Table 2 Rule Mapping Table

[0071] Specifically, for item 1 of the plan table in Table 1, it adopts the first scheduling rule. Combined with Table 2, this scheduling rule indicates that the trigger condition is daily synchronization, the billing period is the previous day as the data billing period, and the synchronization time is 2:00 AM. Therefore, the plan item in column 2 of Table 1 indicates that the first cluster needs to synchronize all data generated nationwide the previous day from the `manchu_user_day` data table in the `cip` database to the second cluster every day.

[0072] Specifically, for item 3 in the plan table of Table 1, it adopts scheduling rule 2. Combined with Table 2, this scheduling rule indicates that the trigger condition is synchronization every Monday, the data billing period is based on the previous Friday, and the synchronization time is 3:00 AM. Therefore, the plan item in column 3 of Table 1 indicates that the first cluster needs to synchronize all data generated in Beijing on the previous Friday from the stay_day data table in the ss_daas database to the second cluster every Monday.

[0073] As for the third scheduling rule in Table 2, it states that the trigger condition is to trigger synchronization on the first day of each month. The account period synchronization rule is that the first day of the previous month is used as the data account period when synchronizing on the first day of each month, and the synchronization time is 4:00 AM.

[0074] It should be noted that this example is for illustrative purposes only and is not intended to be limiting.

[0075] S103. When each cluster determines itself to be the source cluster of any schedule item, it performs synchronization task configuration based on the schedule item; when it determines itself to be the target cluster of any schedule item, it performs partitioning task configuration based on the schedule item.

[0076] In this embodiment, after receiving the task schedule, each cluster can parse it to identify each schedule item related to itself. That is, each cluster needs to parse each schedule item one by one. The specific parsing process of a cluster for each schedule item may include steps S1031 to S1034.

[0077] S1031. For each plan entry, the cluster obtains the IP address of the source cluster, the IP address of the target cluster, the IP address of the synchronization node, and the IP address of the partition node from the preset object mapping table based on the plan entry.

[0078] In this embodiment, the object mapping table mainly includes the IP addresses corresponding to each cluster ID and each node ID, as well as some other information. Therefore, the source cluster ID, target cluster ID, synchronization node ID, and partition node ID in a plan table entry can all be found in the object mapping table with their corresponding IP addresses.

[0079] For example, suppose the object mapping table is as shown in Table 3 below: Table 3 Object Mapping Table

[0080] Therefore, for the source cluster ID, target cluster ID, synchronization node ID, and partition node ID (i.e., cluster_id 1, 2, 3, and 5 respectively) in column 2 of Table 1, the corresponding IP addresses can be found in Table 3 as IP1, IP2, IP3, and IP5. It should be noted that this example is for illustrative purposes only and is not intended to be limiting.

[0081] S1032. The cluster determines the triggering conditions and billing cycle synchronization rules associated with the rule IDs in the schedule table based on the preset rule mapping table.

[0082] The implementation of step S1032 is consistent with the principle of determining the triggering conditions and the account period synchronization rules of the scheduling node mentioned above, and will not be elaborated here.

[0083] S1034. When the cluster's external IP address is the same as the IP address of the source cluster, it notifies the corresponding synchronization node to create a scheduled synchronization task, and uses the triggering condition, the billing cycle synchronization rule, the IP address of the target cluster, the IP address of the partition node, the data table ID, the data table name, the database name, the source path, and the data range in the schedule as the required information for the scheduled synchronization task.

[0084] During this process, the cluster compares its external IP address with the IP address of the source cluster in the schedule entry. If they match, it indicates that this cluster is the data synchronizing party. At this point, the IP address of the synchronization node in the cluster indicates which node is responsible for synchronization. This synchronization node will create a scheduled synchronization task and then save the necessary fields as requirement information. In essence, this scheduled synchronization task simply instructs the synchronization node to periodically access the task repository according to the corresponding trigger conditions to retrieve the tasks to be executed issued by the scheduling node.

[0085] S1035. When the cluster's external IP address is the same as the target cluster's IP address, it notifies the corresponding partition node to create a scheduled partition task and saves the triggering conditions, billing cycle synchronization rules, source cluster IP address, synchronization node IP address, data table ID, data table name, database name, target path, and data range in the schedule table as the requirements information for the scheduled partition task.

[0086] During this process, the cluster compares its external IP address with the target cluster's IP address in the schedule. If they match, it indicates that this cluster is the data receiver (and requires partitioning). At this point, the IP address of the partitioning node in the cluster specifies which node is responsible for partitioning. This partitioning node will create a scheduled partitioning task and then save the necessary fields as required information. Essentially, this scheduled partitioning task only requires the synchronization node to: upon receiving synchronized data, persist it to disk as required, and perform consistency checks and partitioning operations.

[0087] S104. When the scheduling node satisfies any scheduling rule at the current time, it adds a task to be executed to the task list for each timed task corresponding to the scheduling rule.

[0088] It is understandable that scheduling one schedule item corresponds to one scheduled task, and the scheduling rules for different scheduled tasks may be the same or different.

[0089] The scheduling node can enable all scheduling rules used for all schedule entries, while disabling any unused rules. Therefore, when any scheduling rule A is satisfied, the scheduling node needs to publish a pending task in the task list for each scheduled task corresponding to scheduling rule A. The fields of this pending task information can reflect: task ID, specified data table, source cluster IP address, target cluster IP address, the used scheduling rule, and the specified data payment period, etc.

[0090] S105. The scheduling node sets the newly added tasks to be executed to the pending state and initializes the synchronization log and partition log associated with the newly added tasks to be executed.

[0091] In this embodiment, the scheduling node sets the task status field of the newly added task to be executed to "pending execution". The initialization operation includes pre-allocating storage space for synchronization logs and partition logs in the task library for the task to be executed, setting globally unique log IDs for the synchronization logs and partition logs, and writing initial null values ​​or default placeholders to some fields of the synchronization logs and partition logs, ensuring the log structure is ready at the start of the task's lifecycle.

[0092] The synchronization log records relevant operation data during the synchronization process, while the partition log records relevant operation data during the partitioning process. Logically, they belong to the same task to be executed.

[0093] Through the above steps S101~S105, the scheduling node can act as a global scheduling center, responsible for parsing the task schedule table, generating scheduled tasks, and uniformly publishing tasks to be executed. Engineers only need to configure the scheduling node, which ensures high reliability, low maintenance, and easy scalability for cross-cluster data synchronization and partitioned tasks.

[0094] The above describes the scheduler's plan parsing process, the creation of scheduled tasks, the publication of tasks to be executed, and the plan parsing process for each cluster. Next, we will introduce the detailed process of cross-cluster data synchronization and partitioning.

[0095] In an optional implementation, it is assumed that in step S201 above, the synchronization node and partition node involved in the task to be executed obtained by the first cluster are the first synchronization node in the first cluster and the second partition node in the second cluster, respectively.

[0096] In step S210 above, when the first synchronization node determines that the preset timed synchronization task meets the scheduling rules, it retrieves the task to be executed from the task list in the task library. For example, referring to Tables 1 and 2, if the cluster ID of the first synchronization node is 1, then the first synchronization node will retrieve the task to be executed from item 1 of the schedule table from the task list at 2:00 AM every day. It should be noted that this example is only for illustration and is not intended to be limiting.

[0097] The implementation process of step S202 may include S2021 to S2024.

[0098] S2021. Based on the requirements of the scheduled synchronization task, the first synchronization node determines the account period to be synchronized, the specified data table, the specified data range, the specified source path, the IP address of the second cluster, and the IP address of the second partition node.

[0099] S2022, The first synchronization node reads the target data from the specified source path based on the specified data table, specified data range, and the account period to be synchronized.

[0100] S2023, The first synchronization node encapsulates the target data into a synchronization message and sends it to the second partition node based on the task ID of the task to be executed, the IP address of the second cluster, and the IP address of the second partition node.

[0101] It is understandable that the destination IP address of the synchronization message is the IP address of the second cluster. In this way, the synchronization message will first be sent to the external access node in the second cluster, and then the external access node of the second cluster will forward the synchronization message to the second partition node.

[0102] S2024. After the first synchronization node sends the synchronization message, it writes the number of files containing the target data and the file size into the synchronization log.

[0103] The implementation process of step S203 may include S2031 to S2035.

[0104] S2031. When the second partition node receives the synchronization message sent by the first synchronization node, it parses the synchronization message to obtain the target data and the task ID of the task to be executed.

[0105] S2032. The second partition node determines the account period to be synchronized, the specified data table, the specified data range, and the specified target path based on the requirements of the scheduled partitioning task.

[0106] S2033, The second partition node stores the target data into the specified target path.

[0107] S2034. The second partition node reads the number of target data files and file size from the synchronization log associated with the task ID of the task to be executed.

[0108] S2035. The second partition node verifies whether the number and size of the read files are consistent with the number and size of the target data under the specified target path; if yes, the consistency check is confirmed to be successful; otherwise, the consistency check is confirmed to be unsuccessful.

[0109] Simultaneously, upon successful consistency verification, the second partition node writes a synchronization success flag to the corresponding associated synchronization log. At this point, the synchronization log records various information about the synchronization process, such as... Figure 4 The synchronization log shown.

[0110] When the consistency check fails, the second partition node will directly clear the target data under the specified target path and write a synchronization failure flag and failure information into the corresponding synchronization log so that the first synchronization node can resynchronize the data.

[0111] The implementation process of step S204 may include S2051 to S2052.

[0112] S2051. The second partition node generates at least one partition command corresponding to the task to be executed, based on the account period to be synchronized, the specified data table, the specified data range, and the specified target path.

[0113] For example, partitioning operations may involve creating, updating, or adjusting the storage location of partitions, so more than one partitioning command may be generated.

[0114] Taking a target partition that needs to add target data as an example, the process of generating a partition creation command can be as follows: the accounting period to be synchronized determines the value of the time dimension in the partition field (e.g., date_dt='20241231'), the data table is specified to determine the operation object (e.g., ALTER TABLE cip.user_labels_app_daily_kdw), the data range is specified to determine the value of the region dimension (e.g., province='091', indicating the corresponding Liaoning city), and the target path is specified by providing the physical address of the LOCATION clause (e.g., ' / user / ss_deploy / ss_data / user_labels_app_daily / 091 / 2024 / 12 / 31'). After these four elements are combined, the second partition node can construct a partition command that conforms to the target data warehouse syntax specification. Its form can be a standard SQL statement or a proprietary interface call request. When the data range involves multiple discrete regions (e.g., simultaneously including Beijing and Shanghai) or multiple accounting period shards, multiple partition commands are automatically generated to cover all regions.

[0115] For example, a partition creation command can be: ALTER TABLE cip.user_labels_app_daily_kdw ADD IF NOT EXISTS PARTITION(province='091', date_dt='20241231') LOCATION ' / user / ss_deploy / ss_data / user_labels_app_daily / 091 / 2024 / 12 / 31' It should be noted that the above are just examples and are not intended to be limiting.

[0116] S2052. The second partition node uses a batch submission method to execute at least one partition command corresponding to the task to be executed, so as to add the target partition corresponding to the target data in the metadata of the specified data table.

[0117] In this embodiment, the second partition node encapsulates multiple partition commands of the current task to be executed, or even partition commands of other tasks to be executed, into atomic transactions or batch processing units within a single session, which are then uniformly parsed and scheduled by the data warehouse execution engine.

[0118] This significantly reduces the number of network round trips between the second partition node and the target data warehouse, reduces connection establishment and release overhead, and avoids the waste of resources caused by a single command failure leading to a complete rollback. More importantly, batch commits allow multiple partition commands to share the same context (such as transaction ID, session variables, and permission context), ensuring that a logically consistent set of partitions is formed at the metadata level of the specified data table, rather than isolated, fragmented, and potentially metadata-fragmented independent entries. Therefore, batch commits not only avoid the performance bottleneck of traditional row-by-row commits in high-concurrency, large-data-volume scenarios, but also prevent the risk of metadata inconsistency caused by asynchronous execution or asynchronous states of partition commands.

[0119] Optionally, when a partitioning operation fails, the second partitioning node will also write the committed partitioning command and the reason for the failure to the corresponding partitioning log for later traceability. For example, such as... Figure 5 The image shows the partition log after a partitioning operation fails.

[0120] Understandable. Figure 4 , Figure 5 The images shown are for illustrative purposes only and do not fully display the fields of the log. The actual content of some fields is not fully displayed due to the width of the page.

[0121] Regarding step S206 above, after the partition maintenance operation is completed, the second partition node will directly read the storage path of the target partition; if the read is successful, a partition success flag will be written into the partition log, and the task status field of the task to be executed will be updated to the executed status; if the read fails, a partition failure flag and failure information will be written into the partition log.

[0122] This invention also provides a federated cluster, which includes a first cluster and a second cluster, and the first cluster and the second cluster are used together to implement the data synchronization and partitioning methods described above.

[0123] In summary, this invention provides a data synchronization and partitioning method and a federated cluster. First, when a first cluster determines that a preset scheduled synchronization task meets the scheduling rules, it retrieves the task to be executed from the task list in the task library. The task library includes synchronization logs and partition logs associated with the task to be executed. The scheduling rules reflect the triggering conditions and billing cycle synchronization patterns of the task to be executed. Then, the first cluster retrieves the target data based on the requirements of the scheduled synchronization task and synchronizes the target data to the second cluster, writing verification information into the synchronization log. When the second cluster receives the target data, it stores the target data based on the requirements of the preset scheduled partitioning task, retrieves verification information from the synchronization log to perform consistency verification on the target data, and writes a synchronization success flag into the synchronization log when the consistency verification is successful. Next, the second cluster performs partition maintenance operations based on the requirements of the scheduled partitioning task and the storage path of the target data to determine the target partition of the target data. Finally, the second cluster reads the storage path of the target partition. If it can be read normally, it writes a partition success flag into the partition log and updates the task to be executed to the executed state. Compared with existing technologies, the advantages of this invention are as follows: After the target data is synchronized, the first cluster provides verification information to the second cluster through the synchronization log, enabling the second cluster to avoid problems such as missing files, inconsistent file sizes, or incomplete data write-to-disk through consistency verification. After the partition maintenance operation is completed, the second cluster will also verify whether the target data can be read normally to ensure the normal availability of the data. Furthermore, the use of synchronization logs and partition logs for status management during the execution of tasks ensures the traceability of the execution process.

[0124] The above are merely specific embodiments of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in the present invention should be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A data synchronization and partitioning method, characterized in that, include: When the first cluster determines that the preset timed synchronization task meets the scheduling rules, it retrieves the task to be executed from the task list in the task library; the task library includes the synchronization log and partition log associated with the task to be executed; the scheduling rules reflect the triggering conditions and billing cycle synchronization rules of the task to be executed. After the first cluster obtains the target data based on the requirements of the timed synchronization task and synchronizes the target data to the second cluster, it writes verification information into the synchronization log. When the second cluster receives the target data, it stores the target data based on the preset requirements of the timed partitioning task, and then obtains the verification information from the synchronization log to perform consistency verification on the target data. When the second cluster successfully completes the consistency check, it writes a synchronization success flag into the synchronization log. The second cluster performs partition maintenance operations based on the requirements of the scheduled partitioning task and the storage path of the target data to determine the target partition of the target data; The second cluster reads the storage path of the target partition. If the read is successful, a partition success flag is written to the partition log, and the task to be executed is updated to the executed status.

2. The data synchronization and partitioning method according to claim 1, characterized in that, The federated cluster comprises at least two clusters, the first cluster and the second cluster being part of the federated cluster, and the federated cluster includes scheduling nodes; each cluster supports access to the task library; Before the step of retrieving task information of tasks to be executed from the task list in the task library when the first cluster triggers the synchronization condition, the method further includes: The scheduling node obtains the task plan table; the task plan table includes multiple plan table entries, which reflect the data tables, source cluster, target cluster, data range, and scheduling rules involved in a cross-cluster data synchronization and partitioning task; The scheduling node creates a scheduled task for each of the schedule entries, configures the requirements for the scheduled task, and sends the task schedule to each cluster. When each cluster determines that it is the source cluster of any of the aforementioned schedule entries, it performs synchronization task configuration based on the schedule entries; and when it determines that it is the target cluster of any of the aforementioned schedule entries, it performs partitioning task configuration based on the schedule entries. When the scheduling node satisfies any of the scheduling rules at the current time, it adds a task to be executed to the task list for each timed task corresponding to the scheduling rule; The scheduling node sets the newly added tasks to the pending execution state and initializes the synchronization logs and partition logs associated with the newly added tasks.

3. The data synchronization and partitioning method according to claim 2, characterized in that, The plan table entries include data table ID, data table name, database name, rule ID, source path, target path, data range, source cluster ID, target cluster ID, synchronization node ID, and partition node ID; The steps of the scheduling node creating a scheduled task for each of the schedule entries and configuring the requirements information of the scheduled task include: For each of the plan entries, the scheduling node determines the triggering conditions and billing period synchronization rules associated with the rule ID in the plan entry from a preset rule mapping table, thereby obtaining the triggering conditions and billing period synchronization rules of the plan entry. The scheduling node creates a scheduled task based on the triggering conditions of the schedule entry; The scheduling node uses the data table name, database name, source cluster ID, target cluster ID, and billing cycle synchronization rule of the plan table item as the requirement information for the scheduled task.

4. The data synchronization and partitioning method according to claim 2, characterized in that, The plan table entries include data table ID, data table name, database name, rule ID, source path, target path, data range, source cluster ID, target cluster ID, synchronization node ID, and partition node ID; The steps of configuring synchronization tasks based on the schedule entry when each cluster determines itself to be the source cluster of any of the schedule entries, and configuring partitioning tasks based on the schedule entry when it determines itself to be the target cluster of any of the schedule entries, include: For each of the plan entries, the cluster obtains the IP address of the source cluster, the IP address of the target cluster, the IP address of the synchronization node, and the IP address of the partition node from a preset object mapping table based on the plan entry. The cluster determines the triggering conditions and billing cycle synchronization rules associated with the rule IDs in the plan table entries based on a preset rule mapping table. When the cluster's external IP address is the same as the IP address of the source cluster, it notifies the corresponding synchronization node to create a scheduled synchronization task, and uses the triggering condition, the billing period synchronization rule, the IP address of the target cluster, the IP address of the partition node, the data table ID, data table name, database name, source path, and data range in the schedule table as the required information for the scheduled synchronization task. When the cluster's external IP address is the same as the target cluster's IP address, it notifies the corresponding partition node to create a scheduled partition task and saves the triggering condition, the billing cycle synchronization rule, the source cluster's IP address, the synchronization node's IP address, the data table ID, data table name, database name, target path, and data range in the schedule table as the requirements information for the scheduled partition task.

5. The data synchronization and partitioning method according to claim 1, characterized in that, The first cluster and the second cluster each include a first synchronization node and a second partition node; The step of the first cluster obtaining target data based on the requirements of the scheduled synchronization task and synchronizing the target data to the second cluster, and then writing verification information into the synchronization log, includes: Based on the requirements of the scheduled synchronization task, the first synchronization node determines the billing period to be synchronized, the specified data table, the specified data range, the specified source path, the IP address of the second cluster, and the IP address of the second partition node. The first synchronization node reads the target data from the specified source path based on the specified data table, the specified data range, and the billing period to be synchronized; The first synchronization node encapsulates the target data into a synchronization message and sends it to the second partition node based on the task ID of the task to be executed, the IP address of the second cluster, and the IP address of the second partition node; After the synchronization message is sent, the first synchronization node writes the number of files containing the target data and the file size into the synchronization log.

6. The data synchronization and partitioning method according to claim 1, characterized in that, The first cluster and the second cluster each include a first synchronization node and a second partition node; When the second cluster receives the target data, it stores the target data based on the preset requirements of the timed partitioning task, and then retrieves the verification information from the synchronization log to perform consistency verification on the target data. The steps include: When the second partition node receives the synchronization message sent by the first synchronization node, it parses the synchronization message to obtain the target data and the task ID of the task to be executed; The second partition node determines the account period to be synchronized, the specified data table, the specified data range, and the specified target path based on the requirements information of the scheduled partitioning task. The second partition node stores the target data into the specified target path; The second partition node reads the number of files and file size of the target data from the synchronization log associated with the task ID of the task to be executed; The second partition node verifies whether the number and size of the read files are consistent with the number and size of the target data under the specified target path; if yes, the consistency verification is confirmed to be successful; otherwise, the consistency verification is confirmed to be unsuccessful.

7. The data synchronization and partitioning method according to claim 6, characterized in that, The second cluster performs a partition maintenance operation based on the requirements of the scheduled partitioning task and the storage path of the target data to determine the target partition of the target data, including the following steps: The second partition node generates at least one partition command corresponding to the task to be executed based on the billing period to be synchronized, the specified data table, the specified data range, and the specified target path; The second partition node uses a batch submission method to execute at least one partition command corresponding to the task to be executed, so as to add the target partition corresponding to the target data in the metadata of the specified data table.

8. The data synchronization and partitioning method according to claim 6, characterized in that, The method further includes: When the consistency check fails, the second partition node clears the target data under the specified target path and writes a synchronization failure flag and failure information into the synchronization log so that the first synchronization node can resynchronize the data.

9. The data synchronization and partitioning method according to claim 7, characterized in that, The step of the second cluster reading the storage path of the target partition includes: The second partition node reads the storage path of the target partition; The method further includes: If the read fails, the second partition node writes a partition failure flag and failure information into the partition log.

10. A federated cluster, characterized in that, The federated cluster includes a first cluster and a second cluster; The first cluster and the second cluster are used together to implement the data synchronization and partitioning method according to any one of claims 1-9.

Images