Cross-region data synchronization method, system and computer readable medium

Abstract

The application relates to a cross-region data synchronization method, a system and a computer readable medium. The cross-region data synchronization method comprises the following steps: submitting a data synchronization statement according to a syntax structure of a data synchronization language; analyzing the data synchronization statement to obtain a synchronization plan, the synchronization plan being suitable for synchronizing data of unit nodes of one or more synchronization regions to a center node; and executing the synchronization plan to obtain an execution result. The technical scheme of the application standardizes the content of the data synchronization statement through the unified syntax structure of the data synchronization language, obtains the corresponding synchronization plan according to the data synchronization statement, and all the synchronization plans do not need to be assisted by additional scheduling and development, the synchronization plan is easy to maintain and has low development cost; after the synchronization plan is executed, the data of the unit nodes of each synchronization region is synchronized to the center node, and the cross-region data synchronization method can be uniformly deployed at the center node to realize data synchronization.

Description

Technical Field

[0001] This invention relates primarily to the field of computer information technology, and more specifically to a cross-regional data synchronization method, system, and computer-readable medium. Background Technology

[0002] Against the backdrop of the General Data Protection Regulation (GDPR) which strictly regulates the rules for data processing, storage and management, the need for localized data storage across multiple geographical regions in globalized businesses is becoming increasingly urgent. However, some data needs still require global statistics, and traditional data transmission solutions can no longer meet the needs of multi-location deployment. Furthermore, with the increase in business needs, it is necessary to synchronize and aggregate data stored in different geographical regions to a unified node for further processing and calculation.

[0003] Figure 1 This is an exemplary architecture diagram of a cross-regional data synchronization solution in the prior art, for reference. Figure 1 As shown, data storage locations across geographical regions are divided into unit nodes and a central node. Each node has a corresponding data integration service, which is used to receive, process, and store data from the local area. Due to business or technical requirements, it is necessary to synchronize the data from each unit node to a unified central node. Currently, existing data transmission synchronization solutions are mainly divided into offline synchronization and real-time synchronization. Offline synchronization is mainly based on offline transmission schemes using batch file transfer, while real-time synchronization is mainly based on transmission schemes using interface services. Figure 1 The data synchronization methods include: real-time data transmission client, real-time synchronization receiving service, and offline file synchronization scheme. The real-time data transmission client is responsible for obtaining real-time data from each unit node and transmitting the data to the central node through the interface service. The real-time synchronization receiving service is responsible for receiving and processing the data from each unit node and distributing the data to the corresponding data channel of the central node. The offline file synchronization scheme is responsible for comparing and retrieving files from the files stored locally on each unit node through the file interface and writing the files to the central node. It also records the time and status of each file transmission to facilitate restarting the task when the transmission process fails and avoid repeated file transmission.

[0004] refer to Figure 1As shown, existing cross-regional data synchronization solutions require at least two sets of components and corresponding code: an offline file synchronization solution, a real-time synchronization receiving service, and a real-time data transmission client. Furthermore, the offline transmission solution is dependent on other components and complex, while the real-time transmission solution has high development costs, difficulty in guaranteeing data consistency, and requires significant deployment resources. Additionally, unit nodes across different geographical regions may use different local storage systems, requiring different file interfaces during the development of the synchronization solution. As the number of unit nodes increases, the required deployment resources and maintenance costs also increase. Therefore, there is an urgent need for a unified, easily maintainable, and low-development-cost cross-regional data synchronization solution. Summary of the Invention

[0005] The technical problem to be solved by this application is to provide a cross-regional data synchronization method, system and computer-readable medium that can achieve unified deployment, easy maintenance and low development cost.

[0006] The technical solution adopted in this application to solve the above-mentioned technical problems is a cross-regional data synchronization method, which includes: submitting a data synchronization statement according to the syntax structure of a data synchronization language; parsing the data synchronization statement to obtain a synchronization plan, the synchronization plan being suitable for synchronizing the data of unit nodes in one or more synchronization regions to the central node; and executing the synchronization plan to obtain the execution result.

[0007] In one embodiment of this application, before the step of submitting a data synchronization statement according to the syntax structure of the data synchronization language, the method further includes: configuring the address and directory of the Alluxio data orchestration service, which is used to mount the underlying file system for access by the upper-layer computing framework and applications.

[0008] In one embodiment of this application, before the step of submitting a data synchronization statement according to the syntax structure of the data synchronization language, the method further includes: configuring the address of the metadata service, the address of the underlying file system of one or more synchronization zones, the execution directory of the computing engine, the communication address of the synchronization engine, the port of the synchronization engine, the communication address of the synchronization executor, the port of the synchronization executor, the address of the synchronization network service, and the port of the synchronization network service, or any one of these.

[0009] In one embodiment of this application, the syntax structure includes configuration items, which include one or any of the following: scheduling period, scheduling time zone, computing engine, whether to select local time zone, source data table partition, target data table partition, list of partitions to be synchronized, and view name, wherein the source data table corresponds to the unit node and the target data table corresponds to the center node.

[0010] In one embodiment of this application, the synchronization plan includes an offline synchronization plan and / or a real-time synchronization plan, wherein the real-time synchronization plan includes a view name.

[0011] In one embodiment of this application, the steps of parsing data synchronization statements include: parsing the data synchronization statements according to the syntax structure to obtain the parsing result; obtaining a data synchronization syntax tree after the parsing result is successful, the data synchronization syntax tree including configuration item information, the table name of the source data table, and the table name of the target data table; connecting to the metadata service according to the address of the metadata service, and determining whether the target data table exists in the metadata service according to the table name of the target data table; if the target data table exists, creating a mapping directory of Alluxio data orchestration service corresponding to the source data table; mounting the source table directory in the address of the underlying file system of one or more synchronization zones to the mapping directory; creating a mapping directory source table, the location of the mapping directory source table being the mapping directory; determining whether to perform real-time synchronization, and if not, generating an offline synchronization plan; if real-time synchronization is performed, creating a view according to the view name and generating a real-time synchronization plan.

[0012] In one embodiment of this application, the step of parsing the data synchronization statement further includes: generating a file metadata synchronization plan for the Alluxio data orchestration service and a partition metadata synchronization plan for the metadata service based on the configuration item information, the address of the Alluxio data orchestration service, and the address of the metadata service; the computing engine reads the data fields from the source table of the mapping directory and writes the data fields into the target data table; and storing one or more of the offline synchronization plan, the real-time synchronization plan, the file metadata synchronization plan for the Alluxio data orchestration service, and the partition metadata synchronization plan for the metadata service into the database of the central node.

[0013] In one embodiment of this application, after parsing the data synchronization statement and obtaining the synchronization plan, the method further includes: periodically reading the synchronization plan and determining whether the synchronization plan meets the operating conditions, including the plan execution time zone and the plan execution time; if the operating conditions are met, the synchronization plan is executed.

[0014] In one embodiment of this application, after the step of executing the synchronization plan and obtaining the execution result, the method further includes: determining whether the execution result is successful; if the execution result is successful, writing the successful execution result into the log; if the execution result is unsuccessful, issuing an alarm and writing the unsuccessful execution result into the log.

[0015] In one embodiment of this application, after the steps of executing the synchronization plan and obtaining the execution result, the method further includes displaying the synchronization plan and the execution result.

[0016] In one embodiment of this application, in the step of displaying the synchronization plan and execution results, the synchronization plan includes any synchronization plan stored in the database of the central node.

[0017] To address the aforementioned technical problems, this application also proposes a cross-regional data synchronization system, comprising: a synchronization client module for submitting data synchronization statements according to the syntax structure of a data synchronization language; a synchronization engine module for parsing the data synchronization statements and obtaining a synchronization plan, the synchronization plan being suitable for synchronizing data from unit nodes in one or more synchronization regions to a central node; a synchronization executor module for executing the synchronization plan and obtaining execution results; a memory for storing instructions executable by a processor; and a processor for executing the instructions to implement the above-mentioned data synchronization method.

[0018] In one embodiment of this application, the data synchronization system further includes a synchronization network service module for displaying the synchronization plan and execution results on the front end.

[0019] To address the aforementioned technical problems, this application also proposes a computer-readable medium storing computer program code, which, when executed by a processor, implements the data synchronization method described above.

[0020] The technical solution of this application standardizes the content of data synchronization statements through a unified data synchronization language syntax structure. Based on the data synchronization statements, corresponding synchronization plans are obtained. All synchronization plans do not require additional scheduling and development. The synchronization plans are easy to maintain and have low development costs. After the synchronization plan is executed, the data of the unit nodes in each synchronization area is synchronized to the central node. Data synchronization can be achieved by uniformly deploying cross-regional data synchronization methods at the central node. Attached Figure Description

[0021] To make the above-mentioned objectives, features and advantages of this application more apparent and understandable, the specific embodiments of this application will be described in detail below with reference to the accompanying drawings, wherein:

[0022] Figure 1 This is an exemplary architecture diagram of a cross-regional data synchronization scheme in the prior art;

[0023] Figure 2 This is an exemplary flowchart of a cross-regional data synchronization method according to an embodiment of this application;

[0024] Figure 3 This is an exemplary flowchart of a cross-regional data synchronization method according to another embodiment of this application;

[0025] Figure 4 This is an exemplary architecture diagram of a XenSQL data synchronization service according to an embodiment of this application;

[0026] Figure 5 This is an exemplary architecture diagram of a cross-regional data synchronization system according to an embodiment of this application;

[0027] Figure 6This is an exemplary deployment diagram of a cross-regional data synchronization system according to another embodiment of this application;

[0028] Figure 7 This is a system block diagram of a cross-regional data synchronization system according to an embodiment of this application. Detailed Implementation

[0029] To make the above-mentioned objectives, features and advantages of this application more apparent and understandable, the specific embodiments of this application will be described in detail below with reference to the accompanying drawings.

[0030] Many specific details are set forth in the following description in order to provide a full understanding of this application. However, this application may also be implemented in other ways different from those described herein, and therefore this application is not limited to the specific embodiments disclosed below.

[0031] As indicated in this application and claims, unless the context clearly indicates otherwise, the words "a," "an," "an," and / or "the" are not specifically singular and may include plural forms. Generally speaking, the terms "comprising" and "including" only indicate the inclusion of explicitly identified steps and elements, which do not constitute an exclusive list, and the method or apparatus may also include other steps or elements.

[0032] Flowcharts are used in this application to illustrate the operations performed by the system according to embodiments of this application. It should be understood that the preceding or following operations are not necessarily performed in exact order. Instead, various steps can be processed in reverse order or simultaneously. Furthermore, other operations may be added to these processes, or one or more steps may be removed from these processes.

[0033] This application proposes a cross-regional data synchronization method suitable for synchronizing data from unit nodes across geographical synchronization regions to a unified central node, such as synchronizing data from North America and Europe to China. This data synchronization method eliminates the need for users to worry about time zone differences, file system differences, and other issues in cross-regional data synchronization.

[0034] Figure 2 This is an exemplary flowchart of a cross-regional data synchronization method according to an embodiment of this application, with reference to... Figure 2 As shown, the cross-regional data synchronization method of this embodiment includes the following steps:

[0035] Step S110: Submit the data synchronization statement according to the syntax structure of the data synchronization language.

[0036] Step S120: Parse the data synchronization statement to obtain the synchronization plan, which is suitable for synchronizing the data of one or more unit nodes in the synchronization area to the central node.

[0037] Step S130: Execute the synchronization plan and obtain the execution result.

[0038] The following details steps S110 to S130:

[0039] In step S110, a data synchronization statement is submitted according to the syntax structure of the data synchronization language.

[0040] This application introduces a novel Data Synchronization Language (DSL) and standardizes its syntax. Data synchronization statements are used to configure the necessary parameters during the data synchronization process. This application also creates a data synchronization service to parse these statements and execute various operations during the data synchronization process. It should be noted that this application names the data synchronization service Xensql, and Xensql will be used to refer to this data synchronization service in the following text. The syntax structure of the DSL and its statement examples will be detailed later.

[0041] In some embodiments, the syntax structure includes configuration items, which include one or any one of the following: scheduling period, scheduling time zone, computing engine, whether to select local time zone, source data table partition, target data table partition, list of partitions to be synchronized, and view name, wherein the source data table corresponds to the cell node and the target data table corresponds to the center node.

[0042] For example, the configuration item information included in the syntax structure of the DSL of this application is shown in Table 1 below:

[0043] Table 1. List of configuration items for the syntax structure of the DSL in this application.

[0044]

[0045] The following example illustrates the data synchronization statement submitted according to the syntax structure of the DSL in this application:

[0046] Example 1: Offline synchronization.

[0047] The source data table for offline synchronization is ods.a, the target data table is ods.b, and the task name is test_a_to_b.

[0048] Offline data synchronization statement:

[0049] create sync test_a_to_b

[0050] table ods.a(to_id, table_msn)

[0051] to

[0052] table ods.b(cid,msn)

[0053] with(

[0054] schedule.time='0 00 07? **',

[0055] schedule.default.region = ch,

[0056] compute.engine.sql = spark,

[0057] schedule.local.region = true,

[0058] source.partition.field=region,dt,

[0059] sink.partition.field=region,dt,

[0060] sink.partition.list = us,jp )

[0062] Example 2: Real-time synchronization.

[0063] The source data table for real-time synchronization is ods.a, the target data table is ods.b, and the task name is test_real_a_to_b.

[0064] Real-time data synchronization statements:

[0065] create realsync test_real_a_to_b

[0066] table ods.a(to_id, table_msn)

[0067] to

[0068] table ods.b(cid,msn)

[0069] with(

[0070] schedule.time='0 00 07? **',

[0071] schedule.default.region = ch,

[0072] compute.engine.sql = spark,

[0073] schedule.local.region = true,

[0074] source.partition.field=region,dt,

[0075] sink.partition.field=region,dt,

[0076] sink.partition.list=us,jp,

[0077] realsync.view.name=ods.b_view )

[0079] Figure 3 This is an exemplary flowchart of a cross-regional data synchronization method according to another embodiment of this application. (See reference...) Figure 3 As shown, this embodiment primarily utilizes the XenSQL data synchronization service to execute cross-region data synchronization. The XenSQL data synchronization service comprises four modules: the first is the XenSQL synchronization client module 310, which submits the DSL language; the second is the XenSQL synchronization engine module 320, responsible for parsing the DSL syntax and obtaining and scheduling the synchronization plan; the third is the XenSQL synchronization executor module 330, responsible for executing the synchronization plan issued by the XenSQL synchronization engine module 320; and the fourth is the XenSQL synchronization network service module 340, responsible for reading the synchronization plan and execution results for front-end querying and display. The following will combine... Figure 3 The preceding data synchronization statements, Example 1 (offline synchronization) and Example 2 (real-time synchronization), detail the steps involved in cross-region data synchronization.

[0080] In some embodiments, reference Figure 3 As shown, before the step of submitting the data synchronization statement according to the syntax structure of the data synchronization language, the following steps are also included: configuring the address and directory of the Alluxio data orchestration service, which is used to mount the underlying file system for access by the upper-layer computing framework and applications. This step corresponds to... Figure 3 Step S3103.

[0081] For example, the Alluxio data orchestration service unifies data access methods through a unified namespace, providing a unified client application programming interface (API) for underlying storage systems and upper-layer computing frameworks. Alluxio offers a multi-layer caching scheme, accelerating data access and avoiding repeated readings of duplicate data. Utilizing the unified namespace of Alluxio components, it provides a unified access method for various storage systems, reducing data access integration costs and offering multiple file access interfaces to support various upper-layer applications or computing frameworks, such as Hive and Spark. For instance, a schema mapping table based on the Alluxio file system can be established through the Hive metastore, supporting data access and computation for Alluxio without changing the application logic in the existing data warehouse. The Alluxio data orchestration service locally mounts different storage systems in different geographical regions, using the `region` field as the partition identifier for different regions, and sets up memory caching to accelerate file reading and avoid repeated readings of the same file.

[0082] This application uses Alluxio as a data orchestration service to provide a unified access method for storage systems in different geographical regions. It designs and implements both offline and real-time synchronization solutions, reducing later development and maintenance costs. When adding new unit nodes, only the corresponding directory needs to be mounted to the Alluxio directory on the central node, requiring no additional deployment or maintenance costs. Through Alluxio's unified auditing and logging, the overall data synchronization solution is monitored, avoiding the need for monitoring and maintenance across multiple regions.

[0083] In some embodiments, prior to the step of submitting a data synchronization statement according to the syntax structure of the data synchronization language, the method further includes: configuring one or any one of the following: the address of the metadata service, the address of the underlying file system of one or more synchronization zones, the execution directory of the computing engine, the communication address of the synchronization engine, the port of the synchronization engine, the communication address of the synchronization executor, the port of the synchronization executor, the address of the synchronization network service, and the port of the synchronization network service. These steps partially correspond to... Figure 3 Steps S3101 and S3102.

[0084] For example, the Under Database (UDB) can provide data query services for the central node. (See reference...) Figure 3As shown, before the XenSQL data synchronization service executes the cross-region data synchronization method, it is necessary to deploy the XenSQL synchronization client module 310, the XenSQL synchronization engine module 320, the XenSQL synchronization executor module 330, and the XenSQL synchronization network service module 340; and configure the UDB address, the address of the underlying file system (UFS) of each synchronization region, the execution directory of the computing engine, the remote procedure call (RPC) communication address and port of the XenSQL synchronization engine module 320, the RPC communication address and port of the XenSQL synchronization executor module 330, and the service address and port of the XenSQL synchronization network service module 340.

[0085] After the XenSQL synchronization client module 310 starts, the client 311 reads various configuration information, including the address of the UDB used, the UFS address of each synchronization zone, the execution directory of the computing engine, and the RPC communication address of the XenSQL synchronization engine module 320.

[0086] In step S120, the data synchronization statement is parsed to obtain a synchronization plan, which is suitable for synchronizing the data of one or more unit nodes in the synchronization area to the central node.

[0087] In some embodiments, the synchronization plan includes an offline synchronization plan and / or a real-time synchronization plan, wherein the real-time synchronization plan includes a view name.

[0088] For example, refer to Figure 3 As shown in Example 1 (offline synchronization) and Example 2 (real-time synchronization) of the data synchronization statements described above, the data synchronization statement is submitted by client 311 of the XenSQL synchronization client module 310. The XenSQL synchronization engine module 320 parses the data synchronization statement and obtains the corresponding synchronization plan and real-time synchronization plan. The execution flow of the cross-region data synchronization method of this application will be described below based on Example 1 (offline synchronization) and Example 2 (real-time synchronization). For the interpretation of attributes within "with" in the offline synchronization data synchronization statement, please refer to Table 1 (the configuration item list of the syntax structure of the DSL of this application) described above.

[0089] In some embodiments, the steps of parsing data synchronization statements include:

[0090] Step S1201: Parse the data synchronization statement according to the syntax structure to obtain the parsing result;

[0091] Step S1202: After the parsing result is successful, obtain the data synchronization syntax tree. The data synchronization syntax tree includes configuration item information, the table name of the source data table, and the table name of the target data table.

[0092] Step S1203: Connect to the metadata service based on the address of the metadata service, and determine whether the target data table exists in the metadata service based on the table name of the target data table;

[0093] Step S1204: If the target data table exists, create a mapping directory of the Alluxio data orchestration service corresponding to the source data table;

[0094] Step S1205: Mount the source table directory in the address of the underlying file system of one or more synchronization zones to the mapped directory;

[0095] Step S1206: Create a mapping directory source table, the location of which is the mapping directory;

[0096] Step S1207: Determine whether to perform real-time synchronization. If not, generate an offline synchronization plan. If to perform real-time synchronization, create a view based on the view name and generate a real-time synchronization plan.

[0097] For example, refer to Figure 3 The steps S1201 to S1207 above will be explained using the data synchronization statement example 1 (offline synchronization) mentioned above.

[0098] In step S1201 above, after the XenSQL synchronization engine module 320 receives the data synchronization statement submitted by the client 311 of the XenSQL synchronization client module 310 via RPC communication, it parses the data synchronization statement according to the syntax structure of the DSL in step S3201 to obtain the parsing result. If the parsing fails, an exception is thrown in step S3203 and the failure reason is returned to the client 311. If the parsing succeeds, in step S1202 above, the XenSQL synchronization engine module 320 obtains the data synchronization syntax tree, which includes: source table name: ods.a; source table partition fields: region, dt; source table data fields: to_id, table_msn; target table name: ods.b; target table partition fields: region, dt; target table data fields: cid, msn; synchronization task name: test_a_to_b; scheduling period: 0 0007? **(7 AM daily); Scheduling timezone: ch (China); Computing engine: spark; Select local timezone for scheduling: true; List of partitions to be synchronized (synchronization areas): us,jp (United States, Japan).

[0099] In step S1203 above, in Figure 3In step S3202, the metadata service UDB is connected based on the address of the metadata service. In step S3204, it is determined whether the target data table ods.b exists in the metadata service UDB based on the table name ods.b. If it does not exist, an exception is thrown to the client 311 in step S3203. In step S1204 above, if the target data table ods.b exists, then in step S3205, a mapping directory for the Alluxio data orchestration service corresponding to the source data table ods.a is created: / alluxio / warehouse / ods.db / a_sync. Here, / alluxio / warehouse is the configuration of the Alluxio data orchestration service, which can be freely defined. ods.db is the ODS library directory, and a_sync is the source data table a with the suffix _sync added. This part is the directory syntax defined by the XenSQL synchronization engine module 320.

[0100] In step S1205 above, in Figure 3 The step S3206 shown mounts the source table directory in the address of the underlying file system of one or more synchronization zones to the mapping directory. Specifically, based on the parsed synchronization zones us and jp, and based on the configured UFS addresses of each synchronization zone, the UFS addresses of synchronization zones us and jp are obtained, and / ods.db / a in the UFS addresses of synchronization zones us and jp is mounted to the mapping directory of Alluxio, / alluxio / warehouse / ods.db / a_sync.

[0101] In step S1206 above, in Figure 3 The step S3207 shown creates the Alluxio mapping directory source table. Specifically, based on the configured UDB address, the UDB is connected, and the mapping directory source table ods.a_sync is created. The location of the mapping directory source table ods.a_sync is set to the Alluxio mapping directory / alluxio / warehouse / ods.db / a_sync.

[0102] In step S1207 above, in Figure 3 The step S3208 shown determines whether to perform real-time synchronization. If real-time synchronization is not performed, an offline synchronization plan is generated in step S3210. If real-time synchronization is performed, a view is created in step S3209 according to the view name, and a real-time synchronization plan is generated in step S3210. Specifically, the view is created by connecting to the UDB through the realsync.view.name property in the DSL data synchronization statement.

[0103] In some embodiments, the step of parsing data synchronization statements further includes:

[0104] Step S1208: Based on the configuration information, the address of the Alluxio data orchestration service, and the address of the metadata service, generate the file metadata synchronization plan for the Alluxio data orchestration service and the partition metadata synchronization plan for the metadata service;

[0105] Step S1209: The calculation engine reads the data fields from the source table of the mapping catalog and writes the data fields into the target data table;

[0106] Step S1210: Store one or more of the offline synchronization plan, real-time synchronization plan, Alluxio data orchestration service file metadata synchronization plan, and metadata service partition metadata synchronization plan into the database of the central node.

[0107] For example, refer to Figure 3 The steps S1208 to S1210 above will be explained using the data synchronization statement example 1 (offline synchronization) mentioned above.

[0108] In step S1208 above, based on the parsed scheduling cycle, scheduling time zone, partition field, synchronization region, computing engine, synchronization data field configuration, as well as the configured metadata service UDB address and Alluxio data orchestration service address, a file metadata synchronization plan for the Alluxio data orchestration service and a partition metadata synchronization plan for the metadata service UDB are generated. In step S1209 above, based on the parsed computing engine configuration of Spark, Spark SQL is generated to read the data fields to_id and table_msn from the Alluxio mapping directory source table ods.a_sync, and write the data fields cid and msn to the target data table ods.b.

[0109] In step S1210 above, one or more of the offline synchronization plan, real-time synchronization plan, Alluxio data orchestration service file metadata synchronization plan, and metadata service partition metadata synchronization plan are stored in the central node's database 321, where the central node's database 321 is a MySQL database. In step S3211, a message indicating successful DSL execution is returned to the client 311.

[0110] In some embodiments, after parsing the data synchronization statement to obtain the synchronization plan, the method further includes:

[0111] Step S1211: Periodically read the synchronization plan and determine whether the synchronization plan meets the running conditions, including the plan execution time zone and the plan execution time;

[0112] Step S1212: If the operating conditions are met, execute the synchronization plan.

[0113] For example, refer to Figure 3 As shown, the XenSQL synchronization engine module 320 also has a synchronization plan scheduling function. In step S1211 above, the scheduling engine 322 of the XenSQL synchronization engine module 320 reads the synchronization plan periodically every 30 seconds in step S3212, based on the synchronization plan written in the database 321 of the central node. In step S1212 above, in Figure 3 Step S3213 shows that the synchronization plan is determined to meet the running conditions, which include the plan execution time zone and the plan execution time. If the running conditions are not met, the task is terminated in step S3215. If the running conditions are met, the synchronization plan is sent in step S3214, and the synchronization plan is sent to the Xensql synchronization executor module 330 via RPC communication.

[0114] In step S130, the synchronization plan is executed, and the execution result is obtained.

[0115] In some embodiments, after executing the synchronization plan and obtaining the execution result, the method further includes: determining whether the execution result is successful; if the execution result is successful, writing the successful execution result to the log; if the execution result is unsuccessful, issuing an alarm and writing the unsuccessful execution result to the log.

[0116] For example, refer to Figure 3 As shown, the XenSQL synchronization executor module 330 executes the synchronization plan and obtains the execution result. Specifically, the synchronization plan is executed in step S3301, and the execution result is determined in step S3302. If the execution fails, an alarm is triggered in step S3304 and the failed execution result is written to the log; if the execution succeeds, the successful execution result is written to the log in step S3303.

[0117] In some embodiments, after the steps of executing the synchronization plan and obtaining the execution result, the method further includes displaying the synchronization plan and the execution result.

[0118] In some embodiments, in the step of displaying the synchronization plan and execution results, the synchronization plan includes any synchronization plan stored in the database of the central node.

[0119] For example, refer to Figure 3As shown, in the XenSQL Synchronization Network Service module 340, user login occurs in step S3401; in step S3402, a synchronization plan is queried and the corresponding synchronization plan is obtained from the database 321 of the central node; in step S3403, the query results are retrieved, and the logs of the synchronization plan and execution results in the database 321 of the central node are obtained and displayed on the front end. Users can view any synchronization plan stored in the database 321 of the central node, historically decommissioned synchronization plans, failed execution plans, successfully executed synchronization plans, and currently executing synchronization plans through the XenSQL Synchronization Network Service module 340.

[0120] In the context of global data synchronization, the cross-regional data synchronization method and XenSQL data synchronization service of this application eliminate the need for users to worry about time zone differences and file system differences during cross-geographical data synchronization. Users only need to write data synchronization statements that conform to the syntax structure of the Data Synchronization Language (DSL), and the XenSQL data synchronization service will generate corresponding cross-regional offline synchronization plans and real-time synchronization plans based on the data synchronization statements. This application uses Alluxio data orchestration service to provide a unified access method for storage systems in different geographic regions. Combined with metadata service UDB, data from each geographic region is aggregated, calculated, and stored on a central node. This enables data query services on the central node, while the original data remains stored locally, saving time and costs for each query, accumulating data assets, and reducing later development and maintenance costs.

[0121] Figure 4 This is an exemplary architecture diagram of a XenSQL data synchronization service according to an embodiment of this application. (See reference...) Figure 4 As shown, the core of the XenSQL data synchronization service consists of four parts: XenSQL synchronization client 410, XenSQL synchronization engine 430, XenSQL synchronization executor 450, and XenSQL synchronization network service 460.

[0122] refer to Figure 4As shown, the user submits a data synchronization statement 420 in the XenSQL synchronization client 410. After receiving the data synchronization statement 420, the XenSQL synchronization engine 430 parses the syntax of the data synchronization statement 420 in conjunction with the relevant configuration information of the metadata service 440. In step S431, it determines whether the syntax is valid. If the syntax is invalid, an exception is thrown in step S433, and the relevant information is written to the log in step S434. If the syntax is valid, the data synchronization statement is parsed in step S432 and written to the configuration storage. Specifically, after parsing, information such as the offline synchronization plan or real-time synchronization plan, the region and metadata to be synchronized by the user, and the data tables to be synchronized are obtained, and the corresponding synchronization plan is written to the configuration database mysql for storage. The XenSQL synchronization executor 450 executes the synchronization plan according to the corresponding synchronization plan in step S435, and writes the corresponding execution results to the log in step S436. The XenSQL synchronization network service 460 reads the relevant data from the configuration database mysql and the log for front-end display.

[0123] Figure 5 This is an exemplary architecture diagram of a cross-regional data synchronization system according to an embodiment of this application. (Reference) Figure 5 As shown, the architecture of the cross-regional data synchronization system 500 includes: a metadata service 510, an underlying file system 520, an Alluxio data orchestration service 530, a computing engine 540, and a XenSQL data synchronization service 550. Based on this architecture, this application develops a data synchronization service, XenSQL, integrating the Alluxio Software Development Kit (SDK). This application integrates XenSQL and Alluxio to achieve real-time and offline synchronization of data across geographical regions using a DSL.

[0124] Figure 6 This is an exemplary deployment diagram of a cross-regional data synchronization system according to another embodiment of this application. (Reference) Figure 6 As shown, data from unit nodes 630 (North America) and 620 (Europe) needs to be synchronized to the central node 610 (China). For example, an Alluxio data orchestration service cluster is deployed and started on the central node 610 (China); the data directories to be synchronized from China 610, Europe 620, and North America 630 are mounted to the Alluxio data orchestration service 640; the metadata service 650 creates a synchronization plan and adds partitions based on relevant configuration information; the client 660 reads the corresponding synchronization plan and Alluxio data, and writes the relevant information to the mapping directory source table of the synchronization plan in China 610; the scheduling service 670 periodically reads the synchronization plan to schedule its execution.

[0125] This application integrates Alluxio and XenSQL's cross-geographical region synchronization solution, supporting synchronization plans for data in different time zones. Users only need to mount the data for the corresponding region to the directory, without needing to worry about time zone differences. For example, the XenSQL data synchronization service is set to Beijing time zone, and the relevant tables on the central node mount both China and the United States regions. XenSQL generates two synchronization plans accordingly: 1. Synchronize data for the China region at 01:00 Beijing time daily; 2. Synchronize data for the North American region at 13:00 Beijing time daily. The relevant synchronization statements are shown below:

[0126] #Time zone differences:

[0127] Beijing time: 00:00 on July 21 corresponds to US time: 00:00 on July 21 - 12:00 = 12:00 on July 20.

[0128] #Alluxio mount:

[0129] Data generation point in China (Beijing time)

[0130] / region=ch / dt=2022-07-19 07-20 00:00

[0131] / region=ch / dt=2022-07-20 07-21 00:00

[0132] / region=ch / dt=2022-07-21 07-22 00:00

[0133] North American data Beijing time generation point

[0134] / region=na / dt=2022-07-19 07-20 12:00

[0135] / region=na / dt=2022-07-20 07-21 12:00

[0136] / region=na / dt=2022-07-21 07-22 12:00

[0137] This application also includes a cross-regional data synchronization system, comprising: a synchronization client module for submitting data synchronization statements according to the syntax of a data synchronization language; a synchronization engine module for parsing the data synchronization statements and obtaining a synchronization plan, the synchronization plan being suitable for synchronizing data from unit nodes in one or more synchronization regions to a central node; a synchronization executor module for executing the synchronization plan and obtaining execution results; a memory; and a processor. The memory stores instructions executable by the processor; the processor executes these instructions to implement the cross-regional data synchronization method described above.

[0138] In some embodiments, the data synchronization system further includes a synchronization network service module for displaying the synchronization plan and execution results on the front end.

[0139] The foregoing content of this application is intended to explain this cross-regional data synchronization system. (See also...) Figure 3 As shown, the Xensql synchronization client module 310 is a specific implementation of the synchronization client module, the Xensql synchronization engine module 320 is a specific implementation of the synchronization engine module, the Xensql synchronization executor module 330 is a specific implementation of the synchronization executor module, and the Xensql synchronization network service module 340 is a specific implementation of the synchronization network service module. The relevant contents will not be described in detail here.

[0140] Figure 7 This is a system block diagram of a cross-regional data synchronization system according to an embodiment of this application. (Reference) Figure 7As shown, the cross-regional data synchronization system 700 may include an internal communication bus 701, a processor 702, a read-only memory (ROM) 703, a random access memory (RAM) 704, and a communication port 705. When applied to a personal computer, the cross-regional data synchronization system 700 may also include a hard disk 706. The internal communication bus 701 enables data communication between the components of the cross-regional data synchronization system 700. The processor 702 can make judgments and issue prompts. In some embodiments, the processor 702 may consist of one or more processors. The communication port 705 enables data communication between the cross-regional data synchronization system 700 and external systems. In some embodiments, the cross-regional data synchronization system 700 can send and receive information and data from a network through the communication port 705. The cross-regional data synchronization system 700 may also include different forms of program storage units and data storage units, such as the hard disk 706, the read-only memory (ROM) 703, and the random access memory (RAM) 704, capable of storing various data files used for computer processing and / or communication, as well as possible program instructions executed by the processor 702. The processor executes these instructions to implement the main part of the method. The results of the processor's processing are transmitted to the user device via the communication port and displayed on the user interface.

[0141] The aforementioned cross-regional data synchronization method can be implemented as a computer program, stored in the hard disk 706, and loaded into the processor 702 for execution, in order to implement the cross-regional data synchronization method of this application.

[0142] This application also includes a computer-readable medium storing computer program code that, when executed by a processor, implements the cross-regional data synchronization method described above.

[0143] When a cross-regional data synchronization method is implemented as a computer program, it can also be stored as an article of manufacture in a computer-readable storage medium. For example, computer-readable storage media can include, but are not limited to, magnetic storage devices (e.g., hard disks, floppy disks, magnetic stripes), optical discs (e.g., compact discs (CDs), digital multifunction discs (DVDs)), smart cards, and flash memory devices (e.g., electrically erasable programmable read-only memory (EPROM), cards, sticks, key drives). Furthermore, the various storage media described herein can represent one or more devices and / or other machine-readable media used for storing information. The term "machine-readable medium" can include, but is not limited to, wireless channels and various other media (and / or storage media) capable of storing, containing, and / or carrying code and / or instructions and / or data.

[0144] It should be understood that the embodiments described above are merely illustrative. The embodiments described herein may be implemented in hardware, software, firmware, middleware, microcode, or any combination thereof. For hardware implementation, the processor may be implemented within one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field-programmable gate arrays (FPGAs), processors, controllers, microcontrollers, microprocessors, and / or other electronic units designed to perform the functions described herein, or combinations thereof.

[0145] Some aspects of this application can be executed entirely by hardware, entirely by software (including firmware, resident software, microcode, etc.), or by a combination of hardware and software. The aforementioned hardware or software may be referred to as a "data block," "module," "engine," "unit," "component," or "system." The processor may be one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DAPDs), programmable logic devices (PLDs), field-programmable gate arrays (FPGAs), processors, controllers, microcontrollers, microprocessors, or combinations thereof. Furthermore, aspects of this application may manifest as computer products residing in one or more computer-readable media, including computer-readable program code. For example, computer-readable media may include, but are not limited to, magnetic storage devices (e.g., hard disks, floppy disks, magnetic tapes, etc.), optical discs (e.g., compressed CDs, digital multifunction DVDs, etc.), smart cards, and flash memory devices (e.g., cards, sticks, key drives, etc.).

[0146] A computer-readable medium may contain a propagated data signal containing computer program code, for example, on baseband or as part of a carrier wave. This propagated signal may take various forms, including electromagnetic, optical, and so on, or suitable combinations thereof. A computer-readable medium can be any computer-readable medium other than a computer-readable storage medium, which can be connected to an instruction execution system, apparatus, or device to enable communication, propagation, or transmission of a program for use. The program code located on the computer-readable medium can be propagated through any suitable medium, including radio, cable, fiber optic cable, radio frequency signals, or similar media, or any combination of the above media.

[0147] The basic concepts have been described above. Obviously, for those skilled in the art, the above disclosure is merely illustrative and does not constitute a limitation of this application. Although not explicitly stated herein, those skilled in the art may make various modifications, improvements, and corrections to this application. Such modifications, improvements, and corrections are suggested in this application, and therefore remain within the spirit and scope of the exemplary embodiments of this application.

[0148] Furthermore, this application uses specific terms to describe embodiments of the application. For example, "an embodiment," "one embodiment," and / or "some embodiments" refer to a particular feature, structure, or characteristic related to at least one embodiment of the application. Therefore, it should be emphasized and noted that "an embodiment," "one embodiment," or "an alternative embodiment" mentioned twice or more in different locations in this specification do not necessarily refer to the same embodiment. In addition, certain features, structures, or characteristics in one or more embodiments of the application can be appropriately combined.

[0149] In some embodiments, numbers describing the quantity of components and attributes are used. It should be understood that such numbers used in the description of embodiments are modified in some examples with the terms "approximately," "approximately," or "generally." Unless otherwise stated, "approximately," "approximately," or "generally" indicates that the numbers are allowed to vary by ±20%. Accordingly, in some embodiments, the numerical parameters used in the specification and claims are approximate values, which may be changed depending on the characteristics required by individual embodiments. In some embodiments, numerical parameters should take into account specified significant digits and employ a general method of digit reservation. Although the numerical ranges and parameters used to confirm their breadth of scope in some embodiments of this application are approximate values, in specific embodiments, such values ​​are set as precisely as feasible.

Claims

1. A method for cross-region data synchronization, characterized in that, include: Configure the address and directory of the Alluxio data orchestration service; Configure the address of the metadata service and the address of the underlying file system of one or more synchronization zones; Submit a data synchronization statement according to the syntax structure of the data synchronization language. The syntax structure includes configuration items, which include: source data table partition, target data table partition, and view name. Parse the data synchronization statement to obtain a synchronization plan. The synchronization plan is adapted to synchronize the data of the unit nodes of the one or more synchronization regions to the central node. The synchronization plan includes an offline synchronization plan and / or a real-time synchronization plan. The real-time synchronization plan includes the view name. Execute the synchronization plan and obtain the execution result; wherein, The steps for parsing the data synchronization statement include: The data synchronization statement is parsed according to the described syntax structure to obtain the parsing result; After the parsing result is successful, a data synchronization syntax tree is obtained. The data synchronization syntax tree includes the information of the configuration items, the table name of the source data table, and the table name of the target data table. Connect to the metadata service based on the address of the metadata service, and determine whether the target data table exists in the metadata service based on the table name of the target data table; If the target data table exists, then create a mapping directory for the Alluxio data orchestration service corresponding to the source data table; Mount the source table directory in the address of the underlying file system of the one or more synchronization zones to the mapped directory; Create a mapping directory source table, the location of which is the mapping directory; Determine whether to perform real-time synchronization. If not, generate an offline synchronization plan. If real-time synchronization is required, create a view based on the view name and generate a real-time synchronization plan.

2. The data synchronization method of claim 1, wherein, Before the step of submitting data synchronization statements according to the syntax structure of the data synchronization language, the method further includes: the Alluxio data orchestration service is used to mount the underlying file system for access by the upper-layer computing framework and applications.

3. The data synchronization method of claim 2, wherein, Before the step of submitting a data synchronization statement according to the syntax structure of the data synchronization language, the method further includes: configuring one or any one of the following: the execution directory of the computing engine, the communication address of the synchronization engine, the port of the synchronization engine, the communication address of the synchronization executor, the port of the synchronization executor, the address of the synchronization network service, and the port of the synchronization network service.

4. The data synchronization method of claim 3, wherein, The configuration items also include: scheduling cycle, scheduling time zone, computing engine, whether to select local time zone, and one or any item in the list of partitions to be synchronized, wherein the source data table corresponds to the unit node and the target data table corresponds to the central node.

5. The data synchronization method of claim 4, wherein, The steps for parsing the data synchronization statement also include: Based on the information in the configuration items, the address of the Alluxio data orchestration service, and the address of the metadata service, generate a file metadata synchronization plan for the Alluxio data orchestration service and a partition metadata synchronization plan for the metadata service. The computing engine reads the data fields from the source table of the mapping directory and writes the data fields into the target data table; One or more of the offline synchronization plan, the real-time synchronization plan, the file metadata synchronization plan of the Alluxio data orchestration service, and the partition metadata synchronization plan of the metadata service are stored in the database of the central node.

6. The data synchronization method of claim 1, wherein, After parsing the data synchronization statement to obtain the synchronization plan, the process also includes: The synchronization plan is periodically read to determine whether it meets the operating conditions, including the plan execution time zone and the plan execution time. If the operating conditions are met, the synchronization plan will be executed.

7. The data synchronization method as described in claim 1, characterized in that, After executing the synchronization plan and obtaining the execution result, the method further includes: determining whether the execution result is successful; if the execution result is successful, writing the successful execution result to the log; if the execution result is unsuccessful, issuing an alarm and writing the unsuccessful execution result to the log.

8. The data synchronization method of claim 1, wherein, After executing the synchronization plan and obtaining the execution result, the method further includes displaying the synchronization plan and the execution result.

9. The data synchronization method of claim 8, wherein, In the step of displaying the synchronization plan and the execution result, the synchronization plan includes any synchronization plan stored in the database of the central node.

10. A cross-region data synchronization system, comprising: include: The synchronization client module is used to submit data synchronization statements according to the syntax structure of the data synchronization language; The synchronization engine module is used to parse the data synchronization statement and obtain a synchronization plan, which is suitable for synchronizing the data of unit nodes in one or more synchronization areas to the central node; The synchronization executor module is used to execute the synchronization plan and obtain the execution result; Memory is used to store instructions that can be executed by the processor; A processor for executing the instructions to implement the data synchronization method as described in any one of claims 1-9.

11. The data synchronization system of claim 10, wherein, It also includes a synchronization network service module, which is used to display the synchronization plan and the execution results on the front end.

12. A computer readable medium having stored computer program code, characterized in that, The computer program code, when executed by a processor, implements the data synchronization method as described in any one of claims 1-9.

Images