Synchronization system for automatically analyzing off-line data

Abstract

The invention discloses a synchronization system for automatically analyzing off-line data. The synchronization system is characterized in that an analysis adapter enters a relational database to access metadata of a to-be-synchronized off-line data table, a data growth model is matched, synchronous scheduling configuration parameters of the to-be-synchronized off-line data table are generated, and the synchronous scheduling configuration parameters are transmitted into a logic processor; the logic processor generates an adaptation table structure table building statement and a synchronization code or script of the to-be-synchronized offline data table in the corresponding big data platform according to the synchronous scheduling configuration parameters, and sets a synchronization scheme of the to-be-synchronized offline data table in the logic processor; the executor is connected with the big data platform, and selects a proper synchronization scheme for the generated adaptation table structure table building statement to execute an offline data table synchronization task in the big data platform; and the task scheduler performs classification management on the offline data table synchronization tasks and records execution states and execution results. According to the method and the device, synchronization tasks of various offline data tables are realized, and the offline data synchronization efficiency is greatly improved.

Description

Technical Field

[0001] The present invention relates to the technical field of data synchronization, and in particular to a synchronization system for automatically parsing offline data. Background Art

[0002] Data synchronization technology involves big data processing, ETL process, data warehouse design and other aspects. It is a key link in data engineering and provides a high-quality data foundation for subsequent data analysis and decision-making. Therefore, it is necessary to design an efficient, accurate and stable data synchronization solution.

[0003] At present, it is a large and challenging task to integrate and synchronize offline data in relational databases into big data platforms and select appropriate synchronization solutions based on the characteristics of different offline data. Traditional large-scale offline data synchronization requires data developers and business personnel who are more familiar with data tables to assist in jointly determining offline data synchronization solutions and task scheduling logic. During the synchronization process, data developers manually write the structure statements, data synchronization codes or scripts, and scheduling tasks for each database table to be synchronized, and then the operation and maintenance personnel execute and publish the code program in the corresponding environment, and then perform scheduling and monitoring of offline data tasks. This offline data synchronization process requires a synchronization solution to be designed for each offline data synchronization task, which requires a large amount of code, is prone to errors, and has low synchronization efficiency. Summary of the invention

[0004] In view of the problems existing in the prior art, the present invention provides a synchronization system for automatically parsing offline data, which can realize synchronization tasks of multiple offline data tables and greatly improve the efficiency of offline data synchronization.

[0005] To achieve the above technical objectives, the present invention adopts the following technical solutions: a synchronization system for automatically parsing offline data, comprising: a parsing adapter, a logical processor, an executor and a task scheduler;

[0006] The parsing adapter accesses the metadata of the offline data table to be synchronized in the relational database, matches the data growth model, generates synchronization scheduling configuration parameters of the offline data table to be synchronized, and transmits them to the logical processor;

[0007] The logic processor generates a table creation statement and a synchronization code or script for the adaptation table structure of the offline data table to be synchronized in the corresponding big data platform according to the synchronization scheduling configuration parameters, and sets a synchronization scheme for the offline data table to be synchronized in the logic processor;

[0008] The executor is connected to the big data platform, and selects a suitable synchronization solution for the generated adaptation table structure table creation statement to deploy or execute the offline data table synchronization task in the big data platform;

[0009] The task scheduler performs classification management on offline data table synchronization tasks and records execution status and execution results.

[0010] Furthermore, the specific process of the parsing adapter generating the synchronization scheduling configuration parameters of the offline data table is as follows: the parsing adapter enters the relational database to access the metadata of the offline data table to be synchronized, identifies the timestamp according to the field type in the metadata, builds data query logic through the timestamp, matches the data growth model, obtains the recommended parameters and key fields of the offline data table to be synchronized under the corresponding data growth model, and uses the recommended parameters and key fields of the offline data table to be synchronized as the synchronization scheduling configuration parameters of the offline data table to be synchronized.

[0011] Furthermore, the data growth model includes: a linear growth model, an exponential growth model, a cyclical growth model and an irregular growth model.

[0012] Furthermore, the recommended parameters of the offline data table to be synchronized include: scheduling frequency cycle, memory resources allocated for task execution, partition fields, whether to create a history table and data storage duration.

[0013] Furthermore, the synchronization scheme of the offline data table to be synchronized includes: full synchronization of the offline data table to be synchronized, incremental synchronization of the offline data table to be synchronized, and incremental synchronization of the offline data table to be synchronized.

[0014] Furthermore, for the synchronization scheme of the offline data table to be synchronized that matches the linear growth model, a full synchronization scheme or an incremental synchronization scheme of the offline data table to be synchronized is selected according to the data growth rate and current data volume in the offline data table to be synchronized.

[0015] Furthermore, for the synchronization scheme of the offline data table to be synchronized matching the exponential growth model, an incremental synchronization scheme or an incremental variable synchronization scheme of the offline data table to be synchronized is selected according to the data growth rate and current data volume in the offline data table to be synchronized.

[0016] Furthermore, for the synchronization scheme of the offline data table to be synchronized that matches the periodic growth model, a full synchronization scheme or an incremental synchronization scheme of the offline data table to be synchronized is selected according to the data growth rate, current data volume and peak and valley values ​​in the offline data table to be synchronized.

[0017] Furthermore, for the synchronization scheme of the offline data table to be synchronized matching the irregular growth model, a full synchronization scheme or an incremental synchronization scheme of the offline data table to be synchronized is selected according to the current data volume in the offline data table to be synchronized.

[0018] Compared with the prior art, the present invention has the following beneficial effects: the synchronization system for automatically parsing offline data of the present invention realizes intelligent analysis and automated development of offline data tables to be synchronized from relational databases to big data platforms through the configuration of parsing adapters, logical processors, executors and task schedulers, through unified data synchronization logic, thereby improving the efficiency and quality of data synchronization, reducing manpower investment and code error rate, lowering the user's usage threshold, and meeting storage requirements in different data scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] Figure 1 A schematic diagram of a synchronization system for automatically parsing offline data according to the present invention;

[0020] Figure 2 A schematic diagram of the synchronization scheduling configuration parameters for generating the offline data table to be synchronized by the parsing adapter in the present invention;

[0021] Figure 3 It is a schematic diagram of the specific structure of the logic processor in the present invention. DETAILED DESCRIPTION

[0022] The technical solution of the present invention is further explained below in conjunction with the accompanying drawings.

[0023] like Figure 1 The present invention is a schematic diagram of a synchronization system for automatically parsing offline data, the synchronization system comprising: a parsing adapter, a logical processor, an executor and a task scheduler; the parsing adapter enters a relational database to access metadata of an offline data table to be synchronized, matches a data growth model, generates synchronization scheduling configuration parameters of the offline data table to be synchronized, and transmits the parameters to the logical processor; the logical processor generates an adaptation table structure table creation statement and a synchronization code or script for the offline data table to be synchronized on a corresponding big data platform according to the synchronization scheduling configuration parameters, and sets a synchronization scheme for the offline data table to be synchronized in the logical processor, transforms a manual code development mode of manually identifying rules into a program automatically identifying rules to generate corresponding standard codes, improves code accuracy, and reduces human cost investment; the executor is connected to the big data platform, selects a suitable synchronization scheme for the generated adaptation table structure table creation statement, and executes the offline data table synchronization task in the big data platform; the task scheduler classifies and manages the offline data table synchronization tasks, and records the execution status and execution results. The synchronization system for automatically parsing offline data of the present invention realizes intelligent analysis and automated development of offline data tables to be synchronized from relational databases to big data platforms through the configuration of parsing adapters, logical processors, executors and task schedulers, through unified data synchronization logic, thereby improving the efficiency and quality of data synchronization, reducing manpower investment and code error rate, lowering the user's usage threshold, and meeting storage requirements in different data scenarios.

[0024] like Figure 2 , the specific process of the parsing adapter of the present invention to generate the synchronization scheduling configuration parameters of the offline data table is: the parsing adapter enters the relational database to access the metadata of the offline data table to be synchronized, identifies the timestamp according to the field type in the metadata, builds the data query logic sql through the timestamp, analyzes the data growth characteristics of the source table, matches the data growth model, obtains the recommended parameters and key fields of the offline data table to be synchronized under the corresponding data growth model, and uses the recommended parameters and key fields of the offline data table to be synchronized as the synchronization scheduling configuration parameters of the offline data table to be synchronized. Among them, the data growth model includes: linear growth model, exponential growth model, periodic growth model and irregular growth model; the recommended parameters of the offline data table to be synchronized include: scheduling frequency cycle, memory resources allocated for task execution, partition fields, whether to create a historical table and data storage duration. The present invention formulates reasonable task scheduling configuration suggestions through the parsing adapter, which can optimize the resource balance of the server to a certain extent.

[0025] like Figure 3 In the present invention, the synchronization scheme of the offline data table to be synchronized includes: full synchronization of the offline data table to be synchronized, incremental synchronization of the offline data table to be synchronized, and variable synchronization of the offline data table to be synchronized. Among them, for the offline data table to be synchronized with a small data volume and a slow data growth rate, the whole table can be synchronized each time, and the task frequency of synchronizing data can be set to be low; for the offline data table to be synchronized with a large data volume and a fast growth rate, and the data is only added but not modified or the data modification has little impact on the business, incremental synchronization can be performed, and the task frequency of synchronizing data is usually set to be high; for the offline data table to be synchronized, the new data will be modified and have a relatively large impact on the business. The data is processed later. The latest state of the data is required for big data processing, and the data volume is very large. In order to save resources and maintain performance at the same time, variable synchronization is selected, and the task frequency of synchronizing data is usually set to be high.

[0026] For the synchronization scheme of offline data tables to be synchronized that match the linear growth model, since the data growth cycle is relatively stable, the full synchronization scheme or the incremental synchronization scheme of offline data tables to be synchronized is selected according to the data growth rate and current data volume in the offline data tables to be synchronized. The data synchronization cycle is relatively stable and the memory resource allocation is relatively fixed.

[0027] For the synchronization scheme of synchronized offline data tables matching the exponential growth model, since the data growth rate accelerates with the increase of time or other variables, an upward convex curve is formed. This kind of data growth trend means that the data is growing rapidly. To prevent excessive data growth, the original database will perform regular backup and rewrite operations. According to the data growth rate and current data volume in the offline data table to be synchronized, the incremental synchronization scheme or the variable-increment synchronization scheme of the offline data table to be synchronized is selected. Among them, the incremental synchronization scheme adopts a relatively high-frequency, low-memory resource scheduling scheme for offline data tables to be synchronized with normal data growth volume and fast growth rate; the variable-increment synchronization scheme adopts a relatively high-frequency, high-memory resource scheduling scheme for offline data tables to be synchronized with large data volume and fast growth rate.

[0028] For the synchronization solution of the offline data table that matches the periodic growth model, since the data will have a curve growth process such as peak data according to time or other key fields, based on the data analysis of the offline data table to be synchronized, the time period of data surge and data slow growth can be obtained. According to the data growth rate, current data volume and peak and valley values ​​in the offline data table to be synchronized, the full synchronization solution or the incremental synchronization solution of the offline data table to be synchronized is selected. For the incremental synchronization solution, a scheduling solution with dynamic allocation of memory resources and relatively fixed synchronization frequency is generally adopted.

[0029] For the synchronization plan of offline data tables that match the irregular growth model, the full synchronization plan or the incremental synchronization plan of the offline data tables to be synchronized is selected according to the current data volume in the offline data tables to be synchronized. In terms of memory resources and scheduling frequency, scheduling is generally performed according to the default values ​​corresponding to the data volume.

[0030] The synchronization system for automatically parsing offline data of the present invention parses the metadata of the offline data table to be synchronized in the relational database through the parsing adapter, generates the synchronization scheduling configuration parameters of the offline data table to be synchronized, and generates the table building statement of the adaptation table structure of the offline data table to be synchronized in the corresponding big data platform in the logical processor, as well as the synchronization code or script and the synchronization scheme of the offline data table to be synchronized, connects to the physical server through the executor and the scheduler, executes the offline data table synchronization task, and performs configuration scheduling, reasonably arranges the scheduling frequency, and intelligently manages and controls memory resources. The present invention is suitable for automatic synchronization of multi-mode offline data, can effectively improve the synchronization efficiency of relational database data to the big data platform, and has broad application scenarios in the field of data engineering. Therefore, the synchronization system for automatically parsing offline data of the present invention makes the synchronization process of the offline data table to be synchronized from the relational database to the big data platform more flexible, efficient and reliable, reduces the development cost and technical requirements, and improves the quality and efficiency of data synchronization.

[0031] The above are only preferred embodiments of the present invention, and the protection scope of the present invention is not limited to the above embodiments. All technical solutions under the concept of the present invention belong to the protection scope of the present invention. It should be pointed out that for ordinary technicians in this technical field, some improvements and modifications without departing from the principle of the present invention should be regarded as the protection scope of the present invention.

Claims

1. A synchronization system for automatically parsing offline data, characterized in that: include: Parsing adapters, logical processors, executors, and task schedulers; The parsing adapter accesses the metadata of the offline data table to be synchronized in the relational database, matches the data growth model, generates synchronization scheduling configuration parameters of the offline data table to be synchronized, and transmits them to the logical processor; The logic processor generates a table creation statement and a synchronization code or script for the adaptation table structure of the offline data table to be synchronized in the corresponding big data platform according to the synchronization scheduling configuration parameters, and sets a synchronization scheme for the offline data table to be synchronized in the logic processor; The executor is connected to the big data platform, and selects a suitable synchronization solution for the generated adaptation table structure table creation statement to deploy or execute the offline data table synchronization task in the big data platform; The task scheduler performs classification management on offline data table synchronization tasks and records execution status and execution results.

2. A synchronization system for automatically parsing offline data according to claim 1, characterized in that: The specific process of the parsing adapter generating the synchronization scheduling configuration parameters of the offline data table is as follows: the parsing adapter enters the relational database to access the metadata of the offline data table to be synchronized, identifies the timestamp according to the field type in the metadata, builds data query logic through the timestamp, matches the data growth model, obtains the recommended parameters and key fields of the offline data table to be synchronized under the corresponding data growth model, and uses the recommended parameters and key fields of the offline data table to be synchronized as the synchronization scheduling configuration parameters of the offline data table to be synchronized.

3. The synchronization system for automatically parsing offline data according to claim 2, characterized in that: The data growth models include: linear growth model, exponential growth model, cyclical growth model and irregular growth model.

4. The synchronization system for automatically parsing offline data according to claim 3, characterized in that: The recommended parameters of the offline data table to be synchronized include: scheduling frequency cycle, memory resources allocated for task execution, partition fields, whether to create a history table and data storage duration.

5. A synchronization system for automatically parsing offline data according to claim 4, characterized in that: The synchronization scheme of the offline data table to be synchronized includes: full synchronization of the offline data table to be synchronized, incremental synchronization of the offline data table to be synchronized, and incremental synchronization of the offline data table to be synchronized.

6. A synchronization system for automatically parsing offline data according to claim 5, characterized in that: For the synchronization scheme of the offline data table to be synchronized that matches the linear growth model, a full synchronization scheme or an incremental synchronization scheme of the offline data table to be synchronized is selected according to the data growth rate and current data volume in the offline data table to be synchronized.

7. The synchronization system for automatically parsing offline data according to claim 5, characterized in that: For the synchronization scheme of the offline data table to be synchronized that matches the exponential growth model, an incremental synchronization scheme or an incremental variable synchronization scheme of the offline data table to be synchronized is selected according to the data growth rate and current data volume in the offline data table to be synchronized.

8. The synchronization system for automatically parsing offline data according to claim 5, characterized in that: For the synchronization scheme of the offline data table to be synchronized that matches the periodic growth model, a full synchronization scheme or an incremental synchronization scheme of the offline data table to be synchronized is selected according to the data growth rate, current data volume, and peak and valley values ​​in the offline data table to be synchronized.

9. The synchronization system for automatically parsing offline data according to claim 5, characterized in that: For the synchronization scheme for synchronizing offline data tables that matches the irregular growth model, a full synchronization scheme or an incremental synchronization scheme for the offline data tables to be synchronized is selected according to the current data volume in the offline data tables to be synchronized.

Images