Data synchronization method and system of heterogeneous platform and related device
A data synchronization and heterogeneous platform technology, applied in structured data retrieval, file system, database update, etc., can solve the problems of high pressure on the data source, low data accuracy, time-consuming and long-term consumption of the synchronization process, etc., to improve The effect of synchronization efficiency
Pending Publication Date: 2020-10-09
上海微盟企业发展有限公司
View PDF5 Cites 11 Cited by
- Summary
- Abstract
- Description
- Claims
- Application Information
AI Technical Summary
Problems solved by technology
[0002] The data synchronization in the big data system mainly uses data synchronization tools such as sqoop and Datax to perform full and incremental timing synchronization. This synchronization method has many disadvantages: the synchronization data delay is high (the timing execution is at the level of minutes or even hours), and the data The accuracy rate is low (incremental synchronization cannot monitor physically deleted data), the pressure on the data source is high (by pulling data, there is a large reading pressure on the source), and the synchronization process takes a long time (especially in the multi-segment table mode when reading and Writing takes a lot of time), and cannot meet the needs of real-time business statistics, etc.
Method used
the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View moreImage
Smart Image Click on the blue labels to locate them in the text.
Smart ImageViewing Examples
Examples
Experimental program
Comparison scheme
Effect test
Embodiment Construction
[0042] In order to make the purposes, technical solutions and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments It is a part of the embodiments of this application, not all of them. Based on the embodiments in this application, all other embodiments obtained by persons of ordinary skill in the art without making creative efforts belong to the scope of protection of this application.
[0043] In order to implement a heterogeneous platform data synchronization method provided by this application, this application defaults to perform at least the following steps before performing data synchronization:
[0044] The first step is to obtain the database change log;
[0045] This step is to obtain the database change log. There is no spe...
the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More PUM
Login to View More Abstract
The invention provides a data synchronization method of a heterogeneous platform. The data synchronization method comprises the steps of receiving a data synchronization request of a destination end;determining corresponding database change information from a preset message queue according to the data synchronization request; converting the database change information from a preset log format anda preset data type into an operation type and a data type corresponding to the destination end; performing message combination on the database change information to obtain final state data; and writing the final state data into the destination end. The problem of data congestion caused by slow data writing due to different formats or different data types during database synchronization is solved,and the database synchronization efficiency is improved. The data synchronization system of the heterogeneous platform, a computer-readable storage medium and a terminal have the above advantages.
Description
technical field [0001] The present application relates to the field of data synchronization, in particular to a data synchronization method, system and related devices for heterogeneous platforms. Background technique [0002] The data synchronization in the big data system mainly uses data synchronization tools such as sqoop and Datax to perform full and incremental timing synchronization. This synchronization method has many disadvantages: the synchronization data delay is high (the timing execution is at the level of minutes or even hours), and the data The accuracy rate is low (incremental synchronization cannot monitor physically deleted data), the pressure on the data source is high (by pulling data, there is a large reading pressure on the source), and the synchronization process takes a long time (especially in the multi-segment table mode when reading and Writing takes a lot of time), and cannot meet the needs of real-time business statistics. [0003] Therefore, h...
Claims
the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More Application Information
Patent Timeline
Login to View More IPC IPC(8): G06F16/18G06F16/23G06F16/2455G06F16/27
CPCG06F16/1815G06F16/2379G06F16/24552G06F16/27
Inventor 秦鹏飞
Owner 上海微盟企业发展有限公司