Method, device and system for data synchronizing, medium and electronic equipment

A data synchronization and data technology, applied in the computer field, can solve problems such as unsynchronized modification of the update time field, data that cannot be extracted, and data table structure consistency that cannot be guaranteed.

A data synchronization and data technology, applied in the computer field, can solve problems such as unsynchronized modification of the update time field, data that cannot be extracted, and data table structure consistency that cannot be guaranteed.

CN108920698AActive Publication Date: 2018-11-30JINGDONG TECH HLDG CO LTD
7 Cites 20 Cited by

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • Method, device and system for data synchronizing, medium and electronic equipment
  • Method, device and system for data synchronizing, medium and electronic equipment
  • Method, device and system for data synchronizing, medium and electronic equipment

Examples

Experimental program
Comparison scheme
Effect test

Embodiment Construction

[0088] Example embodiments will now be described more fully with reference to the accompanying drawings. Example embodiments may, however, be embodied in many forms and should not be construed as limited to the examples set forth herein; rather, these embodiments are provided so that this disclosure will be thorough and complete and fully convey the concept of example embodiments to those skilled in the art.

[0089] Furthermore, the described features, structures, or characteristics may be combined in any suitable manner in one or more embodiments. In the following description, numerous specific details are provided in order to give a thorough understanding of embodiments of the invention. However, those skilled in the art will appreciate that the technical solutions of the present invention may be practiced without one or more of the specific details, or other methods, components, means, steps, etc. may be employed. In other instances, well-known methods, apparatus, implem...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More

PUM

No PUM Login to View More

Abstract

The embodiment of the invention provides a method, a device and a system for data synchronizing, a medium and electronic equipment. The data synchronizing method comprises the following steps of detecting the change of records and table structures of logs in a database, and obtaining the change data of the records and the change data of the table structures of the logs in the database; sending thechange data of the records to a HBase data control write module; sending the change data of the table structures to a Hive table structure synchronizing module. By adopting the technical scheme, thedata synchronizing method provided by the embodiment has the advantage that the problems of inaccuracy in data and inconsistency of data structures are solved by a log analysis real-time duplicating technique, data version control and delay control, so as to reach the effect of complete consistency between the same-day data of a Hive database and the change data of a source database, and meet theT+1 high-precision data analysis and report requirements of the Hive database.

Description

technical field [0001] The present invention relates to the field of computer technology, in particular to a data synchronization method, device, system, medium and electronic equipment. Background technique [0002] With the promotion and popularization of the Internet and big data technologies, data warehouses based on Hadoop and Hive distributed clusters have gradually become mainstream. At present, the data sources of the data warehouse mainly include structured data such as members and orders stored in relational databases, as well as unstructured data stored in the form of files such as user behavior, application logs, images and videos, among which members stored in relational databases Structured data such as orders and orders is the most important part of data warehouse data sources, and it is also the most valuable part of data analysis and mining. [0003] The current method of synchronizing data T+1 from the relational database to the Hive data warehouse is main...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More

Application Information

Patent Timeline
30 Nov 2018
Publication
CN108920698A
IPC
G06F17/30
Inventors
崔迎; 王书坤