Method and device for synchronizing service data

A technology of business data and data, applied in database indexing, structured data retrieval, electronic digital data processing, etc., can solve problems such as low latency, and achieve the effects of low latency, no development workload, and great flexibility

Pending Publication Date: 2021-02-26
BEIJING UNISOUND INFORMATION TECH +1
View PDF0 Cites 3 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

[0008] The present invention provides a method and device for synchronizing business data, which can solve existing technic

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • Method and device for synchronizing service data
  • Method and device for synchronizing service data
  • Method and device for synchronizing service data

Examples

Experimental program
Comparison scheme
Effect test

Embodiment 1

[0048] A method for synchronizing business data provided by an embodiment of the present invention, such as figure 1 shown, including the following steps:

[0049] Step 101, synchronizing the Mysql data as business data to the Kafka cluster through the Maxwell component;

[0050] Step 102, synchronizing the data in the Kafka cluster to the HBase;

[0051] Step 103, map the table stored in the HBase to the Hive database, and complete the synchronization of the business data.

[0052] In a specific implementation, the "synchronize Mysql data as business data to the Kafka cluster through Maxwell" in step 101 includes:

[0053] Use the Maxwell component to pull the binlog log of Mysql in real time;

[0054] The binlog log is parsed into JSON format by the Maxwell component, and the data in JSON format is sent to the topic in the Kafka cluster.

[0055]Further, before "Using Maxwell components to pull Mysql binlog logs in real time", it also includes:

[0056] Enable the binlo...

Embodiment 2

[0121] Embodiment 2 of the present invention also discloses a device for synchronizing business data, such as Figure 4 shown, including:

[0122] The first synchronization module 201 is used to synchronize Mysql data as business data to the Kafka cluster through the Maxwell component;

[0123] A second synchronization module 202, configured to synchronize data in the Kafka cluster to the HBase;

[0124] The mapping module 203 is configured to map the tables stored in the HBase to the Hive database to complete the synchronization of the business data.

[0125] In a specific embodiment, the first synchronization module 201 is configured to:

[0126] Use the Maxwell component to pull the binlog log of Mysql in real time;

[0127] The binlog log is parsed into JSON format by the Maxwell component, and the data in JSON format is sent to the topic in the Kafka cluster.

[0128] In a specific embodiment, the first synchronization module 201 is also used for:

...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

PUM

No PUM Login to view more

Abstract

The invention relates to a method and device for synchronizing service data. The method comprises the steps of: synchronizing Mysql data serving as the service data to a Kafka cluster through a Maxwell assembly; synchronizing the data in the Kafka cluster to the HBase; and mapping the table stored in the HBase to a Hive library to finish the synchronization of the service data. According to the scheme of the invention, a Maxwell synchronization mode is adopted, no influence is caused to the running service, the delay is lower, more concurrency can be supported, and the flexibility is better.

Description

technical field [0001] The invention relates to the technical field of data synchronization, in particular to a method and equipment for synchronizing service data. Background technique [0002] At present, big data is widely used to provide dimension table data support for big data in building various models and real-time analysis scenarios. Dimension table data is various business data stored in a relational database synchronously in real time. When big data is modeled using models such as stars and constellations, the dimension table data will use the business data stored in the relational mysql database table by the business department, and the business data has a change characteristic, so there is a problem in the existing technology The following synchronization methods: [0003] Master-slave synchronization supported by relational mysql database; [0004] Full or incremental synchronization using the API interface; [0005] However, the following problems exist in ...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

Application Information

Patent Timeline
no application Login to view more
IPC IPC(8): G06F16/22G06F16/25G06F16/27G06F8/41
CPCG06F8/427G06F16/2282G06F16/25G06F16/27
Inventor 刘成
Owner BEIJING UNISOUND INFORMATION TECH
Who we serve
  • R&D Engineer
  • R&D Manager
  • IP Professional
Why Eureka
  • Industry Leading Data Capabilities
  • Powerful AI technology
  • Patent DNA Extraction
Social media
Try Eureka
PatSnap group products

Browse by: Latest US Patents, China's latest patents, Technical Efficacy Thesaurus, Application Domain, Technology Topic.

© 2024 PatSnap. All rights reserved.Legal|Privacy policy|Modern Slavery Act Transparency Statement|Sitemap