Database log synchronization method and device, computer equipment and readable storage medium

A synchronization device and database technology, applied in the field of data processing, can solve problems such as complicated binlog log process

Pending Publication Date: 2020-04-14
CHINA UNITECHS
View PDF3 Cites 3 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

[0004] The embodiment of the present invention provides a method for synchronizing database logs to solve the technical problem of complicated processes in synchronizing binlog logs in the prior art

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • Database log synchronization method and device, computer equipment and readable storage medium
  • Database log synchronization method and device, computer equipment and readable storage medium
  • Database log synchronization method and device, computer equipment and readable storage medium

Examples

Experimental program
Comparison scheme
Effect test

Embodiment Construction

[0021] In order to make the object, technical solution and advantages of the present invention clearer, the present invention will be described in further detail below in conjunction with the embodiments and accompanying drawings. Here, the exemplary embodiments and descriptions of the present invention are used to explain the present invention, but not to limit the present invention.

[0022] In an embodiment of the present invention, a method for synchronizing database logs is provided, which can run in a processing engine, such as figure 2 As shown, the method includes:

[0023] Step 202: Simulate the slave node of MySQL through the fake slave node synchronization service, and extract the binlog log from MySQL;

[0024] Step 204: store the extracted binlog log in the storage module of the processing engine through the pseudo slave node synchronization service;

[0025] Step 206: Read and parse the binlog log from the storage module through the parsing service, and save t...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

PUM

No PUM Login to view more

Abstract

The embodiment of the invention provides a database log synchronization method and device, computer equipment and a readable storage medium. The method comprises the steps of simulating a slave node of a MySQL through a pseudo slave node synchronization service, and extracting a binlog from the MySQL; storing the extracted binlog in a storage module of a processing engine through the pseudo slavenode synchronization service; and reading the binlog log from the storage module through an analysis service, analyzing the binlog log, and storing an analysis result into the target storage. According to the scheme, the whole binlog synchronization process of the MySQL can be completed in the processing engine through a program; extra components such as a third-party synchronization tool, a message queue or JMQ are prevented from being introduced, and meanwhile, multi-party component cooperation can be avoided in the binlog synchronization process of the MySQL; and the processing flow can besimplified.

Description

technical field [0001] The invention relates to the technical field of data processing, in particular to a database log synchronization method, device, computer equipment and readable storage medium. Background technique [0002] Real-time data analysis shows that directly querying MySQL (a database) table or frequently extracting data will put a lot of performance pressure on MySQL. Therefore, the current method is to synchronize the MySQL binlog log, parse the binlog and store it in the target storage. Avoid performance pressure on MySQL. [0003] For example, the conventional binlog log synchronization process, such as figure 1 As shown, you need to use a synchronization tool (such as Maxwall or Canal) to read Binlog, and then synchronize Binlog to middleware such as message queue or JMQ (Java Message Queue) to cache data, and data processing engines (such as Flink, Spark and other big data Computing engine) consumes the Binlog of the message queue, and parses the Binlo...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

Application Information

Patent Timeline
no application Login to view more
Patent Type & Authority Applications(China)
IPC IPC(8): G06F16/27G06F16/28G06F9/54
CPCG06F16/27G06F16/284G06F9/546G06F2209/548G06F2209/547
Inventor 周朝卫
Owner CHINA UNITECHS
Who we serve
  • R&D Engineer
  • R&D Manager
  • IP Professional
Why Eureka
  • Industry Leading Data Capabilities
  • Powerful AI technology
  • Patent DNA Extraction
Social media
Try Eureka
PatSnap group products

Browse by: Latest US Patents, China's latest patents, Technical Efficacy Thesaurus, Application Domain, Technology Topic.

© 2024 PatSnap. All rights reserved.Legal|Privacy policy|Modern Slavery Act Transparency Statement|Sitemap