Database log-based real-time data query method and system

A technology of real-time data and query methods, applied in the field of data communication, can solve the problems of database data query delay and time-consuming, and achieve the effect of improving user experience and reducing delay

Active Publication Date: 2017-08-11
BEIJING DIDI INFINITY TECH & DEV
View PDF2 Cites 64 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

[0005] The technical problem to be solved by the present invention is to provide a real-time data query method and system based on database logs to overcome the problems of database data query delay and high time-consuming in the prior art

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • Database log-based real-time data query method and system
  • Database log-based real-time data query method and system
  • Database log-based real-time data query method and system

Examples

Experimental program
Comparison scheme
Effect test

Embodiment Construction

[0053] The following will clearly and completely describe the technical solutions in the embodiments of the present invention with reference to the drawings in the embodiments of the present disclosure. Apparently, the described embodiments are only some of the embodiments of the present invention, not all of them. Based on the embodiments of the present invention, all other embodiments obtained by persons of ordinary skill in the art without making creative efforts belong to the protection scope of the present invention.

[0054]For the data source stored in the Mysql database, in the prior art, data query and analysis takes a long time, and there is a delay in data statistics. Therefore, the technical solution of the present disclosure aims to realize millisecond-level query of data. The kafka involved in this disclosure is a low-latency, high-throughput, distributed publish-subscribe message system, which can handle large-scale real-time data streams. Even very common hardw...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

PUM

No PUM Login to view more

Abstract

The invention relates to a database log-based real-time data query method and system. The method comprises the following steps of: obtaining increment log information of a main database; analyzing the increment log information to obtain increment data information and storing the increment data information in a kafka message queue; reading the increment data information from the kafka message queue through druid, and constructing a real-time query index of the increment data information according to a pre-defined index rule; and carrying out real-time data query by utilizing the real-time query index. Through the database log-based real-time data query method, a binlog log of the main data is used as a data source, the kafka is used as the message queue, and the druid is used as a distributed index to carry out real-time query with a query engine, so that the delay of data collection and data query analysis is reduced, the time consumption of data query is reduced from several minutes to a millisecond level, and then the user experience is enhanced.

Description

technical field [0001] The invention relates to the technical field of data communication, in particular to a database log-based real-time data query method and system. Background technique [0002] Binlog is a database incremental log. The database incremental log is composed of various database table operation events. The location of a database incremental log (for example, mysql-bin.000002, 15846289) corresponds to a database table operation event. Database table operation events include three types: insert operation, update operation and delete operation. Data recovery and data replication can be performed on the database through the database incremental log. In a database table, a column is a collection of data with the same data type. The main incremental data of the database is stored in the column event type. These data can pass the information provided in the table mapping event (such as Table_map_event): column type, column length Wait for it to be parsed correct...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

Application Information

Patent Timeline
no application Login to view more
Patent Type & Authority Applications(China)
IPC IPC(8): G06F17/30
CPCG06F16/2358G06F16/2477G06F16/27
Inventor 艾毅
Owner BEIJING DIDI INFINITY TECH & DEV
Who we serve
  • R&D Engineer
  • R&D Manager
  • IP Professional
Why Eureka
  • Industry Leading Data Capabilities
  • Powerful AI technology
  • Patent DNA Extraction
Social media
Try Eureka
PatSnap group products

Browse by: Latest US Patents, China's latest patents, Technical Efficacy Thesaurus, Application Domain, Technology Topic.

© 2024 PatSnap. All rights reserved.Legal|Privacy policy|Modern Slavery Act Transparency Statement|Sitemap