Real-time service data merging method and device and electronic equipment

A business data, real-time business technology, applied in database indexing, structured data retrieval, special data processing applications, etc., can solve problems such as slow execution speed, high resource consumption, unstable Clickhouse cluster, etc., to reduce data output time , the effect of reducing resource consumption and improving data backtracking efficiency

Pending Publication Date: 2021-06-18
北京淇瑀信息科技有限公司
View PDF0 Cites 4 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

[0004] However, as the amount of data in individual tables in mysql is increasing, more and more data needs to be merged. The original scheme of merging data through ordinary views has many problems such as slow execution speed, high resource consumption, and instability of the Clickhouse cluster. question

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • Real-time service data merging method and device and electronic equipment
  • Real-time service data merging method and device and electronic equipment
  • Real-time service data merging method and device and electronic equipment

Examples

Experimental program
Comparison scheme
Effect test

Embodiment Construction

[0044] Example embodiments will now be described more fully with reference to the accompanying drawings. Example embodiments may, however, be embodied in many forms and should not be construed as limited to the embodiments set forth herein; rather, these embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the concept of example embodiments to those skilled in the art. The same reference numerals denote the same or similar parts in the drawings, and thus their repeated descriptions will be omitted.

[0045] Furthermore, the described features, structures, or characteristics may be combined in any suitable manner in one or more embodiments. In the following description, numerous specific details are provided in order to give a thorough understanding of embodiments of the present disclosure. However, those skilled in the art will appreciate that the technical solutions of the present disclosure may be practiced without one or mo...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

PUM

No PUM Login to view more

Abstract

The invention relates to a real-time service data merging method, device and system, electronic equipment and a computer readable medium. The method comprises the following steps: acquiring a binary log file from a business database in real time, and distinguishing business data through a primary key; analyzing the binary log file to generate business data; writing the service data into a distributed publishing and subscribing message system in real time to generate cluster data; importing the cluster data into a column database management system from the distributed publishing and subscribing message system in a real-time consumption mode based on a primary key; and merging the business data through the column database management system. According to the real-time service data merging method and device, the system electronic equipment and the computer readable medium, on the basis of ensuring the accuracy, timeliness and orderliness of original real-time data backtracking, the data backtracking efficiency can be improved, the data output time can be shortened, and the resource consumption can be reduced.

Description

technical field [0001] The present disclosure relates to the field of computer information processing, in particular, to a real-time service data merging method, device, system electronic equipment and computer readable medium. Background technique [0002] With the development of each company's business, in order to provide better services to users, deal with sudden data, and monitor real-time operation strategies, more and more operators have shifted data analysis work from offline to online real-time deal with. Operators can monitor operational strategies and adjust strategic layouts through real-time and quasi-real-time indicators. [0003] In order to meet this requirement, existing real-time systems often use the Canal (database incremental log analysis) framework to pull binary log file data from the business mysql in real time, and then write it into kafka (distributed publish-subscribe message system) in real time, and then import clickhouse (column database manag...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

Application Information

Patent Timeline
no application Login to view more
Patent Type & Authority Applications(China)
IPC IPC(8): G06F16/22G06F16/27G06F16/17
CPCG06F16/1734G06F16/221G06F16/2255G06F16/2282G06F16/27
Inventor 田刚
Owner 北京淇瑀信息科技有限公司
Who we serve
  • R&D Engineer
  • R&D Manager
  • IP Professional
Why Eureka
  • Industry Leading Data Capabilities
  • Powerful AI technology
  • Patent DNA Extraction
Social media
Try Eureka
PatSnap group products

Browse by: Latest US Patents, China's latest patents, Technical Efficacy Thesaurus, Application Domain, Technology Topic.

© 2024 PatSnap. All rights reserved.Legal|Privacy policy|Modern Slavery Act Transparency Statement|Sitemap