Supercharge Your Innovation With Domain-Expert AI Agents!

Method and device for synchronizing service data to ODS layer

A technology of business data and ODS, applied in the direction of database model, database update, multi-program device, etc., can solve problems such as delay, impact, and business adverse impact, and achieve the effect of increasing speed and reducing complexity

Pending Publication Date: 2022-04-12
北京发现角科技有限公司
View PDF8 Cites 0 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

[0002] In the traditional offline data warehouse design, the Sqoop tool is used to extract one day or even longer business data from the business database to ODS starting from the specified time. When the amount of data in this process becomes large and there are many sub-tables, it will be huge The delay time reached several hours at one time, which not only made the subsequent layered calculation, index extraction, and index display unable to be executed on time, but also affected the decision-making that needed to use the overall data, and the centralized extraction of business data from online business may also cause business damage. adverse effects

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • Method and device for synchronizing service data to ODS layer
  • Method and device for synchronizing service data to ODS layer
  • Method and device for synchronizing service data to ODS layer

Examples

Experimental program
Comparison scheme
Effect test

Embodiment Construction

[0030] In order to make the purpose, technical solutions and advantages of this application clearer, the following in conjunction with the attached figure 1 , figure 2 And embodiment, this application is described in further detail. It should be understood that the specific embodiments described here are only used to explain the present application, not to limit the present application.

[0031] The embodiment of the present application discloses a method for synchronizing service data to an ODS layer. refer to figure 1 , the method includes:

[0032] Step S100, obtain the incremental data of MySQL's own binlog (binary log) in real time, and store the incremental data in the message queue; the message queue in this embodiment can use a low-latency, high-throughput kafka queue, and the kafka queue The stored data will be backed up for higher reliability.

[0033] Step S200, parse and obtain incremental data stored in the first preset time from the message queue every firs...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More

PUM

No PUM Login to View More

Abstract

The invention relates to a method and device for synchronizing business data to an ODS layer, and the method comprises the steps: obtaining incremental data of a binlog of MySQL in real time, and storing the incremental data to a message queue; analyzing and acquiring incremental data stored in first preset time from the message queue every first preset time, and generating an intermediate file; loading all the intermediate files generated in second preset time to an ODS layer of a data warehouse every second preset time, and aggregating the intermediate files with historical business data of the ODS layer; wherein the second preset time is an integer multiple of the first preset time. According to the invention, the speed of synchronizing the service data to the ODS layer is effectively improved, and the normal work of the online service is not influenced.

Description

technical field [0001] The present application relates to the field of ODS layer data extraction technology, in particular to a method and device for synchronizing business data to the ODS layer. Background technique [0002] In the traditional offline data warehouse design, the Sqoop tool will be used to extract one day or even longer business data from the business database to ODS starting from the specified time. When the amount of data in this process becomes large and there are many sub-tables, the The delay time reached several hours at one time, which not only made the subsequent layered calculation, index extraction, and index display unable to be executed on time, but also affected the decision-making that needed to use the overall data, and the centralized extraction of business data from online business may also cause business damage. adverse effects. Contents of the invention [0003] In order to increase the speed of synchronizing business data to the ODS lay...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More

Application Information

Patent Timeline
no application Login to View More
IPC IPC(8): G06F16/27G06F16/25G06F16/23G06F16/28G06F9/54
Inventor 郭鹏野张振勇
Owner 北京发现角科技有限公司
Features
  • R&D
  • Intellectual Property
  • Life Sciences
  • Materials
  • Tech Scout
Why Patsnap Eureka
  • Unparalleled Data Quality
  • Higher Quality Content
  • 60% Fewer Hallucinations
Social media
Patsnap Eureka Blog
Learn More

Browse by: Latest US Patents, China's latest patents, Technical Efficacy Thesaurus, Application Domain, Technology Topic, Popular Technical Reports.

© 2025 PatSnap. All rights reserved.Legal|Privacy policy|Modern Slavery Act Transparency Statement|Sitemap|About US| Contact US: help@patsnap.com