Big data synchronizing method and system

A big data and data technology, applied in the field of big data synchronization, can solve problems such as increased workload and undue risks, and achieve the effects of low cost, reduced risk, and reduced new scripts

Active Publication Date: 2016-08-10
北京思特奇信息技术股份有限公司
View PDF7 Cites 9 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

Added a lot of unnecessary workload and undue risks

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • Big data synchronizing method and system
  • Big data synchronizing method and system

Examples

Experimental program
Comparison scheme
Effect test

Embodiment Construction

[0044] The principles and features of the present invention are described below in conjunction with the accompanying drawings, and the examples given are only used to explain the present invention, and are not intended to limit the scope of the present invention.

[0045] figure 1 It is a flow chart of the method for realizing big data synchronization described in the present invention.

[0046] Such as figure 1 As shown, a method for realizing big data synchronization includes the following steps:

[0047] In step S1, the Oracle database sends the table name of the table whose data needs to be synchronized and the field names in the table to the Hadoop cluster.

[0048] Wherein step S1 comprises the following steps:

[0049] In step S11, the Oracle database determines the table name of the table whose data needs to be synchronized.

[0050] In a specific embodiment of the present invention, since not all tables are synchronized, it is only necessary to extract tables that...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

PUM

No PUM Login to view more

Abstract

The invention relates to a big data synchronizing method and a system. The method comprises the following steps: 1) sending the names of the tables and the fields in the tables from an Oracle database in need of synchronizing to a Hadoop cluster; 2) generating a synchronizing table by the Hadoop cluster based on the received names of the tables and the fields in the tables; according to the synchronizing table, loading data in need of synchronizing into the synchronizing table; generating a data file; exporting the data file to a server corresponding the Oracle database; and 3) loading the data contained in the data file on the server by the Oracle database into the Oracle database; According to the invention, a database can be synchronized with Hadoop data in a configuring manner. As long as the names of tables are configured into a database, the tables can be synchronized. In comparison to the prior art, this reduces workload for new scripts and trial tests. Free from new synchronization processes each time, the method and the system achieve high efficiency while reducing costs and risks from failed tests.

Description

technical field [0001] The invention relates to the field of data synchronization, in particular to a method and system for realizing big data synchronization. Background technique [0002] In order to cooperate with the realization of cloudification, one database is split into multiple ones. When the amount of data between the databases is large, there will be bottlenecks when using the real-time interface to interact. For example, the handshake time of the interface is limited. When the data has not been synchronized, the service It has timed out, so for data with large data and low real-time performance, the synchronization adopts the processing method of synchronization in advance, which involves the problem of data synchronization. The synchronized big data information is placed on the hadoop cluster, and each system fetches data from the cluster to the local to realize big data synchronization. In this way, a data synchronization job is added. For each additional tabl...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

Application Information

Patent Timeline
no application Login to view more
IPC IPC(8): H04L29/08G06F17/30
CPCG06F16/10G06F16/2282G06F16/25G06F16/27G06F16/284H04L67/06H04L67/1095H04L67/1097
Inventor 闫浩
Owner 北京思特奇信息技术股份有限公司
Who we serve
  • R&D Engineer
  • R&D Manager
  • IP Professional
Why Eureka
  • Industry Leading Data Capabilities
  • Powerful AI technology
  • Patent DNA Extraction
Social media
Try Eureka
PatSnap group products

Browse by: Latest US Patents, China's latest patents, Technical Efficacy Thesaurus, Application Domain, Technology Topic.

© 2024 PatSnap. All rights reserved.Legal|Privacy policy|Modern Slavery Act Transparency Statement|Sitemap