Looking for breakthrough ideas for innovation challenges? Try Patsnap Eureka!

A method for updating a relational database to a non-relational database based on an ETL tool

A non-relational, database technology, applied in the field of updating from a relational database to a non-relational database based on ETL tools, can solve problems such as timed incremental update, save time for finding errors, portability and reusability Highly configurable and highly configurable effects

Inactive Publication Date: 2019-01-18
ANHUI SUN CREATE ELECTRONICS
View PDF3 Cites 7 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

For example, if you want to synchronize data from different existing data sources to the cloud platform, you will face the problem of timed incremental update of data from relational databases to non-relational databases.

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • A method for updating a relational database to a non-relational database based on an ETL tool

Examples

Experimental program
Comparison scheme
Effect test

Embodiment 1

[0038] 1. Data collection, collect the target data of each platform through the collector provided by the ETL tool. First, you need to configure various collectors provided by the ETL tool, including collector configurations for data sources such as MySQL, Kafka, ElasticSearch, HBase, local files, and web pages. Then use the collector to normalize the heterogeneous data according to the set rules, and store it in the relational database.

[0039] 2. ExecuteSQL, fetch data from MySQL with SQL command. Configuration, the delay period in the settings is set to 2sec, the production period is set to 5sec; the current task in the execution plan is set to 1sec, and the running scheduled task is set to 10sec; the Database Connection Pooling Service in the property represents the connected database address, which can be set to MySQL: yourIPAddress:3306port / dataBaseName, the specific SQL statement of SQL select query, can be set to SELECT*FROM TABLENAME; Max Wait Time is set to 0sec; N...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More

PUM

No PUM Login to View More

Abstract

The invention relates to a method for updating a relational database to a non-relational database based on an ETL tool. The method comprises the following steps: collecting source data of each platform through a collector of the ETL tool, thereby obtaining target data; updating a relational database to a non-relational database. Storing the collected target data into a relational database; extracting target data from a relational database as data to be processed, and taking the data to be processed as intermediate state data in a format of binary Avro data; converting the data to be processedin binary Avro data format into the data to be processed in JSON format; the data to be processed in JSON format is temporarily stored in the local disk in the form of FlowFile stream data to obtain the temporarily stored data to be processed; the temporary data to be processed is stored in the non-relational database through the ETL tool. The method of the invention has the advantages of high configurability and strong process controllability, can accurately locate the link where the error occurs, and saves the time for searching the error.

Description

technical field [0001] The invention relates to the technical field of computer application, in particular to a method for updating from a relational database to a non-relational database based on an ETL tool. Background technique [0002] With the continuous development of big data technology, big data technology has become an important pillar supporting the development of information technology in various industries. The current society is stepping into the data era in an all-round way. There are tens of thousands of data generated every moment, and there are various formats. Traditional data storage basically uses relational databases. With the development of the Internet, With the development, the scale and format of data have undergone significant changes. This traditional model can no longer meet the current needs. On the basis of relational databases, non-relational databases reduce the correlation between data and are more suitable for data diversification. It is a ...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More

Application Information

Patent Timeline
no application Login to View More
Patent Type & Authority Applications(China)
IPC IPC(8): G06F16/25G06F16/83
Inventor 金文林范联伟余保华徐圣吉展昭张金国陈钊邓惠元张国林胡刚潘兆轩李伟李鑫孙海龙吴振耀
Owner ANHUI SUN CREATE ELECTRONICS
Who we serve
  • R&D Engineer
  • R&D Manager
  • IP Professional
Why Patsnap Eureka
  • Industry Leading Data Capabilities
  • Powerful AI technology
  • Patent DNA Extraction
Social media
Patsnap Eureka Blog
Learn More
PatSnap group products

Browse by: Latest US Patents, China's latest patents, Technical Efficacy Thesaurus, Application Domain, Technology Topic, Popular Technical Reports.

© 2024 PatSnap. All rights reserved.Legal|Privacy policy|Modern Slavery Act Transparency Statement|Sitemap|About US| Contact US: help@patsnap.com