Looking for breakthrough ideas for innovation challenges? Try Patsnap Eureka!

Data synchronization processing method, device and apparatus and storage medium

A technology of data synchronization and processing method, which is applied in electronic digital data processing, special data processing applications, structured data retrieval, etc., and can solve the problems of low conversion and import efficiency, difficult conversion, and cumbersome conversion steps.

Pending Publication Date: 2021-02-19
PINGAN PUHUI ENTERPRISE MANAGEMENT CO LTD
View PDF0 Cites 5 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

[0003] However, the underlying database system HDFS in the existing commonly used big data platforms is very different from traditional relational databases. The data storage format and storage rules are completely different. Even with the help of existing synchronization tools, there are still problems with cumbersome conversion steps and difficult conversion. At the same time, when it comes to massive data conversion, there is also the problem of low conversion and import efficiency

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • Data synchronization processing method, device and apparatus and storage medium
  • Data synchronization processing method, device and apparatus and storage medium
  • Data synchronization processing method, device and apparatus and storage medium

Examples

Experimental program
Comparison scheme
Effect test

Embodiment Construction

[0027] Embodiments of the present invention provide a data synchronization processing method, device, device, and storage medium, which are used to import a large amount of data into a preset distributed file system in batches, so as to improve data conversion and import efficiency.

[0028] The terms "first", "second", "third", "fourth", etc. (if any) in the description and claims of the present invention and the above drawings are used to distinguish similar objects, and not necessarily Used to describe a specific sequence or sequence. It is to be understood that the terms so used are interchangeable under appropriate circumstances such that the embodiments described herein can be practiced in sequences other than those illustrated or described herein. Furthermore, the term "comprising" or "having" and any variations thereof, are intended to cover a non-exclusive inclusion, for example, a process, method, system, product or device comprising a sequence of steps or elements i...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More

PUM

No PUM Login to View More

Abstract

The invention relates to the technical field of big data, and discloses a data synchronization processing method, device and apparatus and a storage medium, which are used for improving data conversion and import efficiency. The data synchronization processing method comprises the following steps: reading a plurality of table structure data in batches from a preset distributed file system according to a data mining request; according to structured query language (SQL) grammar rules, sequentially performing data cleaning and statement splicing on the plurality of table structure data to obtaina plurality of SQL query statements; connecting the target relational database, and performing data query operation according to the plurality of SQL query statements to obtain a query result set; performing data cleaning and format conversion on the query result set to obtain a plurality of distributed file system files; and respectively importing the plurality of distributed file system files into a preset distributed file system to obtain an updating result, and sending the updating result to the target terminal. In addition, the invention also relates to a blockchain technology, and a plurality of table structure data can be stored in the blockchain node.

Description

technical field [0001] The present invention relates to the incremental synchronization field of big data technology, in particular to a data synchronization processing method, device, equipment and storage medium. Background technique [0002] As the application of big data becomes more and more widespread, it is often necessary to extract data from traditional relational databases (such as oracle or mysql), perform operations such as cleaning and assembling, and finally synchronize it to the data warehouse of the big data platform. [0003] However, the underlying database system HDFS in the existing commonly used big data platforms is very different from traditional relational databases. The data storage format and storage rules are completely different. Even with the help of existing synchronization tools, there are still problems with cumbersome conversion steps and difficult conversion. At the same time, when it comes to the conversion of massive data, there is also th...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More

Application Information

Patent Timeline
no application Login to View More
IPC IPC(8): G06F16/178G06F16/182G06F16/245G06F16/27
CPCG06F16/178G06F16/182G06F16/275G06F16/245
Inventor 李悦雯
Owner PINGAN PUHUI ENTERPRISE MANAGEMENT CO LTD
Who we serve
  • R&D Engineer
  • R&D Manager
  • IP Professional
Why Patsnap Eureka
  • Industry Leading Data Capabilities
  • Powerful AI technology
  • Patent DNA Extraction
Social media
Patsnap Eureka Blog
Learn More
PatSnap group products

Browse by: Latest US Patents, China's latest patents, Technical Efficacy Thesaurus, Application Domain, Technology Topic, Popular Technical Reports.

© 2024 PatSnap. All rights reserved.Legal|Privacy policy|Modern Slavery Act Transparency Statement|Sitemap|About US| Contact US: help@patsnap.com