Looking for breakthrough ideas for innovation challenges? Try Patsnap Eureka!

A data acquisition method and device based on spark computing framework

A computing framework and data acquisition technology, applied in the computer field, can solve problems such as insufficient performance of JdbcRDD functions, affecting spark data import performance, etc., to achieve the effect of improving data import performance, reducing data transmission overhead, and improving parallelism

Active Publication Date: 2022-04-29
NAT COMP NETWORK & INFORMATION SECURITY MANAGEMENT CENT +1
View PDF3 Cites 0 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

[0006] The technical problem to be solved by the present invention is a data acquisition method and device based on the Spark computing framework, which is used to solve the problem in the prior art that the data import performance of spark is affected due to the lack of JdbcRDD function and performance

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • A data acquisition method and device based on spark computing framework
  • A data acquisition method and device based on spark computing framework
  • A data acquisition method and device based on spark computing framework

Examples

Experimental program
Comparison scheme
Effect test

Embodiment Construction

[0023] The present invention will be described in further detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described here are only used to explain the present invention, not to limit the present invention.

[0024] This embodiment provides a data acquisition method based on the Spark computing framework. This embodiment is executed on the Spark side. Such as figure 1 Shown is a flowchart of a data acquisition method based on the Spark computing framework according to an embodiment of the present invention.

[0025] Step S110, after receiving the table object access request, obtain the computing resource information of Spark and the data distribution information of the data tables to be accessed in the MPP cluster (also called MPP database cluster).

[0026] The table object access request is used to request to access the data tables stored in the MPP cluster. According to the table object acce...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More

PUM

No PUM Login to View More

Abstract

The invention discloses a data acquisition method and device based on a Spark computing framework. The method includes: after receiving the table object access request, obtaining the computing resource information of Spark and the data distribution information of the data table to be accessed in the MPP cluster; generating multiple Partitions according to the computing resource information and data distribution information; wherein, each A Partition corresponds to part of the data in the data table; through the generated multiple Partitions, the data table is obtained from the MPP cluster. The present invention makes full use of the data storage characteristics of the MPP cluster, and directly and quickly acquires data sets from the storage nodes of the MPP through multiple Partitions. Furthermore, in the case of sufficient computing resources, the data table of the storage node can be further split to achieve the purpose of increasing parallelism and improving data import performance. According to the data distribution of the MPP cluster, data can be obtained from local storage first, reducing data transmission overhead, saving network bandwidth, reducing network delay, and improving computing performance.

Description

technical field [0001] The present invention relates to the field of computer technology, in particular to a data acquisition method and device based on a Spark computing framework. Background technique [0002] Apache Spark (Spark for short) is a fast and general computing engine designed for large-scale data processing. Spark natively provides access interfaces to file systems including HDFS (Hadoop Distributed File System, Distributed File System), and Spark can usually be used to calculate and analyze structured data in the database. [0003] Specifically, Spark can export the data of the target database to a file through the database export tool, and then calculate and analyze the data through the file, but this method is cumbersome, time-consuming, error-prone, and is largely affected by the environment. limit. [0004] In addition, you can also directly access the database system through the JdbcRDD provided by Spark. JdbcRDD is a general-purpose database access in...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More

Application Information

Patent Timeline
no application Login to View More
Patent Type & Authority Patents(China)
IPC IPC(8): G06F16/25G06F16/22
Inventor 吕雁飞刘欣然张鸿蒋旭马秉楠惠榛朱亚南
Owner NAT COMP NETWORK & INFORMATION SECURITY MANAGEMENT CENT
Who we serve
  • R&D Engineer
  • R&D Manager
  • IP Professional
Why Patsnap Eureka
  • Industry Leading Data Capabilities
  • Powerful AI technology
  • Patent DNA Extraction
Social media
Patsnap Eureka Blog
Learn More
PatSnap group products

Browse by: Latest US Patents, China's latest patents, Technical Efficacy Thesaurus, Application Domain, Technology Topic, Popular Technical Reports.

© 2024 PatSnap. All rights reserved.Legal|Privacy policy|Modern Slavery Act Transparency Statement|Sitemap|About US| Contact US: help@patsnap.com