A data acquisition method and device based on spark computing framework

A computing framework and data acquisition technology, applied in the computer field, can solve problems such as insufficient performance of JdbcRDD functions, affecting spark data import performance, etc., to achieve the effect of improving data import performance, reducing data transmission overhead, and improving parallelism

CN108536808BActive Publication Date: 2022-04-29NAT COMP NETWORK & INFORMATION SECURITY MANAGEMENT CENT +1
3 Cites 0 Cited by

Patent Information

Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Publication Date
2022-04-29

Smart Images

  • Figure 1
    Figure 1
  • Figure 2
    Figure 2
  • Figure 3
    Figure 3
Patent Text Reader

Abstract

The invention discloses a data acquisition method and device based on a Spark computing framework. The method includes: after receiving the table object access request, obtaining the computing resource information of Spark and the data distribution information of the data table to be accessed in the MPP cluster; generating multiple Partitions according to the computing resource information and data distribution information; wherein, each A Partition corresponds to part of the data in the data table; through the generated multiple Partitions, the data table is obtained from the MPP cluster. The present invention makes full use of the data storage characteristics of the MPP cluster, and directly and quickly acquires data sets from the storage nodes of the MPP through multiple Partitions. Furthermore, in the case of sufficient computing resources, the data table of the storage node can be further split to achieve the purpose of increasing parallelism and improving data import performance. According to the data distribution of the MPP cluster, data can be obtained from local storage first, reducing data transmission overhead, saving network bandwidth, reducing network delay, and improving computing performance.
Need to check novelty before this filing date? Find Prior Art

Description

technical field

[0001] The present invention relates to the field of computer technology, in particular to a data acquisition method and device based on a Spark computing framework. Background technique

[0002] Apache Spark (Spark for short) is a fast and general computing engine designed for large-scale data processing. Spark natively provides access interfaces to file systems including HDFS (Hadoop Distributed File System, Distributed File System), and Spark can usually be used to calculate and analyze structured data in the database.

[0003] Specifically, Spark can export the data of the target database to a file through the database export tool, and then calculate and analyze the data through the file, but this method is cumbersome, time-consuming, error-prone, and is largely affected by the environment. limit.

[0004] In addition, you can also directly access the database system through the JdbcRDD provided by Spark. JdbcRDD is a general-purpose database access in...

Examples

Embodiment Construction

[0023] The present invention will be described in further detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described here are only used to explain the present invention, not to limit the present invention.

[0024] This embodiment provides a data acquisition method based on the Spark computing framework. This embodiment is executed on the Spark side. Such as figure 1 Shown is a flowchart of a data acquisition method based on the Spark computing framework according to an embodiment of the present invention.

[0025] Step S110, after receiving the table object access request, obtain the computing resource information of Spark and the data distribution information of the data tables to be accessed in the MPP cluster (also called MPP database cluster).

[0026] The table object access request is used to request to access the data tables stored in the MPP cluster. According to the table object acce...