Transparent partition method and device based on sparksql

Abstract

The invention discloses a SparkSQL-based transparent partition method and device, and the method comprises the steps: obtaining an appointed partition field and an adopted partition strategy when a user creates a partition table after a table building statement is submitted, determining a transparent partition through an SQL parser, and storing the transparent partition information; When a user inquires data through SQL, generating a logic execution plan through SQL analysis; Calculating a data partition of the query according to query conditions in the logic execution plan in combination withthe stored transparent partition information, and then rewriting the logic execution plan and generating an optimized physical execution plan; And according to the execution steps of the generated physical execution plan, dividing a specific task, and reading data from the data partition through the task.

Description

technical field [0001] The invention relates to the field of computers, in particular to a SparkSQL-based transparent partitioning method and device. Background technique [0002] At present, the well-known SparkSQL partitioning technology is to specify a special partition field when creating a data table, and the data table is mapped to a directory on HDFS, and the data of the table is stored in this directory, and the partition field data is not included with the non-partition fields. are stored in data files on HDFS, but instead exist on HDFS as subdirectories of the datatables directory. SparkSQL is an open source technology that uses standard SQL to process and analyze big data. It creates tables to store structured data, uses partitions to refine data storage, and divides data into different partitions and stores them in different subdirectories on HDFS. The subpartition name is the subdirectory name. This patent designs a transparent partition technology based on ex...

AI Technical Summary

Problems solved by technology

[0004] 1. Unable to automatically map non-partition filter conditions to partition filter conditions

Business personnel who use data do not understand Spark Partition technology, or know Spark Partition technology but do not know which partition directory the data they want to query falls in. At this time, business personnel can only scan all the data in the table through full table scanning. Take out and filter all the data to find the data you need accurately. Every time a user submits a SQL query to analyze data, they cannot make full use of the data partition to narrow the query range of the data, resulting in slow query response, and when the amount of data in the table is too large, it is easy Exhausting system computing resources

[0005] 2. Inconsistent partition formats can easily lead to irregular data storage and management

The existing partitioning technology only defines the partition field type, and does not further standardize the specific format of the partition. For example, if the partition field type is String, any partition of the String type can exist as a subdirectory on HDFS, and the chaotic partition format can seriously increase the cost of big data management

[0006] 3. It is inconvenient to delete expired data

The increasing data will inevitably lead to the clearing of historical data to free up storage space. Traditional SparkSQL partitions often require manual intervention when deleting expired data due to irregular partition formats, and cannot well define a set of rules to periodically delete expired data. Data, frequent manual deletion of data will have a great impact on data security, and may even cause irreparable losses

Images