Data processing method and device

A data processing and physical planning technology, applied in the field of big data, can solve problems such as inability to guarantee the security of sensitive data, reduce information security, and leakage

Active Publication Date: 2020-12-01
NEW H3C BIG DATA TECH CO LTD
View PDF7 Cites 0 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

[0004] In related technologies, SparkSQL does not have the function of encrypting data, so when users use Spark API or Spark SQL to operate data, the security of sensitive data (such as contact information, passwords, etc.) cannot be guaranteed. When data files are leaked, Sensitive user information will be leaked, reducing the security of information

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • Data processing method and device
  • Data processing method and device
  • Data processing method and device

Examples

Experimental program
Comparison scheme
Effect test

Embodiment Construction

[0026] Various exemplary embodiments, features, and aspects of the present disclosure will be described in detail below with reference to the accompanying drawings. The same reference numbers in the figures indicate functionally identical or similar elements. While various aspects of the embodiments are shown in drawings, the drawings are not necessarily drawn to scale unless specifically indicated.

[0027] The word "exemplary" is used exclusively herein to mean "serving as an example, embodiment, or illustration." Any embodiment described herein as "exemplary" is not necessarily to be construed as superior or better than other embodiments.

[0028] In addition, in order to better illustrate the present disclosure, numerous specific details are given in the following specific implementation manners. It will be understood by those skilled in the art that the present disclosure may be practiced without some of the specific details. In some instances, methods, means, componen...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

PUM

No PUM Login to view more

Abstract

The invention relates to a data processing method and a device. The method is applied to a driver in a Spark. The method comprises the following steps: when an insertion SQL statement is received, theinsertion SQL statement is parsed to generate an insertion logic plan tree; if the insertion logical plan tree matches the encryption rule, an encryption node is created, and the encryption node is inserted into the insertion node of the insertion logical plan tree to obtain an encryption logical plan tree; the encrypted logical plan tree is converted into an encrypted physical plan tree, and theencrypted physical plan tree is sent to an executor in Spark. By encrypting the data before inserting the data, the data processing method and apparatus according to the embodiments of the present disclosure can implement the data encryption function in SparkSQL.

Description

technical field [0001] The present disclosure relates to the technical field of big data, and in particular to a data processing method and device. Background technique [0002] Spark is a memory-based distributed computing framework. Spark provides one-stop data analysis capabilities, including small batch stream processing, offline batch processing, SQL (Structured Query Language, structured query language) query, data mining, etc. Users can seamlessly combine these capabilities in the same application . Spark improves the real-time performance of data processing in a big data environment, while ensuring high fault tolerance and high scalability, allowing users to deploy Spark on a large number of cheap hardware to form a cluster. [0003] SparkSQL is a Spark-based distributed SQL engine. It is a Spark component used to process structured data. It supports SQL statements, enabling users to quickly and conveniently run Spark computing tasks in SQL. [0004] In related te...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

Application Information

Patent Timeline
no application Login to view more
Patent Type & Authority Patents(China)
IPC IPC(8): G06F16/242G06F9/50G06F21/60G06F21/62
CPCG06F9/5066G06F21/602G06F21/6227
Inventor 史宁宁户蕾蕾杜威科
Owner NEW H3C BIG DATA TECH CO LTD
Who we serve
  • R&D Engineer
  • R&D Manager
  • IP Professional
Why Eureka
  • Industry Leading Data Capabilities
  • Powerful AI technology
  • Patent DNA Extraction
Social media
Try Eureka
PatSnap group products

Browse by: Latest US Patents, China's latest patents, Technical Efficacy Thesaurus, Application Domain, Technology Topic.

© 2024 PatSnap. All rights reserved.Legal|Privacy policy|Modern Slavery Act Transparency Statement|Sitemap