A data security method and system for mapreduce computing

Abstract

The invention belongs to the field of cloud computing data security technology, discloses a data security method and system for MapReduce computing, proposes Full Shuffle and Safe Shuffle under the standard MapReduce computing framework; input data is sorted into the form of key-value key-value pair Preprocessing: Merge the key-value pairs in each map task in the Map sub-stage according to the key value; distribute the merged key-value pairs to each reduce task in the Reducing stage layer; receive the data received by each reduce task in the Reduce sub-stage The data is processed; each reduce task output data size is equal. The invention protects the data and privacy based on the MapReduce framework in the remote execution environment scene, and prevents the data privacy of the application program from being acquired by malicious observers through side channel attacks.

Description

technical field [0001] The invention belongs to the technical field of cloud computing data security, and in particular relates to a data security method and system for MapReduce computing. Background technique [0002] At present, the closest existing technology: MapReduce is a parallel programming model for parallel computing of large-scale data sets, with the characteristics of functional programming language and vector programming language, with data division and computing task scheduling, system optimization, The capabilities of error detection and recovery, therefore, make MapReduce suitable for applications such as log analysis, machine learning, and distributed sorting. A MapReduce job is a unit of work that the user wishes to be executed: it includes input data, the MapReduce program, and configuration information. MapReduce runs a job by dividing it into tasks. There are two types of tasks: map task and reduce task. The standard MapReduce data flow of multiple r...

AI Technical Summary

Problems solved by technology

[0011] 1. The existing solutions have strict requirements on the operation record of the experimental data set, which also leads to the low applicability of the existing technology to solve the problem

[0012] 2. Compared with the standard MapReduce framework, existing schemes have higher performance overhead. For example, the SHUFFLE-IN-THE-MIDDLE scheme will cause 191% to 205% performance overhead, and the SHUFFLE&BALANCE scheme will cause 95% to 101% in the online stage. % performance overhead

[0013] 3. The existing solution only supports the default Partition function, not the user-defined shuffling function

[0014] 4. The premise of the success of the existing scheme is the accurate probability estimation of the input data. Once the estimation is inaccurate, it will lead to the failure of the scheme and unnecessary high-performance overhead

Images