Data secrecy method and system for MapReduce computing

Abstract

The invention belongs to the technical field of cloud computing data secrecy, and discloses a data secrecy method and system for MapReduce computing. The method comprises the steps of proposing Full Shuffle and Safe Shuffle under a standard MapReduce computing framework; sorting input data into a key-value pair form for preprocessing; combining the key value pairs in each map task in a Map sub-stage according to the key value; respectively distributing the combined key value pairs to each reduce task of a Reducing stage layer; processing the data received by each reduce task in the Reduce sub-stage; and enabling the output data of each reduce task to be equal in size. According to the invention, the data and privacy based on the MapReduce framework in a remote execution environment scene are protected, and the data privacy of the application program is prevented from being obtained by malicious observers through side channel attacks.

Description

technical field [0001] The invention belongs to the technical field of cloud computing data security, and in particular relates to a data security method and system for MapReduce computing. Background technique [0002] At present, the closest existing technology: MapReduce is a parallel programming model for parallel computing of large-scale data sets, with the characteristics of functional programming language and vector programming language, with data division and computing task scheduling, system optimization, The capabilities of error detection and recovery, therefore, make MapReduce suitable for applications such as log analysis, machine learning, and distributed sorting. A MapReduce job is a unit of work that the user wishes to be executed: it includes input data, the MapReduce program, and configuration information. MapReduce runs a job by dividing it into tasks. There are two types of tasks: map task and reduce task. The standard MapReduce data flow of multiple r...

AI Technical Summary

Problems solved by technology

[0011] 1. The existing solutions have strict requirements on the operation record of the experimental data set, which also leads to the low applicability of the existing technology to solve the problem

[0012] 2. Compared with the standard MapReduce framework, existing schemes have higher performance overhead. For example, the SHUFFLE-IN-THE-MIDDLE scheme will cause 191% to 205% performance overhead, and the SHUFFLE&BALANCE scheme will cause 95% to 101% in the online stage. % performance overhead

[0013] 3. The existing solution only supports the default Partition function, not the user-defined shuffling function

[0014] 4. The premise of the success of the existing scheme is the accurate probability estimation of the input data. Once the estimation is inaccurate, it will lead to the failure of the scheme and unnecessary high-performance overhead

Images