Method and device for verifying processed data accuracy under MapReduce environment

Abstract

The invention provides a method for verifying processed data accuracy under a MapReduce environment. The method comprises the steps that a hijacking code is added to a MapReduce source code to generate a data processing program with the hijacking code; the data processing program with the hijacking code runs to process input data, output data are obtained, and triple data are formed; the triple data with identical coverage information are removed, and a monitoring triple set is formed; when processed data accuracy needs to be verified, the input data in the monitoring triple set are returned to the data processing program with the hijacking code to be processed again, output data are obtained, and an output data set is formed; whether each piece of data in the output data set is accurate or not is verified. According to the method for verifying processed data accuracy under the MapReduce environment, the manual cost and implementation time for verifying processed data accuracy can be reduced. The invention further provides a device for verifying processed data accuracy under the MapReduce environment. The method can be achieved.

Description

technical field [0001] The invention relates to the field of computer computing, in particular to a method and a device for verifying the correctness of processing data in a MapReduce environment. Background technique [0002] With the development of computer technology, the amount of data that the computer needs to process is also increasing, and a single computer can no longer process some large-scale data. scale data. Cloud computing is the organization and management of equipment through the use of well-designed system architecture, which can provide very powerful computing power. MapReduce is a programming model that is usually used to run large-scale data sets in parallel and distributed in large-scale clusters. The large-scale data sets generally refer to data sets larger than 1TB (trillion bytes). [0003] The process of MapReduce's parallel processing of the data set may specifically include: decomposing the data set into multiple data blocks according to the numb...

AI Technical Summary

Problems solved by technology

[0008] When using the method of writing test cases to verify the correctness of data processing in the MapReduce environment, it is necessary to manually write test cases. In order to ensure that the written test cases can reflect data processing errors when the operation results in any MapReduce environment should change, it is necessary to ensure that The written test cases should be as comprehensive as possible, which requires a lot of labor costs; at the same time, due to the complexity of the operating environment of MapReduce, errors that may occur during the operation cannot be completely predicted, so manual writing of test cases is usually not guaranteed to be successful. Reflects errors in all data processing, so processing data for test cases may not be reliable

When the real data method is used to verify the correctness of data processed in the MapReduce environment, the real data usually has a huge amount of data. Generally, the real data can reach between hundreds of GB (gigabytes) and several TB (terabytes). Therefore, whether it is to perform data processing on the real data or to compare the results of data processing, it will take several hours or even a day or two.

Images