Data processing method and relevant equipment

Abstract

A data processing method and a related device, implementing automatic conversion of data format and automatic splicing of data in a node device by a Hadoop. The method mainly comprises: a data preprocessor reads metadata from a first buffer area of a CPU, reads data of a data collection from the first buffer area on the basis of a memory address indicated by the metadata, converts, on the basis of a preset analytic function, the data of the data collection into a data format indicated by the preset analytic function, and stores data blocks generated with the converted data collection in a second buffer area of the CPU, thus allowing a data splicer to read from the second buffer area the data blocks and to splice same to a GPU.

Description

technical field [0001] The invention relates to the technical field of information processing, in particular to a data processing method and related equipment. Background technique [0002] Together with cloud computing, big data has brought a new revolution to information technology (IT, Information Technology). Cloud computing has powerful big data computing capabilities and very fast computing speed, but the transmission of big data has become a major problem. [0003] MapReduce (there is no unified Chinese translation in this field) is a famous cloud computing architecture provided by Google search engine Google, which is used for parallel computing on large-scale data sets (greater than 1TB). Hadoop (there is no unified Chinese translation in this field) Chinese translation) is the specific implementation of the MapReduce architecture, which is divided into master node devices and slave node devices in the Hadoop cluster. Among them, the Map function provided by MapRed...

AI Technical Summary

Problems solved by technology

[0004] The technical personnel of the present invention finds when realizing above-mentioned scheme, because no analytic function is provided in the MapReduce architecture, when analyzing the value of key-value pair, need to rely on the corresponding program written by the programmer; The size of the buffer for the correct value may not be consistent with the size of the buffer allocated by the GPU to store data, and the MapReduce architecture does not provide a corresponding judgment method, which also depends on the corresponding judgment function written by the programmer. Whether it is consistent to judge and reduce the execution efficiency of the slave node device

Images