Orthogonal decomposition construction and optimization method and system for big data processing system

Abstract

The invention discloses an orthogonal decomposition construction and optimization method and system for a big data processing system. The method comprises: performing offline static analysis in a layer, used for executing an intermediate representation code or a binary hardware instruction of a logic, of the big data processing system with a super optimization method, and extracting an optimized instruction sequence as an optimized basic block; creating an optimized code database, calculating a fingerprint of the optimized basic block, and storing the fingerprint and the optimized basic block into a fingerprint Map and the optimized code database respectively; and acquiring an online input program, extracting a basic block of the program, calculating a fingerprint of the basic block, comparing the fingerprint of the basic block of the program with the fingerprint in the optimized code database, and if the matching is successful, inputting the basic block of the program into a just-in-time compilation engine for execution, wherein the big data processing system is only in charge of most core resources and task management, works in an intermediate layer, and is unrelated with platforms and languages.

Description

technical field [0001] The invention relates to the field of big data processing, in particular to a method and system for orthogonal decomposition construction and optimization of a big data processing system. Background technique [0002] The complexity, uncertainty, and emergence of network big data have brought challenges and opportunities to the architecture and computing power of current IT systems, and have given birth to big data processing frameworks. Around these computing frameworks, various big data processing systems have been born. , such as Google GFS and MapReduce for batch big data processing, Nokia's Disco, Google Dremel for stream processing, Microsoft's Dryad, Twitter's Storm, Yahoo's S4, etc., the academic and open source communities also focus on batch big data processing In-depth research has been carried out on Apache Hadoop, the more real-time Impala based on Hadoop, and the RDD-based Spark of Berkeley AMPLab, which is oriented to the iterative appli...

AI Technical Summary

Problems solved by technology

[0004] However, due to the large scale of the big data processing system, it emphasizes platform independence and avoids being linked to specific operating systems and hardware platforms. The evolution of the above systems ignores the use of underlying platform technologies. The computing engine module extends part of the internal computing of the Hadoop computing engine to the Hotspot virtual machine, and has achieved a certain localization effect. The idea is worth learning, but it has not yet fully utilized the storage node, the local operating system of the computing node, and the hardware platform. The domestic Baidu company also proposed the C++ extension of Hadoop, which uses a protocol similar to Pipe to replace the two-stage JAVA execution logic of Map and Reduce with binary executable files written in C++ and precompiled, making it a step towards localization. One step, but it loses the flexibility of intermediate logic representation, and localization is limited to Map and Reduce user logic, and there is no room for deep mining code optimization

Images