A source code compiling method and system based on a MapReduce framework distributed computing environment

Abstract

The invention relates to a source code compiling method and system based on a MapReduce framework distributed computing environment, and the method comprises the steps: enabling source code compilingto be better allocated to a cluster machine for compiling by analyzing the dependency relationship between files at different stages; Providing a concurrent write lock module to enable the cluster machines to write files in a mutual concurrent manner; according to the source code compiling method and system based on the MapReduce framework distributed computing environment, a MapReduce mode is used for organizing compiling work of different stages and forming an assembly line, multi-machine simultaneous compiling and multi-order concurrent assembly line is achieved, and network transmission resources are reduced; Moreover, through a synergy mechanism provided by the concurrent write lock. The source code compiling speed can be increased, and the compiling task speed and the task throughputcan be increased.

Description

technical field [0001] The present invention relates to a software source code compiling method, more specifically to a source code compiling method and system based on a MapReduce framework distributed computing environment. Background technique [0002] Under the background of informatization construction, there are more and more fields of computer software application. With the continuous deepening of application fields, computer software is constantly expanding, and the scale of software code is also increasing. Now the number of kernel source codes of the Linux operating system has reached the million level. It takes nearly 1-4 hours to compile on a server with 4 CPUs. A complete OS compilation takes about 1-2 weeks. Therefore, there is a great need for an efficient, stable, and scalable compilation technology to improve the compilation speed. [0003] Under the existing compilation technology conditions, in order to improve the compilation speed, the compilation equ...

AI Technical Summary

Problems solved by technology

[0004] At present, the existing compilation technology has the following shortcomings: the compiler can only use multiple CPUs and multiple cores on a single device, and cannot reasonably and efficiently allocate tasks to multiple machines for compilation; the existing distributed compilation also has the following shortcomings: Lack of multi-level pipeline processing leads to low throughput of compilation tasks; task allocation is simple without multi-stage division; dynamic scheduling algorithms are complex and difficult to run efficiently and stably; resource constraints on one node lead to task failures, tasks cannot be restarted, and compilation tasks often end in failure; Network resource utilization is not planned, etc.

Therefore, there are problems with the stability, usability, and efficiency of the existing distributed compilation system

Images