Hadoop transplanting method and system based on domestic SW processor

Abstract

The invention belongs to the technical field of program transplantation, and particularly relates to a hadoop transplantation method and system based on a domestic SW processor. Comprising the following steps: step 1, modifying a Hadoop source code to enable the Hadoop source code to pass through a compiling environment of a SW compiler; step 2, in the compiling link rule file, adding a compilation option of the Shenwei openeller system and a corresponding compilation rule of the Shenwei openeller system; 3, setting a link dependence library of the project sub-project; and step 4, under a compilation option of the SW openeller system, compiling the modified hadoop source code through a SW compiler, and generating an executable program of the SW processor. By means of the method, domestic transplantation and one-key compiling effects of Hadoop are achieved, and working efficiency and accuracy are greatly improved.

Description

Technical Field

[0001] The invention belongs to the technical field of program transplantation, and in particular relates to a Hadoop transplantation method and system based on a domestically produced Shenwei processor. Background Art

[0002] In recent years, data has continued to increase in all walks of life at a large scale, and big data is changing people's lives, work, and thoughts. Faced with such a large amount of data, more and more companies are beginning to pay attention to data. Hadoop is a distributed system developed by the Apache Open Source Foundation, consisting of HDFS and YARN. It is suitable for distributed storage and computing in the field of big data and is currently a widely used big data computing tool.

[0003] Based on the domestic Shenwei processor as a domestically produced self-controlled high-performance computing CPU, it has powerful computing capabilities. OpenEuler (Euler operating system of the Open Atom Open Source Foundation) is an open source operating system that currently supports Shenwei processors. Hadoop is transplanted to the domestic Shenwei OpenEuler system to provide a leading example demonstration for big data processing.

[0004] Currently, Hadoop programs cannot run on the domestic Shenwei platform. In the process of porting Hadoop to the domestic Shenwei Openeuler system, a series of challenges need to be overcome. These challenges include: software compatibility; since the domestic Shenwei processor uses specific hardware construction and operating system, it is necessary to ensure that Hadoop can run normally in these environments; this involves modifying the Hadoop source code to adapt to the domestic Shenwei compiler, library and system configuration. Summary of the invention

[0005] The purpose of the present invention is to overcome the above-mentioned deficiencies in the prior art. The present invention provides a Hadoop transplantation method and system based on the domestic Shenwei processor, which transplants Hadoop to the domestic Shenwei Openeuler operating system for the first time, promotes its application in the field of big data processing, and can be provided to customers as IaaS (software as a service) of the Shenwei platform.

[0006] In order to solve the above technical problems, the present invention provides a Hadoop transplantation method based on a domestic Shenwei processor, comprising the following steps: Step 1: Modify the Hadoop source code to pass the compilation environment of the Shenwei compiler; Step 2: In the compilation and linking rule file, add the Shenwei Openeuler system compilation options and the corresponding Shenwei Openeuler system compilation rules; Step 3: Set the link dependency library of the project sub-project; Step 4: Under the Shenwei Openeuler system compilation option, compile the modified Hadoop source code through the Shenwei compiler to generate the Shenwei processor executable program.

[0007] Preferably, in step 1, modifying the Hadoop source code specifically includes: adding the bulk_crc32.c verification file of the Shenwei architecture.

[0008] Preferably, in step 2, in the compilation link rule file, add the Shenwei Openeuler system compilation option and the corresponding Shenwei Openeuler system compilation rule, specifically: Add compilation instructions in build.sh; Add the Shenwei architecture to the HadoopJNI.cmake file; Add bulk_crc32_sw_64.c Shenwei verification file option in hadoop-common-project / hadoop-common / src / CMakeLists.txt.

[0009] Preferably, in step three, the link dependency library of the project sub-project is set, specifically: during compilation, the hadooputils module and the tirpc module are set to be linked as a static library through a configuration file.

[0010] Preferably, in step 4, the modified Hadoop source code is compiled by the Shenwei compiler, specifically: compiling into a running program based on the Shenwei architecture is selected; and an executable program for the Shenwei processor is generated by a local compilation method.

[0011] Preferably, it also includes: in the step three: specifying the required dependent warehouse through pom.xml.

[0012] The present invention also provides a Hadoop transplantation system based on a domestically produced Shenwei processor, which adopts the method described above and includes: The code modification module is configured to: modify the Hadoop source code so that it passes the compilation options of the Shenwei compiler; The option adding module is configured to: add the Shenwei Openeuler system compilation options and the corresponding Shenwei Openeuler system compilation rules in the compilation and linking rule file; The verification file module is configured to: add Shenwei's crc32 verification file to support the big data verification function; The code compilation module is configured as follows: under the Shenwei Openeuler system compilation option, the modified Hadoop source code is compiled by the Shenwei compiler to generate an executable program for the Shenwei processor.

[0013] The present invention also provides a storage medium that non-temporarily stores computer-readable instructions, wherein when the non-temporary computer-readable instructions are executed by a computer, the method described above is executed.

[0014] The present invention also provides an electronic device, comprising: a memory for non-transitory storage of computer readable instructions; A processor is used to execute the computer-readable instructions; wherein, when the computer-readable instructions are executed by the processor, the method described above is performed.

[0015] Compared with the prior art, the present invention has the following beneficial effects: The present invention successfully compiles and runs Hadoop on the Shenwei processor openeuler system, filling the gap in big data software on the Shenwei processor openeuler system; transplanting Hadoop on the domestic Shenwei openeuler system provides a leading example demonstration for big data processing. The actual operation of the solution is not complicated, and provides a convenient installation solution for installation and use or program optimization on the Shenwei processor, thereby greatly reducing the installation time and providing a solid foundation for use and optimization.

[0016] The Hadoop compilation method of the present invention is different from the existing compilation method in that: 1) the Shenwei processor platform is a self-developed platform, and some compilation rules need to be specified in the cmake file and modified accordingly. 2) The Hadoop source code needs to be modified to adapt and pass the Shenwei compiler. Through the above method, the effect of domestic transplantation and one-click compilation of Hadoop is achieved, which greatly improves work efficiency and accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1 The present invention is a flow chart of a Hadoop transplantation method based on domestic Shenwei processing.

[0018] Figure 2 This is a flow chart of a Hadoop transplantation system based on domestic Shenwei processing in the present invention. DETAILED DESCRIPTION

[0019] The present invention is further described in detail below in conjunction with the accompanying drawings and specific embodiments. The advantages and features of the present invention will become more apparent from the following description. It should be noted that the accompanying drawings are in very simplified form and in non-precise proportions, and are only used to conveniently and clearly assist in explaining the purpose of the embodiments of the present invention. Example

[0020] like Figure 1 As shown, the embodiment of the present invention specifically provides a Hadoop transplantation method based on domestic Shenwei processing, including: Step S1: Modify the Hadoop source code to pass the compilation environment of the Shenwei compiler; Step S2: Add the Shenwei Openeuler system compilation options and the corresponding Shenwei Openeuler system compilation rules in the compilation and linking rule file; Step S3: Set the link dependency library of the project sub-project; Step S4: Under the compilation option of the Shenwei Openeuler system, the modified Hadoop source code is compiled by the Shenwei compiler to generate a Shenwei processor executable program.

[0021] As a preferred embodiment, the specific implementation of the Hadoop transplantation method based on the domestic Shenwei processor is as follows: Step 1: Modify the Hadoop source code to pass the compilation environment of the Shenwei compiler; Specifically, modify the source code, the bulk_crc32.c file of the crc data verification module in Hadoop, the modified bulk_crc32_sw_64.c is as follows: Will static uint32_t crc32c_sb8(uint32_t crc, const uint8_t *buf, size_tlength) { uint32_t running_length = ((length) / 8)*8; uint32_t end_bytes = length - running_length; int li; for (li=0; li <running_length / 8; li++) { … } for (li=0; li <end_bytes; li++) { crc = CRC32C_T8_0[(crc ^ *buf++)&0x000000FF] ^ (crc>>8); } return crc; } Modified to: static uint32_t crc32c_sb8(uint32_t crc, const uint8_t *buf, size_tlength) { while (length&&((ptrdiff_t)buf&3)) { crc = CRC32C_T8_0[(crc ^ *buf++)&0x000000FF] ^ (crc>>8); length--; } while (length>= 8) { … length -= 8; } if (length) do { crc = CRC32C_T8_0[(crc ^ *buf++)&0x000000FF] ^ (crc>>8); } while (--length); return crc; } The above two crc32c_sb8 functions can both calculate the CRC-32C check value. The difference lies in the different ways of processing input data. The main difference is that according to the characteristics of Shenwei CPU, the data alignment method is used when processing input data to improve memory access efficiency and facilitate the improvement of large block data processing efficiency. The following is an analysis of calculating the CRC-32 check value: The first implementation (code lines 0050-0060); uint32_t running_length = ((length) / 8)*8; uint32_t end_bytes = length - running_length; In the above functions, running_length calculates the length of the input data that is divisible by 8. end_bytes calculates the length of the remaining bytes of the input data.

[0022] for (li=0; li <running_length / 8; li++) This is the first loop, processing data of running_length length. Each iteration processes 8 bytes to obtain the CRC-32 check value.

[0023] for (li=0; li <end_bytes; li++) This is the second loop that processes the remaining end_bytes of input data, that is, those bytes that are not divisible by 8. A table lookup method (via the CRC32C_T8_0 array) is used here to update the CRC value byte by byte.

[0024] return crc; After the loop is completed, the CRC-32 checksum result is finally returned.

[0025] The second implementation (code lines 0063-0075); while (length&&((ptrdiff_t)buf&3)) This loop handles the situation where the address of the input buffer buf is not aligned to a 4-byte boundary. Here the data is processed byte by byte until the address of buf is aligned to 4 bytes.

[0026] while (length>= 8) The second loop processes data blocks with a length greater than or equal to 8 bytes and obtains a CRC-32 checksum.

[0027] ‌Process remaining bytes‌: if (length) do The function of this code is to process these data byte by byte through a do-while loop if there are any remaining bytes (that is, length is not 0).

[0028] return crc; After the loop is completed, the final CRC-32 check value is returned.

[0029] static uint32_t crc32_zlib_sb8( uint32_t crc, const uint8_t *buf, size_t length) { uint32_t running_length = ((length) / 8)*8; / / Calculate the length of the 8-byte aligned input data uint32_t end_bytes = length - running_length; / / Calculate the remaining unaligned bytes int li; for (li = 0; li < running_length / 8; li++) { / / Traverse each 8-byte block to get the CRC-32 checksum … } for (li = 0; li < end_bytes; li++) { / / Traverse the remaining bytes to get the CRC-32 checksum crc = CRC32_T8_0[(crc ^ *buf++)&0x000000FF] ^ (crc >> 8); } return crc; / / Return the calculated CRC-32 checksum } Modified to: static uint32_t crc32_zlib_sb8( uint32_t crc, const uint8_t *buf, size_t length) { while (length && ((ptrdiff_t)buf & 3)) { / / When the length is not zero and the buf pointer is not aligned to a 4-byte boundary crc = CRC32C_T8_0[(crc ^ *buf++)&0x000000FF] ^ (crc >> 8); / / Update the crc value using the CRC32C algorithm and move the buf pointer length--; / / Decrease the remaining length } while (length >= 8) { / / When the remaining length is greater than or equal to 8 bytes, batch calculate the CRC-32 checksum … length -= 8; / / Decrease the processed 8-byte length } if (length) do { / / When there are still remaining bytes crc = CRC32_T8_0[(crc ^ *buf++)&0x000000FF] ^ (crc >> 8); / / Update the crc value using the CRC32C algorithm and move the buf pointer} while (--length); / / Reduce the remaining length until it reaches 0 return crc; / / Return the calculated CRC-32 checksum } The above two codes are annotated with the function statement function. For example, in the statement "return crc; / / Return the calculated CRC-32 check value", "return crc;" is the code part, and " / / Return the calculated CRC-32 check value" is the comment part. In the C language rule syntax, the symbol " / / " is a single-line comment symbol. When the original function crc32_zlib_sb8 in the original code processes data, it first calculates the number of complete 8-byte blocks and then processes the remaining bytes. It mainly implements the CRC-32 check value calculation through two for loops, but does not specially handle memory alignment. The modified code on the Shenwei side is that the new function first processes the unaligned bytes until the pointer is aligned to the 4-byte boundary, then processes the data blocks of 8 bytes and above, and finally processes the remaining bytes. This modification improves the memory access efficiency and execution speed of the code, especially optimizing the unaligned data access.

[0030] Step 2: Modify the compilation and linking rule file CMakeLists.txt, add the Shenwei openeuler system compilation option based on the cmake compilation method, and add the Shenwei openeuler system compilation rule to the Hadoop compilation and linking rule file, so that the program can be compiled into a running program based on the Shenwei architecture and run on the Shenwei platform. Specifically: 1) Add the Shenwei openeuler system compilation option based on the .buld.sh file: mvn clean package -DskipTests -Pdist,native -Dtar -Dsnappy.lib= / usr / lib / sw_64-linux-gnu -Dbundle.snappy -Dmaven.javadoc.skip=true This Maven compilation command is used to clean up the previous build files, skip the test phase, activate the dist and native build profiles, set some properties related to the Snappy library, skip Javadoc generation, and finally package the project.

[0031] 2) Add the Shenwei openeuler system compilation option based on the HadoopJNI.cmake file: elseif(CMAKE_SYSTEM_PROCESSOR MATCHES "^sw_64")

[0110] set(_java_libarch "sw64") The above code sets the _java_libarch variable to "sw64" when CMake detects that the processor architecture of the target system is "sw_64".

[0032] 3) Add Shenwei crc verification support based on the project module hadoop-common-project / hadoop-common / src / CMakeLists.txt file, specifically: # Build CRC32 on SW_64. if(CMAKE_SYSTEM_PROCESSOR MATCHES "sw_64") / / Judge whether the value of the CMAKE_SYSTEM_PROCESSOR variable matches the regular expression "sw_64". If it matches, it means that the current system processor is SW_64 architecture.

[0033] set(BULK_CRC_SOURCE_FILE "${SRC} / util / bulk_crc32_sw_64.c") / / Set the value of the BULK_CRC_SOURCE_FILE variable to "${SRC} / util / bulk_crc32_sw_64.c", which is the CRC32 source file path dedicated to the sw_64 architecture.

[0034] else() set(BULK_CRC_SOURCE_FIlE "${SRC} / util / bulk_crc32.c") endif() Cancel line 270 and add content ${BULK_CRC_SOURCE_FIlE} / / Add variable ${BULK_CRC_SOURCE_FIlE}, which is the path of the "bulk_crc32_sw_64.c" file, and then you can compile the corresponding CRC-32 function module of Shenwei.

[0035] Cancel line 295 and add content ${BULK_CRC_SOURCE_FIlE} / / Add variable ${BULK_CRC_SOURCE_FIlE}, which is the path of the "bulk_crc32_sw_64.c" file, and then you can compile the corresponding CRC-32 function module of Shenwei.

[0036] The purpose of doing this is to add the CRC verification module of the Shenwei platform to avoid the failure to find the data verification function module of the Shenwei platform during compilation.

[0037] Step 3: Based on step 2, set the link dependency library of the project sub-project.

[0038] In the hadoop-tools / hadoop-pipes submodule, modify the link mode to static link in CMakelists.txt (CMake configuration file, which describes the compilation rules and dependencies of the project and guides CMake to generate build files suitable for a specific platform and compiler.) as follows: Add target_link_libraries(hadooputils tirpc) based on line 53 / / Add the target_link_libraries command to specify that the hadooputils target needs to link the tirpc library.

[0039] The purpose of doing this is because the Shenwei computing platform is relatively friendly to static library support, avoiding errors caused by setting dynamic links during the compilation process.

[0040] Step 4: Based on the above three steps, execute . / build.sh to compile Hadoop based on the Shenwei platform. The generated executable program is the Shenwei processor architecture.

[0041] Embodiment 2 like Figure 2 As shown, the embodiment of the present invention specifically provides a Hadoop transplantation system based on a domestic Shenwei processor, including: The source code modification module is configured to: modify the Hadoop source code so that it passes the compilation environment of the Shenwei compiler; The option adding module is configured to: add the Shenwei Openeuler system compilation options and the corresponding Shenwei Openeuler system compilation rules in the compilation link rule file; Set the sub-project link module, which is configured to: set the link dependency library of the project sub-project; The code compilation module is configured to compile the modified Hadoop source code through the Shenwei compiler under the compilation option of the Shenwei Openeuler system to generate an executable program for the Shenwei processor.

[0042] Embodiment 3 An embodiment of the present invention specifically provides a storage medium that non-temporarily stores computer-readable instructions, wherein when the non-temporary computer-readable instructions are executed by a computer, the method described above is executed.

[0043] Embodiment 4 An embodiment of the present invention specifically provides an electronic device, including: a memory for non-transitory storage of computer readable instructions; A processor is used to execute the computer-readable instructions; wherein, when the computer-readable instructions are executed by the processor, the method described above is performed.

[0044] In summary, the present invention has the following advantages: a) Expanding application areas: The first transplantation of Hadoop to the domestic Shenwei Openeuler system can promote its application in the domestic big data field, such as e-commerce industry, financial industry, medical industry, industrial industry, etc.

[0045] b) Improve the research level of domestic big data technology: The Hadoop big data cluster based on the Shenwei platform supports communication operators in providing PaaS and SaaS services to the outside world, and provides a leading example demonstration for Shenwei-based big data processing.

[0046] c) Optimize resource utilization: By adding a static link method, the Hadoop compilation and linking process is optimized, which facilitates the multi-core optimization compilation process of the program and improves the work efficiency of developers.

[0047] d) Improve the processing speed of big data clusters: By making full use of the powerful computing power of the domestic Shenwei platform, Hadoop can greatly improve the computing speed in big data tasks, bringing higher engineering efficiency to the field of big data processing.

[0048] The above description is only a description of the preferred embodiments of the present invention, and is not intended to limit the scope of the present invention. Any changes or modifications made by a person skilled in the art in the field of the present invention based on the above disclosure shall fall within the scope of protection of the claims.

Claims

1. A Hadoop transplantation method based on domestic Shenwei processor, characterized in that: The steps include: Step 1: Modify the Hadoop source code to pass the compilation environment of the Shenwei compiler; Step 2: In the compilation and linking rule file, add the Shenwei Openeuler system compilation options and the corresponding Shenwei Openeuler system compilation rules; Step 3: Set the link dependency library of the project sub-project; Step 4: Under the Shenwei Openeuler system compilation option, compile the modified Hadoop source code through the Shenwei compiler to generate the Shenwei processor executable program; In the step 2, in the compilation link rule file, add the Shenwei Openeuler system compilation option and the corresponding Shenwei Openeuler system compilation rule, specifically: Add compilation instructions in build.sh; Add the Shenwei architecture to the HadoopJNI.cmake file; Add bulk_crc32_sw_64.c Shenwei verification file option in hadoop-common-project / hadoop-common / src / CMakeLists.txt.

2. A Hadoop transplantation method based on a domestic Shenwei processor as claimed in claim 1, characterized in that: In the step 1, modifying the hadoop source code specifically includes: adding the bulk_crc32.c verification file of the Shenwei architecture.

3. The Hadoop transplantation method based on the domestic Shenwei processor as claimed in claim 1 is characterized in that: In the step 3, the link dependency library of the project sub-project is set, specifically: during compilation, the hadooputils module and the tirpc module are set to be linked as static libraries through the configuration file.

4. The Hadoop transplantation method based on the domestic Shenwei processor as claimed in claim 1 is characterized in that: In the step 4, the modified Hadoop source code is compiled by the Shenwei compiler, specifically: compiling into a running program based on the Shenwei architecture is selected; and an executable program for the Shenwei processor is generated by a local compilation method.

5. The Hadoop transplantation method based on the domestic Shenwei processor as claimed in claim 1, characterized in that: Also includes: In step 4: specify the required dependent warehouse through pom.xml.

6. A Hadoop transplantation system based on a domestic Shenwei processor, using the method described in any one of claims 1 to 5, characterized in that: include: The code modification module is configured to: modify the Hadoop source code so that it passes the compilation options of the Shenwei compiler; The option adding module is configured to: add the Shenwei Openeuler system compilation options and the corresponding Shenwei Openeuler system compilation rules in the compilation and linking rule file; The verification file module is configured to: add Shenwei's crc32 verification file to support the big data verification function; The code compilation module is configured as follows: under the Shenwei Openeuler system compilation option, the modified Hadoop source code is compiled by the Shenwei compiler to generate an executable program for the Shenwei processor.

7. A storage medium, characterized in that: Non-transitory storage of computer-readable instructions, wherein when the non-transitory computer-readable instructions are executed by a computer, the method according to any one of claims 1 to 5 is executed.

8. An electronic device, characterized in that: include: a memory for non-transitory storage of computer readable instructions; A processor, configured to execute the computer-readable instructions; wherein, when the computer-readable instructions are executed by the processor, the method according to any one of claims 1 to 5 is executed.

Images