A many-core hybrid compilation method and system

Abstract

The present application relates to the technical field of software compiling installation, and provides a kind of many-core hybrid compiling method and system, comprising: obtaining the source program after function insertion of the first principle software is inserted;Compiling the source program inserted, and the master-slave core intermediate function and slave core function written, and the target program file generated after the source program is compiled, the master-slave core intermediate function target program file and the slave core function target program file are linked into executable program;And the call logic between source program, master-slave core intermediate function and slave core function is specified.The first principle software can be run on slave core to exert the computing power of slave core, and the problem of hybrid compiling, incremental compiling of first principle software programming language and many-core programming language is solved.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of software compilation and installation, and particularly relates to a many-core hybrid compilation method and system. BACKGROUND

[0002] The statements in this section merely provide background information related to the present application and do not necessarily constitute prior art.

[0003] A new generation of Shenwei processor is composed of 6 core groups, each of which contains an operation control core (main core) and an operation core array (i.e. 64 slave cores) of 8*8. This processor is also called a many-core processor.

[0004] The first principle software VASP (Vienna Ab-initio Simulation Package) is a general material calculation simulation software based on density functional theory, which mainly performs electronic structure calculation and quantum mechanics, molecular dynamics simulation. With the rapid improvement of supercomputer performance, the computing power of VASP has also been rapidly developed. However, VASP only supports commonly used processor architectures, and there are compatibility problems on Shenwei processors.

[0005] Due to the architecture of the many-core processor, the program main body of VASP can only run on the main core after compilation and installation, and the computing performance of the slave core cannot be utilized, resulting in a great waste of resources. On the other hand, VASP is written in Fortran language, while the recommended language for Shenwei processor many-core programming is C language, and the inconsistency of the languages causes great programming difficulties. SUMMARY

[0006] In order to solve the technical problems existing in the background art, the present application provides a many-core hybrid compilation method and system, which realizes that the first principle software can run on the slave core to utilize the computing power of the slave core by methods such as compiling and installing the first principle software, selecting the function insertion position, inserting the master-slave core intermediate function and the slave core function, and linking the target program file, and solves the problems of mixed compilation and incremental compilation of the first principle software programming language and the many-core programming language.

[0007] In order to achieve the above purpose, the present application adopts the following technical solutions:

[0008] The first aspect of the present application provides a many-core hybrid compilation method, which comprises:

[0009] obtaining a source program obtained after function insertion of the first principle software;

[0010] compile the source program, the master-slave core intermediate function and the slave core function, and link the object program file generated after the source program is compiled, the master-slave core intermediate function object program file and the slave core function object program file into an executable program;

[0011] The calling logic between the source program, the master-slave core intermediate function and the slave core function is as follows: the source program calls the master-slave core intermediate function at the function plug-in position, and transmits parameter data or a calculation task required by the master-slave core intermediate function into the master-slave core intermediate function; the master-slave core intermediate function initializes thread communication according to the parameter data or the calculation task, and transmits the parameter data or the calculation task to the slave core through thread communication; the slave core function obtains a calculation result according to the parameter data or the calculation task, and transmits the calculation result back to the master-slave core intermediate function through thread communication; the master-slave core intermediate function transmits a memory address where the calculation result is located back to the source program; and the source program obtains a data variable stored in the memory address.

[0012] Further, after the function plug-in position completes the function of calling the function, a flag bit indicating that the function is successfully calculated is returned.

[0013] Further, when the source program calls the slave core function, the master-slave core communication Athread thread library is used to be responsible for data transmission.

[0014] Further, when the source program plugged in is compiled, a Fortran language compiler is used for compilation.

[0015] Further, when the master-slave core intermediate function is compiled, a C language compiler is used for compilation.

[0016] Further, the master-slave core intermediate function is specified to run in a master core mode during compilation.

[0017] Further, the slave core function is specified to run in a slave core mode during compilation.

[0018] The second aspect of the application provides a many-core hybrid compilation system, which comprises:

[0019] a source program compilation module configured to obtain a source program plugged in after a first principle software is functionally plugged in;

[0020] a hybrid compilation module configured to compile the source program plugged in, the master-slave core intermediate function and the slave core function, and link the object program file generated after the source program is compiled, the master-slave core intermediate function object program file and the slave core function object program file into an executable program;

[0021] The calling logic between the source program, the master-slave kernel intermediate function and the slave kernel function is as follows: the source program calls the master-slave kernel intermediate function at the function insertion position, and transmits the parameter data or the calculation task required by the master-slave kernel intermediate function into the master-slave kernel intermediate function; the master-slave kernel intermediate function initializes thread communication according to the parameter data or the calculation task, and transmits the parameter data or the calculation task to the slave kernel through the thread communication; the slave kernel function obtains the calculation result according to the parameter data or the calculation task, and transmits the calculation result back to the master-slave kernel intermediate function through the thread communication; the master-slave kernel intermediate function transmits the memory address where the calculation result is located back to the source program; and the source program obtains the data variable stored in the memory address.

[0022] The third aspect of the present application provides a computer readable storage medium, which stores a computer program, and the program is executed by a processor to realize the steps of the many-core hybrid compilation method.

[0023] The fourth aspect of the present application provides a computer device, which comprises a memory, a processor and a computer program stored in the memory and executable on the processor, and the processor realizes the steps of the many-core hybrid compilation method when executing the program.

[0024] Compared with the prior art, the present application has the following beneficial effects:

[0025] The present application provides a many-core hybrid compilation method, which realizes that the first principle software can run on the slave core to exert the computing power of the slave core by compiling the first principle software, selecting the function insertion position, inserting the to-be-called function, linking the target program file and other methods, and solves the problems of mixed compilation and incremental compilation of the first principle software programming language and the many-core programming language.

[0026] The present application provides a many-core hybrid compilation method, which can provide incremental compilation, mixed compilation and other methods according to the compilation process of VASP and the many-core programming characteristics of the Sunway processor, provides technical support for many-core optimization and debugging of VASP based on the Sunway many-core processor, reduces the technical threshold of many-core hybrid compilation, and improves the optimization efficiency. BRIEF DESCRIPTION OF DRAWINGS

[0027] The drawings accompanying the specification of the present application serve to provide a further understanding of the present application, and the illustrative embodiments of the present application and their descriptions serve to explain the present application, and do not constitute an improper limitation on the present application.

[0028] Figure 1 A many-core hybrid compilation method flow chart for the first embodiment of the present application;

[0029] Figure 2 A VASP master core mode running diagram for the first embodiment of the present application;

[0030] Figure 3 The source program and the main kernel function relationship diagram of the first embodiment of the present application;

[0031] Figure 4 The VASP main kernel function calls the slave kernel function relationship diagram of the first embodiment of the present application. DETAILED DESCRIPTION

[0032] The present application will be further described below in conjunction with the drawings and embodiments.

[0033] It should be pointed out that the following detailed description is exemplary and is intended to provide further description of the present application. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as generally understood by those skilled in the art to which the present application belongs.

[0034] Term explanation:

[0035] Source program: refers to a text file written in accordance with certain programming language specifications without being compiled.

[0036] Object program: refers to the binary code obtained by compiling the source program. Due to the lack of library functions, it cannot be directly run on the computer.

[0037] Executable program: the complete program formed by connecting the object program and the library functions, which can be independently executed under the operating system.

[0038] Embodiment I

[0039] The present embodiment provides a many-core hybrid compilation method, which realizes that VASP can run on the slave kernel to exert the computing power of the slave kernel by compiling and installing VASP, selecting the function insertion position, inserting the to-be-called function, linking the target program file, etc., and solves the problem of hybrid compilation and incremental compilation of VASP programming language and many-core programming language. As shown in Figure 1 The specific steps include the following steps:

[0040] Step 1, compile and install VASP.

[0041] Before performing the many-core hybrid compilation, the first principle software VASP should be compiled on the main kernel, so that the main kernel running mode of VASP can run correctly, and the VASP source program is obtained. As shown in Figure 2 The compiling and installing steps are as follows:

[0042] Step 101, compile and install the mathematical library dependent on VASP.

[0043] The mathematical library dependent on VASP includes FFTW, BLAS, LAPACK, etc. basic mathematical library, which needs to be downloaded and installed in the specified directory of the Sunway processor.

[0044] Step 102, compile and install VASP.

[0045] The compilation and installation includes configuring the compiler, selecting the required mathematical library, compiling, and installing. Based on the characteristics of the compilation environment of the new generation Sunway processor, the MPI version of VASP is selected.

[0046] Step 1021, configure the compiler.

[0047] Modify the VASP compilation and installation configuration file makefile.include, for example, specify the C and Fortran compilers as the Sunway many-core processor adapted C and Fortran MPI compilers, modify the compiler specific compilation options, for example, specify the master-slave core mode, the processing method of the calculation result precision, etc.

[0048] Step 1022, select the mathematical library.

[0049] Modify the reference path of the mathematical library FFTW, BLAS, and LAPACK in the configuration file to the installation path of the mathematical library of the Sunway many-core processor.

[0050] Step 1023, compile and install.

[0051] The files in the VASP source code are compiled separately, that is, the Fortran source code is compiled into an object program, and then all the compiled files are linked to generate a VASP executable program. The purpose of generating a VASP executable program first is to check whether there is a problem in the compilation and installation, and to exclude program itself problems when performing mixed compilation later.

[0052] Step 103, example test.

[0053] After installing the program, an example test is performed to check whether the installed program has errors and whether there are compatibility problems. A calculation example is calculated using the VASP installation generated executable program, and after submitting the job, the output result is viewed to check whether there is a problem in the output result and whether the precision meets the requirements.

[0054] Step 2, select function insertion position.

[0055] The function insertion position can analyze the source program function calculation solving process, and can also be used as an entry to call other functions.

[0056] Step 201, selection of insertion position.

[0057] When performing many-core mixed compilation on the VASP source program, the appropriate insertion position should be selected to call the slave function in combination with the parameters, data dependence, function call relationship, etc. of the source program file context.

[0058] Step 202, insert the position of the function call flag bit.

[0059] After the function call function is completed at the insertion position, the flag bit flag: 1 is returned to indicate the success of the function call, otherwise 0 is set, representing the success and failure of the kernel function execution respectively. Different results of the flag bit correspond to different processing flows. In addition, the introduction of the flag bit can verify the execution result of the function at the insertion position, which facilitates the judgment of the execution of the function call.

[0060] Step 3, write the function to be called.

[0061] When performing the many-core mixed compilation, the function to be called can be divided into the master kernel function and the slave kernel function according to the different execution positions. The master kernel function can only run on the master core after compilation and use the computing power of the master core; the slave kernel function can only run on the slave core after compilation and use the computing power of the slave core.

[0062] In addition, when writing the master kernel function, the insertion position is selected first, and the method is shown in step 200. After determining the insertion position, the function call is divided into the source program calling the master kernel function and the source program calling the slave kernel function according to the different calling purposes.

[0063] Step 301, the source program calls the master kernel function.

[0064] The relationship between the source program and the master kernel function is shown in Figure 3 , which mainly extends the function of the source program. In this call, the function call follows the general calling process, that is, the function is declared first, then defined, and finally called.

[0065] Step 302, the source program calls the slave kernel function.

[0066] When the source program calls the slave kernel function, the source program and the slave kernel function need to use the master-slave core communication Athread thread library provided by the Shenwei many-core processor to transfer data due to the different execution positions. In addition, VASP is written in Fortran language, Athread programming mainly uses C language, and the programming languages are inconsistent, so the middleware function based on C language is written, which is called the master-slave kernel middleware function.

[0067] The master-slave kernel intermediate function reduces the coupling between the source program and the slave kernel function on one hand, and facilitates error troubleshooting and interface writing; on the other hand, the master-slave kernel intermediate function focuses on Athread programming, undertakes the entire life cycle of the Athread thread library, and uniformly manages the Athread thread library, thereby avoiding repeated creation of threads and conflicts between threads. The life cycle of the Athread thread library includes initialization of the thread library, creation of the slave kernel thread, waiting for the slave kernel thread to end, and destruction of the slave kernel thread.

[0068] In order to facilitate understanding of the process of the source program calling the slave kernel function, the calling relationship among the source program, the master-slave kernel intermediate function and the slave kernel function is shown in FIG. 1, and the detailed steps are as follows: Figure 4

[0069] 1) The source program calls the master-slave kernel intermediate function at the function insertion position, and transmits the parameter data or calculation task required by the master-slave kernel intermediate function into the master-slave kernel intermediate function.

[0070] 2) The master-slave kernel intermediate function receives the parameter data or calculation task of the source program, and then initializes the Athread thread communication according to the content of the parameter data or calculation task, and transmits the parameter data or calculation task to the slave kernel through the Athread thread library, and hands over the processing to the slave kernel function.

[0071] 3) The slave kernel function completes the processing according to the data or task to be calculated, and then transmits the calculation result back to the master-slave kernel intermediate function through the Athread thread communication.

[0072] 4) The master-slave kernel intermediate function receives the data transmitted back through the Athread thread communication, and then returns the memory address of the calculation result to the source program.

[0073] 5) The source program reads the memory address of the data returned by the master-slave kernel intermediate function, and obtains the data variable stored in the memory address.

[0074] Step 4, compile the code files of the source program and the master-slave kernel intermediate function and the slave kernel function inserted with the plug.

[0075] The modified source program and the master-slave kernel intermediate function and the slave kernel function are compiled by using the Fortran language and C language compilers respectively, and in addition, the code file of the slave kernel function needs to be specified as the slave kernel function on the compilation command, so as to limit it to run only on the slave kernel.

[0076] Step 401, compile the source program.

[0077] ​When compiling the patched source program, make command is not needed to be executed in software directory to compile all, only the modified source file needs to be compiled, and the Fortran language compiler self-installed in Shen Wei processor should be used for compilation. Compiling only the modified source file can not only save a lot of time, but also better locate the problems.

[0078] Step 402, compile the master-slave core intermediate function.

[0079] Since the master-slave core intermediate function is written in C language, the C language compiler self-installed in Shen Wei processor is used for compilation, and the-mhost function option is added during compilation to specify that the master-slave core intermediate function running mode is master core mode.

[0080] Step 403, compile the slave core function.

[0081] The slave core function is different from the master-slave core intermediate function in running position, so the-mslave function option is added during compilation to specify that the slave core function running mode is slave core mode.

[0082] Step 5, link the target program files generated after source program compilation, the master-slave core intermediate function target program files and the slave core function target program files into executable program.

[0083] Step 501, extract the linking command of the compiled file.

[0084] The linking command during VASP compilation installation file is extracted, the target program files after source program compilation, the master-slave core intermediate function target program files and the slave core function target program files are added in the command, and the-mhybrid function option is added in the compiler to specify that the compilation option is hybrid compilation.

[0085] Step 502, execute the linking command.

[0086] Enter the common directory of the target program files after source program compilation, the master-slave core intermediate function target program files and the slave core function target program files using command line mode, and execute the modified linking command.

[0087] The target program files of the source program, the master-slave core intermediate function target program files and the slave core function target program files are in one directory.

[0088] Step 503, move the executable program.

[0089] The linked executable program is renamed, and the version number needs to be specified to facilitate testing and maintenance, and finally the executable program is moved and copied to the directory where the example is located, and the executable program is further tested.

[0090] Step 504, command scripting.

[0091] The compiling command, the linking command and the file moving command in the above steps are packaged into a script, facilitating automatic execution of the commands and simplifying the steps.

[0092] Step 6, test and evaluate the executable program using an example to verify the effect of hybrid compilation under the many-core processor.

[0093] The example refers to a calculation instance, which can analyze the calculation result of the example on one hand, and can verify whether the compilation and installation of VASP exist errors on the other hand, and both can be verified mutually. The calculation instance of Hg is used to test the VASP after hybrid compilation of many cores, and the test steps are as follows:

[0094] Step 601, prepare the example file.

[0095] Prepare the example file of Hg, wherein the input files include: INCAR, which is mainly the control parameter file of Hg; POSCAR, which is a file describing the structure of Hg system; KPOINTS, which is a k-point sampling setting file; POTCAR: a pseudopotential file of Hg. Prepare the input file of Hg to a specified directory, and ensure that it is in the same directory as the VASP executable program.

[0096] Step 602, submit the job to the queue.

[0097] The job queue command of the Sunway many-core processor is different from other cluster commands, and when writing the submission command, the number of cores and the size of memory and other parameters need to be specified.

[0098] Step 603, view the output file.

[0099] After the job execution is completed, the log and the OUTCAR calculation result file of the example output are viewed to verify whether the above-mentioned many-core hybrid compilation method has problems.

[0100] The many-core hybrid compilation method of the embodiment can provide incremental compilation, hybrid compilation and other methods according to the compilation process of VASP and the many-core programming characteristics of the Sunway processor, provide technical support for many-core optimization and debugging work of VASP based on the Sunway many-core processor, reduce the technical threshold of many-core hybrid compilation, and improve the optimization efficiency.

[0101] Embodiment two

[0102] The embodiment provides a many-core hybrid compilation system, which specifically comprises the following modules:

[0103] The source program compiling module is configured to: obtain the source program after function insertion of the first-principle software.

[0104] The hybrid compilation module is configured to compile the instrumented source program, the compiled master-slave kernel intermediate function and the slave kernel function, and link the target program file generated after the source program is compiled, the master-slave kernel intermediate function target program file and the slave kernel function target program file into an executable program.

[0105] The calling logic between the source program, the master-slave kernel intermediate function and the slave kernel function is that the source program calls the master-slave kernel intermediate function at the function instrumented position, and transmits the parameter data or the calculation task required by the master-slave kernel intermediate function into the master-slave kernel intermediate function; the master-slave kernel intermediate function initializes thread communication according to the parameter data or the calculation task, and transmits the parameter data or the calculation task to the slave kernel through the thread communication; the slave kernel function obtains the calculation result according to the parameter data or the calculation task, and transmits the calculation result back to the master-slave kernel intermediate function through the thread communication; the master-slave kernel intermediate function transmits the memory address where the calculation result is located back to the source program; and the source program obtains the data variable stored in the memory address.

[0106] It should be noted that each module in the embodiment corresponds to each step in the first embodiment one by one, and the specific implementation process is the same, which will not be repeated here.

[0107] Embodiment three

[0108] The embodiment provides a computer readable storage medium, which stores a computer program, and the program is executed by a processor to realize the steps in the many-core hybrid compilation method in the first embodiment.

[0109] Embodiment four

[0110] The embodiment provides a computer device, which comprises a memory, a processor and a computer program stored in the memory and executable on the processor, and the processor realizes the steps in the many-core hybrid compilation method in the first embodiment when executing the program.

[0111] Those skilled in the art should understand that the embodiments of the present application can be provided as a method, a system or a computer program product. Therefore, the present application can be in the form of a hardware embodiment, a software embodiment or an embodiment combining software and hardware aspects. Moreover, the present application can be in the form of a computer program product implemented on one or more computer usable storage media (including but not limited to disk storage and optical storage, etc.) containing computer usable program code.

[0112] The embodiments of methods, apparatuses (systems) and computer program products according to the present application can be described in the general context of method steps and processes, which can be implemented in one embodiment by a program of instructions on a computer-readable storage medium executed by a computer or other programmable apparatus. The apparatuses can be specially constructed for executing the embodiments of methods, apparatuses (systems) and computer program products according to the present application or can include a computer or other programmable apparatus. Figure 1 The flow or flows and / or blocks in a flowchart and / or a block diagram Figure 1 The flow or flows and / or blocks in a flowchart and / or a block diagram

[0113] The computer program instructions can also be loaded onto a computer or other programmable apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer-implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the flow Figure 1 The flow or flows and / or blocks in a flowchart and / or a block diagram Figure 1 The flow or flows and / or blocks in a flowchart and / or a block diagram

[0114] The computer program instructions can also be loaded onto a computer or other programmable apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer-implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the flow Figure 1 The flow or flows and / or blocks in a flowchart and / or a block diagram Figure 1 The flow or flows and / or blocks in a flowchart and / or a block diagram

[0115] Those skilled in the art can understand that all or part of the flow of the above-mentioned embodiment method can be completed by computer program instructions instructing related hardware, and the program can be stored in a computer-readable storage medium. When the program is executed, it can include the flow of the above-mentioned embodiment of each method. The storage medium can be a magnetic disc, an optical disc, a read-only memory (ROM) or a random access memory (RAM), etc.

[0116] The above only describes the preferred embodiments of the present application and is not intended to limit the present application. For those skilled in the art, the present application can have various changes and modifications. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included in the protection scope of the present application.

Claims

1. A many-core hybrid compilation method, characterized by, The method comprises the following steps: obtaining a source program after function insertion of a first principle software; compiling the inserted source program, the master-slave core intermediate function and the slave core function, and linking the target program file generated after the source program is compiled, the master-slave core intermediate function target program file and the slave core function target program file into an executable program; when the source program is compiled, the appropriate insertion position is selected to call the slave core function in combination with the parameters, data dependency and function call relationship of the context of the source program file; when the inserted source program is compiled, only the modified source file needs to be compiled; wherein the calling logic among the source program, the master-slave core intermediate function and the slave core function is that the source program calls the master-slave core intermediate function at the function insertion position, and transmits the parameter data or calculation task required by the master-slave core intermediate function into the master-slave core intermediate function; the master-slave core intermediate function initializes thread communication according to the parameter data or calculation task, and transmits the parameter data or calculation task to the slave core through thread communication; the slave core function obtains the calculation result according to the parameter data or calculation task, and transmits the calculation result back to the master-slave core intermediate function through thread communication; the master-slave core intermediate function transmits the memory address of the calculation result back to the source program; the source program obtains the data variable stored in the memory address; after the function calling function at the insertion position is completed, a flag bit indicating that the function calling is successful is returned; different results of the flag bit correspond to different processing flows.

2. The many-core hybrid compilation method of claim 1, wherein, When the source program calls the slave core function, the master-slave core communication Athread library is used to transmit data.

3. The many-core hybrid compilation method of claim 1, wherein, When the inserted source program is compiled, a Fortran language compiler is used for compilation.

4. The many-core hybrid compilation method of claim 1, wherein, When the master-slave core intermediate function is compiled, a C language compiler is used for compilation.

5. The many-core hybrid compilation method of claim 1, wherein, When the master-slave core intermediate function is compiled, the running mode of the master-slave core intermediate function is specified as the master core mode.

6. The many-core hybrid compilation method of claim 1, wherein, When the slave core function is compiled, the running mode of the slave core function is specified as the slave core mode.

7. A many-core hybrid compilation system, characterized by, The method comprises the following steps: a source program compiling module configured to obtain a source program after function insertion of a first principle software; a mixed compiling module configured to compile the inserted source program, the master-slave core intermediate function and the slave core function, and link the target program file generated after the source program is compiled, the master-slave core intermediate function target program file and the slave core function target program file into an executable program; when the source program is compiled, the appropriate insertion position is selected to call the slave core function in combination with the parameters, data dependency and function call relationship of the context of the source program file; when the inserted source program is compiled, only the modified source file needs to be compiled; The calling logic between the source program, the master-slave kernel intermediate function and the slave kernel function is as follows: the source program calls the master-slave kernel intermediate function at the function plug-in position, and transmits parameter data or a calculation task required by the master-slave kernel intermediate function into the master-slave kernel intermediate function; the master-slave kernel intermediate function initializes thread communication according to the parameter data or the calculation task, and transmits the parameter data or the calculation task to the slave kernel through the thread communication; the slave kernel function obtains a calculation result according to the parameter data or the calculation task, and transmits the calculation result back to the master-slave kernel intermediate function through the thread communication; the master-slave kernel intermediate function returns a memory address where the calculation result is located to the source program; the source program acquires a data variable stored in the memory address; after the function at the plug-in position is called and the function is completed, a flag bit indicating that the function is successfully calculated is returned; for different results of the flag bit, different processing flows are entered.

8. A computer-readable storage medium having stored thereon a computer program, characterized in that, The program is executed by the processor to implement the steps in the many-core hybrid compilation method according to any one of claims 1-6.

9. A computer device comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, characterized in that, The processor executes the program to implement the steps in the many-core hybrid compilation method according to any one of claims 1-6.

Images