High-Performance Computing Platform Application Packaging Method, System, and Storage Medium

Abstract

The present invention discloses a method, system and storage medium for packaging applications of a high-performance computing platform. By downloading the source code of the application software, the source code is compiled into a binary file; a running specification file of the application software is written; the binary file and the running specification file are packaged together to obtain the packaged application software; the packaged application software is uploaded to the central application server. Each application corresponds to a running specification file. Through the corresponding running specification file, the application and the corresponding dependent programs can be downloaded, and then the corresponding loading and configuration programs are run, enabling the software to be directly used in the downloaded machine, eliminating the cumbersome process of compilation and installation, saving a lot of time and effort. Through this method, the software can be directly run on the corresponding operating system and the corresponding hardware architecture. While reducing the cumbersome compilation process, the original running efficiency of the software is maximally retained, providing great convenience for software management.

Description

Technical Field

[0001] The present invention relates to the field of high-performance computing, and particularly to a method, a system and a storage medium for packaging applications on a high-performance computing platform. Background Art

[0002] In a high-performance computing platform, in order to meet various computing requirements, a variety of application software needs to be installed. In the case of multiple computing clusters, each computing cluster has to be recompiled once. When compiling, a lot of dependencies and environments are required, and there are many error reports, which is very cumbersome. Most of the software used in high-performance computing does not support cross-platform. Different processor architectures, such as X86_64, ARM, AMD64, have different instruction sets, which also results in that the applications compiled on the current processor architecture cannot be moved to another processor architecture for use. Therefore, it is necessary to compile and install multiple times on machines with different processors.

[0003] Nowadays, popular container technologies such as docker can provide the same running environment in computing clusters with different processor architectures, enabling software to be used across platforms without depending on the instruction set of the processor. However, in high-performance computing scenarios, container technologies have significant performance losses and cannot achieve the running efficiency of direct compilation and installation, resulting in few users.

[0004] In addition, software package management tools provided by Linux, such as RPM, require many system dependencies. There are many problems when installing them independently into a certain directory and they may not be able to run directly. Moreover, there are no corresponding RPM software packages for open-source software. Summary of the Invention

[0005] The main purpose of the present invention is to provide a method, a system and a storage medium for packaging applications on a high-performance computing platform, so as to solve the problem of the cumbersome process of installing applications in a high-performance computing cluster, save time and effort, enable the software to run directly on the corresponding operating system and the corresponding hardware architecture, and retain the original running efficiency of the software to the greatest extent while reducing the cumbersome compilation process, and also provide great convenience for software management.

[0006] To achieve the above object, the present invention proposes a method for packaging applications on a high-performance computing platform, and the method includes the following steps:

[0007] Download the source code of the application software and compile the source code into a binary file;

[0008] Write the running specification file of the application software;

[0009] Package the binary file and the running specification file together to obtain the packaged application software;

[0010] Upload the packaged application software to the central application server.

[0011] Among them, the method further includes:

[0012] Unpack the packaged application software through the central application server, obtain the corresponding running specification file, then read the ResourceUrl, store the software package in the corresponding path, and save the running specification file.

[0013] Among them, the steps of downloading the source code of the application software and compiling the source code into a binary file include:

[0014] Download the source code of the application software and configure various required dependency conditions;

[0015] Execute the compilation command to compile the source code into a directly usable binary file.

[0016] Among them, the method further includes:

[0017] When receiving an application download command, send the application download command to the central application server;

[0018] Download and install the dependent software of the application from the central application server;

[0019] Download the application from the central application server and unpack the application;

[0020] Execute the setup script to configure the software.

[0021] Among them, the steps of downloading and installing the dependent software of the application from the central application server include:

[0022] Find the corresponding running specification file through the central application server and read the dependent software;

[0023] Download the dependent software of the application from the central application server.

[0024] Among them, the steps of downloading the dependent software of the application from the central application server include:

[0025] Download the dependent software of the application from the central application server. After reading the software dependency Requires in the running specification file, traverse the dependency list, and then install the dependent software one by one. Among them, when the dependent software has dependencies, first install the dependent software of the current dependent software, then install the current dependent software after installation, and finally install the current software.

[0026] Among them, the steps of downloading the application from the central application server and unpacking the application include:

[0027] Read the software storage path in the operation specification file from the central application server and perform download and application.

[0028] Read the software storage path in the operation specification file from the central application server and unpack the application software into the corresponding folder.

[0029] Among them, the methods for uploading files to the central application server and downloading files from the central application server include: scp, ftp, http; the forms of the operation specification file include: json, xml, yml.

[0030] The present invention also proposes a high-performance computing platform application packaging system, which includes: a memory and a processor. A computer program is stored on the memory, and when the computer program is executed by the processor, the steps of the method described above are implemented.

[0031] The present invention also proposes a computer storage medium, on which a computer program is stored, and when the computer program is executed by the processor, the steps of the method described above are implemented.

[0032] Compared with the prior art, for the high-performance computing platform application packaging method, system and storage medium proposed by the present invention, each application corresponds to an operation specification file. Through the corresponding operation specification file, the application and the corresponding dependent programs can be downloaded. After the download is completed, the corresponding loading and configuration programs are run to enable the software to be directly used in the downloaded machine, eliminating the cumbersome process of compilation and installation, saving a lot of time and effort. Through this method, the software can be directly run on the corresponding operating system and the corresponding hardware architecture, and while reducing the cumbersome compilation process, the original operating efficiency of the software is maximally retained, providing great convenience for software management.

[0033] Compared with the prior art, the solution of the present invention solves the cumbersome process of installing applications in a high-performance computing cluster. First, it does not require compiling the application and can be used after the download is completed. Second, after the installation is completed, it will configure the software to enable it to be used on the target machine. In addition, it maximally retains the original performance of the application, which is very important in a high-performance computing cluster and can better exert the performance of the machine. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] Figure 1 It is a schematic flowchart of the high-performance computing platform application packaging method of the present invention;

[0035] Figure 2 It is a general flowchart of the high-performance computing platform application packaging method of the present invention;

[0036] Figure 3 Schematic diagram of the process of downloading an application according to the present invention;

[0037] Figure 4 Schematic diagram of the process of resolving dependencies when downloading dependent software of an application according to the present invention. Detailed implementation manners

[0038] It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.

[0039] Referring to Figures 1-4 As shown, the present invention provides a method for packaging applications on a high-performance computing platform, and the method includes the following steps:

[0040] S10, downloading the source code of the application software and compiling the source code into a binary file;

[0041] Download the source code of the application software and configure various required dependency conditions;

[0042] Execute the compilation command to compile the source code into a directly usable binary file.

[0043] S20, writing a running specification file for the application software;

[0044] S30, packaging the binary file and the running specification file together to obtain the packaged application software;

[0045] S40, uploading the packaged application software to the central application server.

[0046] Further, the method further includes:

[0047] Unpacking the packaged application software through the central application server to obtain the corresponding running specification file, then reading the ResourceUrl, storing the software package in the corresponding path, and saving the running specification file.

[0048] Further, the method further includes:

[0049] When receiving an application download command, sending the application download command to the central application server;

[0050] Downloading and installing the dependent software of the application from the central application server;

[0051] Downloading and unpacking the application from the central application server;

[0052] Executing the setup script to configure the software.

[0053] Among them, the steps of downloading and installing the dependent software of the application from the central application server include:

[0054] Find the corresponding running specification file through the central application server and read the dependent software;

[0055] Download the dependent software of the application from the central application server.

[0056] Among them, the steps of downloading the dependent software of the application from the central application server include:

[0057] Download the dependent software of the application from the central application server. After reading the software dependency Requires in the running specification file, traverse the dependency list, and then install the dependent software one by one. Among them, when a dependent software has a dependency, first install the dependent software of the current dependent software, and then install the current dependent software after installation, and finally install the current software.

[0058] Among them, the steps of downloading the application from the central application server and unpacking the application include:

[0059] Read the software storage path in the running specification file from the central application server and download the application;

[0060] Read the software storage path in the running specification file from the central application server and unpack the application software into the corresponding folder.

[0061] Among them, the methods of uploading files to the central application server and downloading files from the central application server include: scp, ftp, http, etc.; the forms of the running specification files include: json, xml, yml, etc.

[0062] Compared with the prior art, for the application packaging method, system and storage medium of the high-performance computing platform proposed by the present invention, each application corresponds to a running specification file. Through the corresponding running specification file, the application and the corresponding dependent programs can be downloaded. After the download is completed, run the corresponding loading and configuration programs so that the software can be directly used in the downloaded machine, eliminating the cumbersome process of compilation and installation, saving a lot of time and effort. Through this method, the software can be directly run on the corresponding operating system and the corresponding hardware architecture. While reducing the cumbersome compilation process, the original running efficiency of the software is retained to the greatest extent, providing great convenience for software management.

[0063] The following elaborates on the solution of the present invention in detail:

[0064] The application packaging method of the present invention is as Figure 2 shown:

[0065] InFigure 2 In this case, the entire process of packaging the application is shown, specifically including:

[0066] S1: Download the source code of the software, configure various required dependencies, and then execute the compilation command to compile it into a directly usable binary file.

[0067] S2: Write the run specification file run.spec for the software. The run specification file can have various forms, such as json, xml, yml, etc. It mainly has the following content:

[0068]

[0069] S3: Package the application. Package the binary file directory generated by compiling the software and the run specification file (run.spec) together. There are many ways to package, for example, it can be packaged in a compressed manner, and then the corresponding compressed file tar.gz is generated. The name of the compressed file can generally be defined as "software name_version_CPU architecture_package format suffix". For example, python3_3.10_X86_.tar.gz.

[0070] S4: Upload to the central application server. Upload the packaged software to the central application server. There are various ways to upload, such as scp, ftp, http, etc. After uploading to the central application server, the central application server will unpack it, obtain the corresponding run specification file, then read the ResourceUrl, and store the software package in the corresponding path. At the same time, the central application server will save the content of the spec. There are many ways to save, such as storing it in a database.

[0071] The above is the complete process of packaging.

[0072] After that, the application software can be downloaded.

[0073] The download operation process is as Figure 3 shown:

[0074] In Figure 3 this case, the entire process of downloading the application is shown, specifically including:

[0075] S1: Enter the command to download the application. For example, enter "app install python_3.9_X86", and then the central application server will search for the corresponding run specification run.spec file and read the software dependencies.

[0076] S2: Download the dependent software of the application from the central application server. After reading the software dependencies Requires in run.spec, traverse the dependency list and then install the dependent software one by one. When the dependent software also has dependencies, first install the dependencies of the dependent software, and then install the dependent software after the installation is complete. Finally, install the current software. Among them, the process of resolving dependencies can refer to Figure 4 as shown.

[0077] S3: Download the application from the central application server. After reading the software storage path (ResourceUrl) in run.spec, download the application. Download methods such as http, ftp, and scp can be used.

[0078] S4: Unpack the application. After the application is downloaded, read the software storage path Path in the run specification file run.spec and unpack it into the corresponding folder. There are various ways to implement unpacking, such as decompressing files (unzip, tar, rar).

[0079] S5: Execute the setup script. After the software is installed, execute the setup script to configure the software. Only after the software is configured can it run. For example, for an Oracle database, it needs to be configured before starting the database. This step usually involves modifying the software configuration file. The setup script can be a few commands or a path pointing to a shell script.

[0080] Through the above steps, the software can be downloaded and then run.

[0081] If the user wants to use the software, then the ImportScript script in run.spec also needs to be executed. This step will import the software into the environment variables, allowing the user to directly use the binary files of the software.

[0082] The above packaging method solves the cumbersome process of installing applications in a high-performance computing cluster. First, it does not require compiling the application and can be used after downloading. Second, after installation, it will configure the software so that it can be used on the target machine. In addition, it also retains the original performance of the application to the greatest extent, which is very important in a high-performance computing cluster and can better utilize the performance of the machine.

[0083] It should be noted that in other embodiments, the content of the run specification file run.spec can also be modified, such as adding or reducing content, or merging multiple contents into one; in other embodiments, the packaging method can also be modified to avoid compression in other ways; in other embodiments, the entire process of downloading the application can also be modified.

[0084] In addition, the present invention also provides a high-performance computing platform application packaging system, which includes: a memory and a processor. A computer program is stored on the memory, and when the computer program is executed by the processor, the steps of the method described above are implemented.

[0085] The present invention also provides a computer storage medium, on which a computer program is stored, and when the computer program is executed by the processor, the steps of the method described above are implemented.

[0086] Compared with the prior art, for the high-performance computing platform application packaging method, system and storage medium provided by the present invention, each application corresponds to a running specification file. Through the corresponding running specification file, the application and the corresponding dependent programs can be downloaded. After the download is completed, the corresponding loading and configuration programs are run, enabling the software to be directly used in the downloaded machine, eliminating the cumbersome process of compilation and installation, saving a large amount of time and effort. Through this method, the software can be directly run on the corresponding operating system and the corresponding hardware architecture. While reducing the cumbersome compilation process, the original running efficiency of the software is maximally retained, providing great convenience for software management.

[0087] Compared with the prior art, the solution of the present invention solves the cumbersome process of installing applications in a high-performance computing cluster. First, it does not require compiling the application and can be used after the download is completed. Second, after the installation is completed, it will configure the software to enable it to be used on the target machine. In addition, it maximally retains the original performance of the application, which is very important in a high-performance computing cluster and can better exert the performance of the machine.

[0088] The above are only the preferred embodiments of the present invention, and do not limit the patent scope of the present invention accordingly. Any equivalent structural or equivalent process transformation made by using the specification and drawings of the present invention, or directly or indirectly applied in other related technical fields, shall be similarly included in the patent protection scope of the present invention.

Claims

1. A method for packaging applications on a high-performance computing platform, characterized in that, the method comprises the following steps: Download the source code of the application software and compile the source code into a binary file; Write the running specification file of the application software; Package the binary file and the running specification file together to obtain the packaged application software; Upload the packaged application software to the central application server; The method further comprises: Unpack the packaged application software through the central application server, obtain the corresponding running specification file, then read the ResourceUrl, store the software package in the corresponding path, and save the running specification file; The step of downloading the source code of the application software and compiling the source code into a binary file comprises: Download the source code of the application software and configure various required dependency conditions; Execute the compilation command to compile the source code into a directly usable binary file; The method further comprises: modifying the content of the running specification file, including adding or reducing content, or merging multiple contents into one; The method further comprises: When receiving an application download command, send the application download command to the central application server; Download and install the dependent software of the application from the central application server; Download the application from the central application server and unpack the application; Execute the setup script to configure the software; The step of downloading the dependent software of the application from the central application server comprises: Download the dependent software of the application from the central application server, after reading the software dependencies Requires in the running specification file, traverse the dependency list, and then install the dependent software one by one. Among them, when a dependent software has a dependency, first install the dependent software of the current dependent software, then install the current dependent software after installation, and finally install the current software.

2. The method according to claim 1, characterized in that, the step of downloading and installing the dependent software of the application from the central application server comprises: Find the corresponding running specification file through the central application server and read the dependent software; Download the dependent software of the application from the central application server.

3. The method according to claim 2, characterized in that, the step of downloading the application from the central application server and unpacking the application comprises: Read the software storage path in the running specification file from the central application server to download the application; Read the software storage path in the running specification file from the central application server and unpack the application software into the corresponding folder.

4. The method according to claim 2, characterized in that, The ways of uploading files to the central application server and downloading files from the central application server include: scp, ftp, http; the forms of the running specification file include: json, xml, yml.

5. A high-performance computing platform application packaging system, characterized in that, The system includes: a memory and a processor, and a computer program is stored on the memory. When the computer program is executed by the processor, the steps of the method according to any one of claims 1-4 are implemented.

6. A computer storage medium, characterized in that a computer program is stored on the computer storage medium, and when the computer program is executed by the processor, the steps of the method according to any one of claims 1-4 are implemented.

Images