Containerization construction method and system for application software

Abstract

The invention provides an application software containerization construction method and system. The system comprises a container mirror image warehouse, a software information base, a construction file generation module, a container construction module and an application test module. Wherein the container mirror image warehouse is used for storing a plurality of basic mirror images required in the application software construction process and the constructed application software mirror images; the software information base is used for storing application software construction information and assisting in generating a construction file; the construction file generation module is used for combining application software parameters specified by a user with an existing template to generate a YAML format file required by container construction; the container construction module is used for automatically completing the containerization construction process of the application software according to the construction file generated by the configuration module; and the application test module is used for completing the correctness test of the application software mirror image. The invention provides an application software container mirror image which is correct in function, equivalent in performance and small in size, and compatibility and migration of application software and a high-performance computing system are achieved.

Description

Technical Field

[0001] The present application relates to the field of high-performance computing, and specifically to a method and system for constructing application software containers. Background Art

[0002] High-performance computing is an indispensable and important means in the field of cutting-edge scientific research. However, deploying application software on high-performance computing systems faces many challenges, mainly due to the heterogeneity of the underlying hardware, the differences in parallel computing frameworks, and the sensitivity of software dependencies.

[0003] Currently, software installation on high-performance computing systems is typically performed using a package manager. A package manager is a tool used to automatically manage software packages. It resolves dependencies during application installation and simplifies software installation, updates, configuration, and uninstallation. The operating system used in high-performance computing is typically Linux, and package managers suitable for Linux systems include DNF, YUM, APT, and Spack. However, due to limitations in user permissions and network connectivity on high-performance computing systems, ordinary users cannot use package managers to complete software installation.

[0004] A container is a standardized software unit. Docker and Singularity are examples of container technologies widely used in high-performance computing. Their core concept is to package software source code and its dependencies, enabling fast and reliable migration between different system environments. Containerized application software requires customizing dependencies and writing build steps to achieve a standardized, automated build process. This can be challenging for users without relevant knowledge.

[0005] In response to the above problems, how to quickly complete the compatibility of application software with the high-performance computing system's own environment, and make the application software portable within a certain range, and provide services to users quickly and efficiently, is an issue that needs to be solved urgently. Summary of the Invention

[0006] The purpose of the present invention is to solve the above-mentioned problems existing in the prior art, realize the automatic and rapid containerization construction of high-performance computing application software, provide users with application software container images with correct functions, comparable performance and small size, and realize the compatibility and migration of application software with high-performance computing systems.

[0007] To achieve the above objectives, on the one hand, the present invention provides an application software container construction system, which includes a container image repository, a software information library, a construction file generation module, a container construction module and an application testing module.

[0008] Among them, the container image repository is used to store several basic images required in the application software construction process, as well as the application software images after the construction is completed, to facilitate image reuse and avoid repeated construction.

[0009] The software repository is used to store application software build information, including software name, software version, parallel computing framework used, compilation and installation commands, runtime environment variables, and other information. All software stored in this repository is verified for correctness. During the containerized build process, application-related information can be extracted from this repository to assist in completing the build. Furthermore, if the software information being built does not exist in the repository, after the containerized build is complete and tested, the application information is stored in the repository for subsequent use.

[0010] The build file generation module combines user-specified application software parameters with existing templates to generate the YAML format file required for container construction, which is used by the container construction module. The configuration generation module in this invention provides two modes: Basic and Advanced. The Basic build mode is suitable for commonly used application software, and the software-related information already exists in the software information library or package manager. The Advanced mode is suitable for application software with special requirements, such as the need to add specific modules during the compilation process. This mode requires the user to specify the compilation and installation commands, software source code, and test-related information.

[0011] The container build module automatically completes the containerized build process of the application software based on the build files generated by the configuration module. This module adopts a phased build approach, divided into two steps: pre-image build and application image build. The first step is to build the pre-image, complete the software compilation and installation according to the relevant configuration in the build file, analyze the software runtime environment, and generate the software runtime environment configuration file in the specified format. The second step is to build the application image, first installing the necessary general tools and auxiliary software. Then, to reduce the image size, only the libraries, executable programs, environment variable configurations, and other contents required for the software runtime are copied from the pre-image to the application image, completing the application image build.

[0012] The application testing module primarily tests the correctness of application software images. For applications not already in the software repository, the user specifies a test method. The application image is then tested according to that test method. Once the test passes, the software build is stored in the software repository.

[0013] In another aspect, the present invention provides a method for constructing application software containerization, comprising the following steps:

[0014] Stores several basic images required during the application software construction process, as well as the application software images after the construction is completed;

[0015] Store application software build information and assist in generating build files. At the same time, when the built software information does not exist in the software information library, after the containerized build is completed and passed the test, the application software information is stored in the software information library for subsequent use.

[0016] Combine the user-specified application software parameters with the existing template to generate the YAML format file required for container construction for use by the container construction module;

[0017] Automatically complete the containerized building process of the application software based on the build files generated by the configuration module;

[0018] Complete the correctness test of the application software image; for application software that does not exist in the software information library, the user must specify the test method and test the application software image according to the test method. After the test passes, the software build will be stored in the software information library.

[0019] The present invention can automatically generate container build files by specifying simple and limited build parameters, and save the verified build information for subsequent reuse; it can automatically resolve application software dependencies and complete application software installation. At the same time, only the application software runtime environment is retained in the constructed application image, effectively reducing the image size; finally, for the generated application image, automatic testing can be selected to ensure the correctness of the application software. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] In order to more clearly illustrate the technical solutions of the multiple embodiments disclosed in this specification, the following will briefly introduce the drawings required for the description of the embodiments. Obviously, the drawings described below are only the multiple embodiments disclosed in this specification. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0021] Figure 1 A schematic diagram of a system framework for building containerized application software according to an embodiment of the present invention;

[0022] Figure 2 for Figure 1 Schematic diagram of the workflow of the build file generation module;

[0023] Figure 3 for Figure 1 Schematic diagram of the container building module workflow;

[0024] Figure 4 for Figure 1 Schematic diagram of the application test module workflow. DETAILED DESCRIPTION

[0025] This specification is further described in detail below with reference to the accompanying drawings and examples. It will be understood that the specific embodiments described herein are intended only to explain the relevant invention and are not intended to limit the invention. The embodiments described herein are merely a portion of the embodiments of this specification, not all of them. All other embodiments derived by persons of ordinary skill in the art based on the embodiments in this specification without creative effort are intended to fall within the scope of protection of this application.

[0026] Figure 1 This is a schematic diagram of a framework for a containerized application software build system provided by an embodiment of the present invention. The containerized application software build system can be deployed in any environment, and the generated container image is suitable for high-performance computing systems.

[0027] like Figure 1 As shown, the application software containerized build system includes a container image repository, a software information repository, a build file generation module, a container build module, and an application testing module. The container image repository is used to store the base images required for containerized builds, as well as tested application container images. The software information repository is used to store information related to application builds and assist in generating build files. The build file generation module generates build files based on user parameters and templates and provides them to the container build module. The container build module completes the application software containerized build, forming the application software container image. The application testing module tests the application images and stores the images and build information that have passed the test in the corresponding repository.

[0028] Figure 2The schematic diagram of the workflow of the build file generation module provided in the embodiment of the present invention is that the user fills in the application software parameters, such as the software name (APP_name), software version (APP_version), parallel computing framework used (MPI_mode), build mode (BUILD_mode) and other necessary parameters in the embodiment of the present invention, wherein the build mode has two types, Basic and Advanced, and the default is Basic. In Basic mode, the build file generation module queries the software parameters filled in by the user through the package manager (software package management system) and the software information library. If the software information that matches the specified software parameters can be queried, the corresponding template is pulled to generate a YAML format build file; if the matching software information cannot be found, feedback is given and the user modifies the build parameters. In Advanced mode, first determine whether the build parameters are complete. In addition to APP_name, APP_version, and MPI_mode, it is also necessary to include the software installation command (INSTALL_cmd), source code location (SRC), environment variables (ENV), test command (TEST_cmd) and use case (TEST_case), etc. After ensuring that the build parameters are complete, the corresponding template is pulled to generate a YAML format build file. Build file templates are divided into two types: package manager installation templates and standard installation templates. In Basic mode, you need to select the package manager installation template when installing with a package manager; in other cases, select the standard installation template. The build file template includes a list of software tools required for the container build phase, initially including ssh, build-essential, and environment-modules. In Advanced mode, the list of required software tools is supplemented by extracting the installation command (INSTALL_cmd) and basic libraries (such as fftw and blas) from the environment variable (ENV).

[0029] Figure 3 The container construction module workflow diagram provided for the embodiment of the present invention adopts a phased construction method, which is divided into two stages: preliminary image construction and application image construction. In the preliminary image construction stage, the YAML format construction file formed by the construction file generation module is used to complete the software compilation and installation, analyze the application software running environment, and generate a software running environment configuration file in a specified format; in the application image construction stage, the software package is first installed according to the list of required software tools in the construction file, and then, in order to reduce the image size, only the libraries, executable programs, environment configuration files, etc. required for the software running state are copied from the preliminary image to the application image to generate the application software container image.

[0030] Figure 4The application test module workflow diagram provided for the embodiment of the present invention mainly involves first reading the test command (TEST_cmd) and test case (TEST_case) from the build file, preparing the test case. The test case can be specified as a file in the example that comes with the software source code, or given a URL to download it by itself; then starting the application image. When starting, the test case-related files need to be mounted into the container, and the test command (TEST_cmd) is run in the container. The test result is determined according to the return value. If the result is 0, the test is considered successful, and if it is not 0, the test is considered failed. After the application image passes the test, the application image is stored in the container image warehouse, and the build information is stored in the software information library for subsequent use; if it fails the test, the test result will be fed back, the cause of the error will be checked, the build parameters will be modified, and the software will be re-containerized and built.

[0031] The embodiments of the present invention simplify the application software construction process. Users only need to provide limited construction parameters to complete the container construction, which lowers the usage threshold. It can automatically handle the dependencies during the application software installation process and automatically complete the construction process without manual intervention. Compared with traditional construction methods, it is more efficient and convenient. The construction process adopts a segmented construction method, automatically extracting the application software running state information, and only retaining the running state environment in the application image, effectively reducing the image size and improving the image transmission and startup efficiency.

[0032] The specific implementation methods described above further illustrate the purposes, technical solutions and beneficial effects of the multiple embodiments disclosed in this specification. It should be understood that the above is only within the scope of this specification. Any modifications, equivalent replacements, improvements, etc. made on the basis of the technical solutions of the multiple embodiments disclosed in this specification should be included in the protection scope of the multiple embodiments disclosed in this specification.

Claims

1. A containerized application software construction system, characterized in that: include: Container image repository, software information repository, build file generation module, container building module and application testing module; among them, The container image repository is used to store several basic images required during the application software construction process, as well as the application software images after the construction is completed; The software information library is used to store application software build information and assist in generating build files. At the same time, when the built software information does not exist in the software information library, the application software information is stored in the software information library after the containerized build is completed and passed the test. The build file generation module is used to combine the user-specified application software parameters with the existing template to generate the YAML format file required for container construction for use by the container construction module; The container building module is used to automatically complete the containerized building process of the application software based on the build files generated by the configuration module; The application testing module is used to complete the correctness test of the application software image; for application software that does not exist in the software information library, the user needs to specify the test method and test the application software image according to the test method. After the test passes, the software build is stored in the software information library.

2. The method according to claim 1, characterized in that The application software construction information includes the software name, software version, parallel computing framework used, compilation and installation commands, and runtime environment variable information.

3. The method according to claim 1, wherein the construction file generation module is specifically configured to: Provides two modes: Basic and Advanced, among which, The Basic build mode is suitable for commonly used application software, and the software-related information already exists in the software information library or package manager. The Advanced mode is suitable for application software with special requirements, such as the need to add specific modules during the compilation process. This mode requires the user to specify the compilation and installation commands, software source code, and test-related information.

4. The method according to claim 1, wherein The container construction module is specifically used to: A phased construction method is adopted, which is divided into two steps: preliminary image construction and application image construction. The first step is to build the preliminary image, complete the compilation and installation of the software according to the relevant configuration in the construction file, analyze the software running environment, and generate a software running environment configuration file in the specified format; the second step is to build the application image, first install the necessary general tools and auxiliary software, and then, in order to reduce the image size, only the libraries, executable programs, and environment variable configuration contents required for the software running state are copied from the preliminary image to the application image to complete the application image construction.

5. The method according to claim 1, wherein If the application test module fails the correctness test, it will feedback the test results, check the cause of the error, modify the construction parameters, and re-containerize the software.

6. A method for constructing application software containerization, applied to the system according to claim 1, characterized in that: The following steps are involved: Stores several basic images required during the application software construction process, as well as the application software images after the construction is completed; Store application software build information and assist in generating build files. At the same time, when the built software information does not exist in the software information library, after the containerized build is completed and passed the test, the application software information is stored in the software information library for subsequent use. Combine the user-specified application software parameters with the existing template to generate the YAML format file required for container construction for use by the container construction module; Automatically complete the containerized building process of the application software based on the build files generated by the configuration module; Complete the correctness test of the application software image; for application software that does not exist in the software information library, the user must specify the test method and test the application software image according to the test method. After the test passes, the software build will be stored in the software information library.

7. The method according to claim 6, characterized in that Application software build information, including software name, software version, parallel computing framework used, compilation and installation commands, and runtime environment variable information.

8. The method according to claim 6, wherein the construction file generation module is specifically configured to: Provides two modes: Basic and Advanced, among which, The Basic build mode is suitable for commonly used application software, and the software-related information already exists in the software information library or package manager. The Advanced mode is suitable for application software with special requirements, such as the need to add specific modules during the compilation process. This mode requires the user to specify the compilation and installation commands, software source code, and test-related information.

9. The method according to claim 6, characterized in that The container construction module is specifically used to: A phased construction method is adopted, which is divided into two steps: preliminary image construction and application image construction. The first step is to build the preliminary image, complete the compilation and installation of the software according to the relevant configuration in the construction file, analyze the software running environment, and generate a software running environment configuration file in the specified format; the second step is to build the application image, first install the necessary general tools and auxiliary software, and then, in order to reduce the image size, only the libraries, executable programs, and environment variable configuration contents required for the software running state are copied from the preliminary image to the application image to complete the application image construction.

10. The method according to claim 6, characterized in that If the application test module fails the correctness test, it will feedback the test results, check the cause of the error, modify the construction parameters, and re-containerize the software.

Images