A Repack image file packaging method to solve the running dependency of HPC applications

Abstract

The present invention belongs to the technical field of electronic digital data processing, and more specifically, to a Repack image file encapsulation method for solving the running dependency of HPC applications. The method packages the HPC high-performance computing application and all its runtime dependencies into an independent executable image file, and publishes it in the form of a statically linked executable file. The file has its own container engine and can be run directly, solving the dependency problem encountered in deploying HPC software in different system environments. The method not only supports mainstream architectures, but is also compatible with HPC computing environments. In addition, the Repack image file can be recompiled and optimized to meet the computing needs of different users.

Description

Technical Field

[0001] The invention belongs to the technical field of electronic digital data processing, and in particular relates to a Repack image file encapsulation method for solving the running dependency of HPC applications. Background Art

[0002] With the rapid development of computing technology, high-performance computing has become an important means of scientific research. More and more disciplines use high-performance computing to simulate and emulate scientific problems, improve scientific research efficiency, and reduce experimental costs. However, high-performance computing has many application areas, various application software is growing, and software version iteration is even faster. How to effectively improve software installation, management, and existing dependency issues is a huge challenge.

[0003] At present, the relatively mature and reliable software package management tools yum and apt in Linux systems have greatly solved the problem of solving multiple dependencies during the installation and deployment of applications. However, in the field of high-performance computing, professional software usually has the need for multiple versions to coexist. The same user uses different versions of the same software for comparative experiments, or different users use different versions of the same software for their own research. However, the design of yum or apt is not aimed at this scenario. Only one version of the same software is allowed to exist in the system at the same time. Installing a higher version will overwrite the old version of the software in the replacement system.

[0004] In response to these problems in the installation of application software in the field of high-performance computing, experts in the field of high-performance computing have launched package management tools designed specifically for HPC scenarios, such as Spack from Lawrence Livermore National Laboratory and Easybuild from Ghent University. Both have been widely accepted and applied in many supercomputing centers. By automatically resolving build dependencies, they have achieved automatic compilation and deployment of high-performance computing application software, greatly reducing the difficulty of application deployment. However, both have high requirements for users, high learning costs, and require knowledge of complex compilation options. In addition, when Spack has a large number of dependencies, it still takes a relatively long time to build a software package from scratch, and in some cases, the new version of the library may be incompatible with the existing software package or may be unable to obtain network resources, requiring users to manually intervene to solve these problems.

[0005] In response to various problems in HPC scenarios, virtual environments, modern package management tools, and container technologies have been introduced into scientific computing. Conda can create multiple isolated virtual environments within the system, allowing users to independently manage and control the dependencies in each environment. Docker and Singularity allow applications and their dependent environments to be packaged in an isolated container, thereby providing a consistent operating environment on different platforms. However, these methods all have defects. For example, although Conda supports cross-platform, some packages may have compatibility issues on specific platforms. When executing MPI parallel programs, the Singularity container has strict requirements on the MPI environment inside and outside the container, and has a certain dependence on the user's own system environment. Since many container repositories and github repositories are located abroad, domestic network access is unstable, and some links to old versions of software have problems such as failure, resulting in the inability to guarantee image repeatability. In addition, there are a large number of traditional codes and executable files left over from history in the field of high-performance computing. In addition to complex compiler dependencies and software library dependencies, they also have dependencies on the bottom-level glibc of the system. Glibc involves many system components, so it is generally impossible to deploy a higher version of glibc on the platform as a whole, and the package management tools built on the operating system are still unable to manage and replace the system-level glibc library. Users who need a higher version of glibc can only install it themselves through the source code, which is highly professional and difficult for general users to operate.

[0006] None of the above methods and tools can solve the long-standing problem of complex software installation and configuration dependencies in the HPC field, especially the need to install its own software environment, which increases the complexity of software deployment. There is currently no patent for container encapsulation that solves application dependencies in the HPC application field. Chinese patent document CN 113986395A disclosed a patent on June 23, 2023, which solves glibc dependency by modifying the path in the dynamic linker elf file, but in addition to glibc, many applications are dependent on python, and there is no way to solve the dependency of self-generated software on the operating system when it is deployed. Chinese patent document CN 110543311 directly uses singularity to encapsulate container images, but singularity itself has system dependency problems when deployed.

[0007] In view of this, the present invention designs a Repack image file packaging method for solving the running dependency of HPC applications. Summary of the invention

[0008] In order to overcome at least one defect of the above-mentioned prior art, the present invention provides a Repack image file packaging method for solving the running dependencies of HPC applications. The method encapsulates the dependency files and application files of high-performance computing HPC applications together and packages them into an independent executable repack image file, thereby providing a solution independent of the running environment and effectively solving the problems of dependency complexity, environmental consistency, dependency conflict and version control, portability and reusability, and maintenance cost and efficiency.

[0009] The detailed technical scheme of the present invention is as follows:

[0010] A Repack image file encapsulation method for solving HPC application running dependencies, the method comprising:

[0011] S1. Software analysis: Perform dependency analysis on the target HPC application software using static and dynamic analysis techniques, and identify all direct and indirect dependent libraries, configuration files, compilers, tool chains, and runtime environments through automated scripts;

[0012] S2. Collection of installation files: Based on the analysis results of step S1, an automated package management script is used to automatically collect all third-party software and source code installation packages that the target HPC application depends on;

[0013] S3. Image preparation: Generate an installation file script using automated script generation technology based on the target application's installation rules. The generated installation file script is based on the target application's dependencies and installation order, and uses a template generation system and dynamically adjusted configuration files to ensure that all dependencies can be installed in the correct order.

[0014] The automated script generation technique utilizes a graph algorithm based on target application dependencies;

[0015] S4. Image creation system preparation: In a Linux environment with root privileges, use automated scripting technology to create images. At the same time, use automatic permission verification and configuration mechanisms to automatically detect and repair permission issues during the image creation process.

[0016] S5. Target HPC application image structure: Using hierarchical directory structure technology, create a hierarchical directory structure of the target HPC application under the image production system, and place the dependencies of each application in a directory isolated from the operating system;

[0017] S6. Create a basic image: Download the basic operating system image and generate a head file in the process. The head file implements the mounting of the fuse file system and the startup of the container engine.

[0018] S7. System directory binding: mount the current system's / dev / proc file system to the directory rootdir;

[0019] S8. Create an environment variable file: Generate an environment variable script repack.sh, which automatically loads the environment variables required by the application when the image is started, and makes necessary path adjustments;

[0020] The configuration of the environment variables is dynamically generated, and can automatically generate and adjust environment variables according to the needs of different applications;

[0021] S9. Write application installation shell script: Use the automated script generation tool to generate the installation process script of the target HPC application and automatically monitor the system environment;

[0022] S10. Image entry creation: Encapsulate the application installation shell script into repack.sh, and create an entry script run through automated script generation technology. The script run loads environment variables and determines how to run the application based on the parameters passed in by the specific HPC application.

[0023] S11. squashfs file encapsulation: Use the mksquashfs tool to package the rootdir directory into a compressed squashfs file system, that is, use the squashfs efficient file system tool to compress the dependent files and application files into a read-only file system;

[0024] S12. Image merging: Use the versioning management mechanism and modularization mechanism to merge the generated squashfs file system and the head header file generated when creating the base image to generate the final repack image;

[0025] The version management mechanism means that each image file is accompanied by complete version information to facilitate version control and update management;

[0026] The modular management means that the repack image includes a head header file module and a squashfs file system module, each module can be updated and maintained independently, the head header file module provides the container engine file required for the image runtime, and the squashfs file system module contains all the files and directories required for the system operation;

[0027] S13. Image naming: Name the image to distinguish the target HPC application version number.

[0028] Furthermore, the above method supports importing singularity images, directly converting singularity images into repack images. The conversion steps of singularity images are as follows:

[0029] 1) Get the rootfs file system of the singularity image;

[0030] 2) Unzip the rootfs file system to the directory corresponding to each application;

[0031] 3) Delete the environment variables in the rootfs file system;

[0032] 4) Create a repack image to automatically start the default application;

[0033] 5) Package the above rootfs file system directory into squashfs;

[0034] 6) Merge the squashfs and head files to generate a repack image.

[0035] Furthermore, the repack image not only supports the X86 architecture, but is also compatible with the ARM architecture and other specific HPC computing environments; this cross-architecture support ensures that HPC applications can be seamlessly migrated and deployed on various hardware platforms, greatly improving the portability of applications.

[0036] Furthermore, the static and dynamic analysis technology refers to: first, using static code analysis tools to check unused dependency packages, check whether the dependencies used in the project are compatible, and whether there are version conflicts, and at the same time use runtime monitoring tools to analyze the application's dependencies and dynamically analyze the communication and performance of the MPI program, and track the usage of its dependency packages while the application is running, so as to identify which dependencies are active and which are redundant.

[0037] Furthermore, the automated package installation script includes error detection and handling mechanisms, configuration settings, and various software dependency package security detection mechanisms for automated software package installation and configuration.

[0038] Furthermore, the automated script generation technology in step S3 utilizes a graph algorithm based on target application dependencies, specifically:

[0039] First, create a directed graph containing all software packages and their dependencies, where nodes represent software packages and directed edges represent dependencies, and output an optimized list of software package loading order;

[0040] The graph is then topologically sorted to determine a linear order so that each package is loaded before its dependencies, which is achieved through the Kahn algorithm or depth-first search DFS. During the topological sorting process, if a circular dependency is detected, that is, the sorting cannot be completed, an error is reported and the user is prompted to resolve these circular dependencies. For dependencies with the same package name but different versions, a version conflict resolution strategy is used.

[0041] Finally, based on the results of topological sorting and version conflict resolution, the final software package loading order list is generated.

[0042] Furthermore, the intelligent automatic mounting mechanism refers to: using system API or script to detect newly connected devices, intelligently allocating mount points and mount options according to the characteristics of the storage device and the current status of the system, while detecting the file system used by the connected storage device, and attempting automatic recovery or remounting when an abnormality is detected.

[0043] Furthermore, the repack image has a built-in lightweight container engine, so users can run HPC applications without installing an additional container environment. This built-in engine design greatly simplifies the application deployment process and ensures the consistency of applications in different environments.

[0044] Furthermore, the repack image adopts an "out-of-the-box" design, which does not require users to perform complex configuration or compilation operations. The image file can be directly run on any supported HPC platform, and users only need simple command line operations to start and use the application.

[0045] Furthermore, the repack image file is released as a statically linked binary executable file.

[0046] Compared with the prior art, the present invention has the following beneficial effects:

[0047] (1) The present invention identifies all dependencies through static and dynamic analysis methods, then automatically collects installation packages and generates installation scripts, encapsulating the installation dependencies of all HPC application software in an independent Repack image, so that users can quickly deploy applications without manual intervention. The Repack image automated deployment of the present invention not only saves time and reduces operational complexity, but also ensures the consistency of the application running environment on different nodes, effectively avoiding the problem of unstable operation caused by environmental differences, and improving deployment efficiency.

[0048] (2) The present invention creates directory structures and environment configurations that adapt to different platforms for different hardware architectures and computing environments, and fully considers the compatibility of different hardware architectures, such as X86 and ARM, and different HPC computing environments, such as GPU and Infiniband, through system directory / dev binding, to ensure that target applications can be seamlessly migrated and deployed on different platforms without recompiling or adjusting the software.

[0049] (3) The present invention converts applications managed by existing tools and their dependencies into the Repack image format through an automated conversion tool, and ensures that the image format is compatible with container platforms such as Docker and Singularity. Users can easily convert applications managed by these tools into Repack images and run them directly. This compatibility expands the scope of application of Repack images, improves the flexibility and portability of images, and provides users with more choices and convenience.

[0050] (4) The present invention designs a multi-level directory structure, and each version of the application and its dependencies are independently packaged. Users can run multiple versions of applications on the same system without worrying about conflicts between versions. The Repack image of the present invention supports the coexistence of multiple versions of HPC applications, which is particularly suitable for scenarios where the performance or compatibility of different versions needs to be compared.

[0051] (5) The present invention ensures that all files and permissions in the image are set correctly through the environment configuration of root permissions, and further reduces the interference of the external environment on the application operation by using the Squashfs file system to compress the image. The present invention ensures that the application can still maintain stable operation after a long period of operation by completely encapsulating the runtime environment, avoiding the impact of external environmental changes on the application, and improving the reliability and stability of operation.

[0052] (6) The Repack image of the present invention supports secondary compilation and optimization. By creating a dynamically adjustable environment variable file repack.sh and an installation script, unlike the standard compilation environment provided by the prior art, users can adjust compilation options, optimize algorithms, improve I / O performance, etc., and customize the image to meet specific needs. The present invention allows users to adjust the configuration and performance of applications in the image according to their own needs, which is particularly suitable for the optimization of complex computing tasks and improves computing efficiency.

[0053] (7) The present invention ensures that the dependencies of each image file are complete and consistent through intelligent dependency management and environment configuration, avoiding calculation errors or inconsistent results caused by platform differences or different dependencies. Through the packaging of Repack images, users can ensure that all dependencies are uniformly managed and packaged in the image, thereby eliminating the impact of environmental differences on calculation results and ensuring the rigor and credibility of scientific research. BRIEF DESCRIPTION OF THE DRAWINGS

[0054] Figure 1 This is a flow chart of a Repack image file encapsulation method for resolving HPC application running dependencies described in the present invention.

[0055] Figure 2 It is a schematic diagram of the Repack mirroring architecture of the present invention.

[0056] Figure 3 The results of running the repack image on three different models of GPU servers in the embodiment of the present invention are shown. DETAILED DESCRIPTION

[0057] The technical concept of the present invention is:

[0058] The installation dependencies required by the target HPC application software are encapsulated in an image called Repack. By writing an installation script, the Repack image file is automatically generated. The file is released as a static binary file, and the user does not need to install any additional dependent software. The application can be directly run in different computing environments. Relying on this method, the installation and operation of HPC application software can be realized without relying on any system environment, and the HPC application software can get rid of the dependence on the glibc version. It can manage various HPC application software with different versions and different dependencies on large-scale supercomputing platforms. This method can not only maintain the repeatability of HPC software calculations, but also perform secondary compilation and optimization according to user needs. This method also supports the import of spack, conda, and singularity software management tools. In addition to supporting the current mainstream X86 architecture, Repack can also run on ARM architecture CPUs and is also compatible with HPC computing environments such as GPU, Infiniband, and resource management.

[0059] Embodiment 1,

[0060] like Figure 1 As shown, this embodiment discloses a Repack image file packaging method for solving the running dependency of HPC applications, including:

[0061] S1. Software Analysis: Dependency analysis of target HPC applications is performed using static and dynamic analysis techniques. This analysis identifies all directly and indirectly dependent library files, configuration files, compilers, tool chains, and runtime environments through automated scripts (e.g., based on the combined use of the ldd command and the strace tool). The present invention combines static and dynamic analysis methods to capture not only explicit dependencies, but also implicit dependencies through runtime behavior, thereby ensuring the integrity of dependencies. This technical approach can significantly improve the accuracy of dependency identification, avoid missing important dependencies, and ensure the correctness of subsequent steps.

[0062] The static and dynamic analysis techniques are as follows: first, static code analysis tools are used to check unused dependency packages. These tools include SonarQube, Checkstyle, PMD, etc. They can help find out the dependency packages that are not used in the project, and can check whether the dependencies used in the project are compatible and whether there are version conflicts. At the same time, runtime monitoring tools such as Integrated Performance Monitoring are used to analyze the dependencies of the application and dynamically analyze the communication and performance of the MPI program. In this way, the usage of its dependency packages can be tracked while the application is running, so as to identify which dependencies are active and which are redundant.

[0063] S2. Installation file collection: Based on the analysis results of step S1, all third-party software and source code installation packages required for the target HPC application are automatically collected through an automated package management script. Unlike the prior art, which manually searches and downloads them one by one, the present invention uses an automated package management script to automatically download related dependent software packages from a specified package management library (such as apt, yum, or a custom source) and organize them into a folder structure suitable for image production. This technical means significantly reduces manual operations and ensures the efficiency and accuracy of the dependency collection process.

[0064] The automated package installation script is a script for automating the installation and configuration of software packages, using Shell script (Bash). This approach can simplify and standardize the software installation process, especially when the same software package needs to be repeatedly installed on multiple systems or environments. The script can be reused on different machines or environments to ensure the consistency of the installation process. It can identify and install the dependencies of the software package to ensure that all required components are installed correctly. This script contains error detection and handling mechanisms to provide feedback when problems occur during the installation process. The script contains configuration settings such as environment variables, service configuration, etc. to ensure that the software package can run as expected after installation. In addition, the script provides security checks for various software dependency packages, such as using secure download sources and verifying the integrity of downloaded software packages. Compared with existing technologies such as Dock and Singularity, only basic configuration and dependent installation are considered, without automated processing, security, version control, etc.

[0065] S3. Image preparation: In this step, according to the installation rules of the target application, the installation file script is automatically generated through the automated script generation technology. These scripts are based on the dependencies and installation order of the target application, and through the template generation system and dynamically adjusted configuration files, ensure that all dependencies can be installed in the correct order. The automated script generation technology of the present invention utilizes a graph algorithm based on the target application dependency relationship to automatically generate the installation order in an efficient manner and prevent installation conflicts, thereby simplifying the preparation work for image production.

[0066] The automated script generation technology utilizes a graph algorithm based on the target application dependencies, specifically: a directed graph containing all software packages and their dependencies, where nodes represent software packages and directed edges represent dependencies, and outputs an optimized software package loading order list. The graph is topologically sorted to determine a linear order so that each package is loaded before its dependencies, which is achieved by the Kahn algorithm or depth-first search (DFS). During the topological sorting process, if a circular dependency is detected (i.e., the sorting cannot be completed), an error is reported and the user is prompted to resolve these circular dependencies. For dependencies with the same package name but different versions, a version conflict resolution strategy is used, such as selecting the highest version or a user-specified version. Based on the results of the topological sorting and the results of the version conflict resolution, a final software package loading order list is generated. This method can ensure that no dependencies are missed and all dependencies are loaded in the correct order.

[0067] S4. Image creation system: In a Linux environment with root privileges, use automated scripting technology to create images. This step requires that all file permissions in the image are correctly configured. The present invention uses an automatic permission verification and configuration mechanism, which can automatically detect and repair permission problems during the image creation process, ensuring that files and applications in the target image can run correctly in different systems.

[0068] S5. Target application image structure: In this step, the present invention designs a unique directory structure, which adopts the application hierarchical directory structure technology (such as / rootdir / , / opt / repack, / run, etc.) compared with the existing image structure. The purpose of this design is to achieve independent encapsulation of application dependencies and reduce dependency conflicts between systems. The dependencies of each application are placed in a directory isolated from the operating system, avoiding version conflicts with other libraries of the system, thereby ensuring efficient resource management and operational stability. The traditional container image directory consists of multiple read-only layers, each of which is a modification or addition to the file system. These layers are stacked together to form a complete file system for use by the container. When the container needs to modify a file, Do will use the Copy-on-Write (CoW) mechanism to copy the file to the container layer for modification, while keeping the original image layer unchanged, ensuring the immutability of the image. However, this method will take up more disk space, especially when the image contains unnecessary components and dependencies, it may also increase potential attacks, and the image build time is long, especially when large applications need to be compiled. If you need to follow up on updates and bug fixes of dependencies in a timely manner, this may increase maintenance costs and may also cause operating system incompatibility issues. The layered design adopted by the present invention adds different levels of application directories, and the same layers can be shared in different images, reducing storage space. In addition, the layered structure allows the image to be reused by multiple different application images independently, reducing the time and resource consumption of repeated construction. Multiple read-only layers logically look like a whole, thereby achieving efficient storage and fast application startup.

[0069] S6. Create a basic image: Download and use a basic operating system image, such as Ubuntu, CentOS, etc., to ensure the consistency and stability of the system. The present invention uses a customized standard operating system image to ensure that the operating system environment used on all target platforms is consistent, thereby improving the portability and compatibility of the image.

[0070] S7. System directory binding: In order to compile and install software in the chroot environment, the current system's / dev and / proc file systems are mounted into the chroot environment by using an intelligent automatic mounting mechanism to ensure that the necessary device files and process information are in the chroot environment. The intelligent automatic mounting mechanism is used to automatically adjust the mounting content according to the needs of the target environment, avoiding the tedious operation of manual configuration, while improving the security and stability of the system.

[0071] The intelligent automatic mounting mechanism means that this method uses the system API (udev rules in Linux) or scripts to detect newly connected devices, and intelligently allocates mount points and mount options (noexec, nosuid, nodev) according to the characteristics of the storage device (such as capacity, type) and the current status of the system. At the same time, it can detect the file system used by the connected storage device, and can also try to automatically recover or remount when an abnormality is detected. Compared with traditional manual mounting, which requires users to have certain technical knowledge, the intelligent mechanism simplifies the operation process, lowers the usage threshold, and quickly responds to the connection of new devices. In addition, traditional mounting may require different mount scripts to be written for different devices and file systems, while the intelligent mechanism can adapt to a variety of file systems and devices.

[0072] S8. Create an environment variable file; this step generates an environment variable script repack.sh, which automatically loads the environment variables required by the application when the image is started, and makes necessary path adjustments. The environment variable configuration of the present invention is dynamically generated, and can automatically generate and adjust environment variables according to the needs of different applications, ensuring that the environment is consistent when the application runs on different nodes.

[0073] S9. Write application installation shell script: Write the installation process script of the target application. Use the automated script generation tool to generate the installation script and automatically detect the system environment to avoid installation errors caused by environmental differences. This automated installation process improves the efficiency and accuracy of the installation.

[0074] The automated script generation tool is an automated script generation tool built using AgileGen based on the aforementioned automated script technology.

[0075] S10. Image entry creation: Encapsulate the shell script of the installation process into repack.sh, and create an entry script run, which starts the application according to the parameters passed in. The automated script generation technology not only improves the efficiency of entry script generation, but also enhances the versatility of the script, supporting rapid switching of multiple operating environments and configurations.

[0076] S11. Squashfs file packaging: Use the mksquashfs tool to package the installed directory structure into a compressed Squashfs file system. Compared with traditional file system packaging methods, Squashfs has higher compression efficiency and smaller image size, which helps to reduce storage space and speed up data transmission, especially suitable for large-scale distribution and deployment.

[0077] S12. Image merging: The generated Squashfs file is merged with the header file using the versioning management mechanism and modularization mechanism to generate the final Repack image. This merging process ensures that all files and configurations of the image are correctly packaged to avoid file omissions or conflicts. The versioning management mechanism and modularization mechanism are adopted, and each image file is accompanied by complete version information for version control and update management; modularization makes it easier to modify the version and update the functions of each module.

[0078] The version management mechanism means that each image file is accompanied by complete version information to facilitate version control and update management;

[0079] The modular management means that the repack image includes a head header file module and a squashfs file system module, each module can be updated and maintained independently, the head header file module provides the container engine file required for the image runtime, and the squashfs file system module contains all the files and directories required for the system operation; the modular design enables the image to be flexibly adjusted according to user needs to adapt to different application scenarios and performance requirements;

[0080] S13. Image naming: The present invention identifies image files by formulating a unified naming rule (application name-version number-target architecture name-image description.sfs). This naming rule is different from the arbitrary naming method in the prior art, ensuring the traceability and version management of image files, making it easier for users to quickly identify and select the correct image version.

[0081] The present invention also supports singularity image import, and can directly convert singularity image files into repack images. The conversion steps of singularity files are as follows:

[0082] 1) Get the singularity image file rootfs file system;

[0083] 2) Unzip the rootfs in the sif file to the directory corresponding to each application;

[0084] 3) Delete the singularity rootfs file system environment variables;

[0085] 4) Create a repack image to automatically start the default application;

[0086] 5) Package the above rootfs directory into squashfs;

[0087] 6) Merge the squashfs and head files to generate the repack image file.

[0088] By packaging the application and all its runtime dependencies into an independent executable image file and publishing it as a statically linked executable file, the file comes with a container engine and can be run directly, solving the dependency problem encountered when deploying HPC software in different system environments. This method not only supports mainstream architectures such as X86 and ARM, but is also compatible with HPC computing environments such as GPU, Infiniband, and slurm. In addition, the Repack image file can be recompiled and optimized to meet the computing needs of different users.

[0089] The Repack image consists of two parts, the Repack Header and a Squashfs compressedrootfs. The Repack image architecture is as follows Figure 2 As shown, Repack Header provides files such as the container engine required for image runtime, and Squashfs compressed rootfs contains all files and directories required for system operation.

[0090] The key technical point of this implementation plan is not only to solve the dependency management problem of high-performance computing (HPC) applications, but also to provide a flexible, efficient and highly compatible solution. Through the following key technical points, this implementation plan realizes the seamless deployment and efficient operation of HPC applications in different environments:

[0091] 1. Comprehensive integration and management of dependencies

[0092] Multi-level dependency management: In an HPC environment, applications usually depend on multiple levels of libraries and tool chains, including the underlying glibc, MPI library, compiler, linker, etc. The present invention comprehensively integrates these dependencies through multi-level dependency analysis and encapsulates them in the Repack image to ensure that all necessary dependencies can be correctly loaded and used in different environments.

[0093] Integration of dynamic and static libraries: Many HPC applications rely on specific versions of dynamic link libraries and static link libraries. The present invention can automatically detect and collect these library files and integrate them into a unified image file. Through this integration, conflicts between library files of different versions are avoided, ensuring the stability of application operation.

[0094] Configuration files and environment variable management: For HPC applications, the correct configuration of configuration files and environment variables such as LD_LIBRARY_PATH, PATH, etc. is crucial. The present invention centrally manages these configuration files and environment variables in the Repack image to ensure that the application can correctly obtain its dependencies in different operating environments.

[0095] 2. High compatibility and scalability of Repack images

[0096] Cross-architecture support: The Repack image of the present invention not only supports the X86 architecture, but is also compatible with the ARM architecture and other specific HPC computing environments such as GPU and Infiniband. This cross-architecture support ensures that HPC applications can be seamlessly migrated and deployed on various hardware platforms, greatly improving the portability of applications.

[0097] Compatible with existing package management tools: Repack image design fully considers compatibility with existing mainstream tools, including Spack, Singularity, Conda, and Docker. Users can directly encapsulate applications managed by these tools into Repack images without complex conversion and adjustment, thus reducing the difficulty of deployment.

[0098] Modular design: Repack images adopt modular design. Each component in the image, such as Repack Header and Squashfs rootfs, can be updated and maintained independently. This design allows the image to be flexibly adjusted according to user needs to adapt to different application scenarios and performance requirements.

[0099] 3. Efficient image building process

[0100] Efficient file system compression and optimization: Use efficient file system tools such as Squashfs to compress dependent files and application files into a read-only file system, which not only saves storage space but also increases the speed of file reading. Through this compression, Repack images can be quickly deployed and run on large-scale supercomputing platforms.

[0101] Built-in container engine integration: Repack images have a built-in lightweight container engine, allowing users to run HPC applications without installing an additional container environment. This built-in engine design greatly simplifies the application deployment process and ensures the consistency of applications in different environments.

[0102] 4. Usability and User Experience

[0103] Out-of-the-box design: The Repack image of the present invention adopts an "out-of-the-box" design, which does not require users to perform complex configuration or compilation operations. The image file can be directly run on any supported HPC platform. Users only need simple command line operations to start and use the application, which greatly reduces the threshold for use.

[0104] Simple and intuitive operation interface: Provides a simple and intuitive command line interface, through which users can easily create, manage and deploy Repack images. Detailed help documents and sample scripts further reduce the user's learning curve, allowing even non-professional users to quickly get started.

[0105] High reliability and fault-tolerant design: Repack images are designed with high reliability and fault-tolerant capabilities in mind, ensuring that applications can still run normally in various extreme situations, such as hardware failures and network interruptions. Strict permission management and error handling mechanisms further improve the stability and security of images.

[0106] 6. Wide range of application scenarios

[0107] HPC large-scale computing scenarios: The Repack image of the present invention is particularly suitable for HPC large-scale computing scenarios, in which applications need to run efficiently on thousands of nodes. Repack images ensure rapid deployment and stable operation in such large-scale environments through unified dependency management and efficient file system compression.

[0108] Scientific computing and simulation: Repack images support various scientific computing applications including meteorological simulation, earthquake prediction, bioinformatics, astrophysics, etc. These applications usually rely on complex algorithms and multi-version computing libraries. Repack images ensure the repeatability of calculations and the accuracy of results through unified packaging and version management.

[0109] Data analysis and artificial intelligence: Repack images can also be applied to data analysis and artificial intelligence, such as deep learning model training, big data processing, image processing, etc. These applications usually require efficient parallel computing and full utilization of computing resources. Repack images meet the performance requirements of these applications through efficient resource management and flexible secondary compilation support.

[0110] Next, we test this method. First, we configure the operating environment: deploy repack to the production environment, configure an operating environment that matches the package file on the target system, and ensure that the application can run seamlessly; then we test the image: we test the generated Repack image on a variety of HPC platforms to ensure that its compatibility and performance meet expectations. The specific application examples are as follows:

[0111] Specific example application 1:

[0112] CP2K is an open source first-principles material calculation and simulation software that can study large systems of thousands of atoms and is widely used in the simulation of solids, liquids, molecules, periods, materials, crystals, and biological systems. To make a CP2K software repack image, you need a basic operating system, compiler, and rely on BLAS and LAPAC math libraries. These software installation packages are generated according to the repack production process to generate the cp2k-2023.2-x86_64-oneapi.rpk repack image. Users can use the CP2K Repack image directly in the job scheduling system for calculations. The specific script is as follows:

[0113] #! / bin / bash

[0114] #BATCH --nodes=1

[0115] #SBATCH --ntasks=1

[0116] #SBATCH --cpus-per-task=40

[0117] #SBATCH --partition=high

[0118] . / cp2k-2023.2-x86_64-oneapi.rpk bash -c 'source / etc / profile.d / cp2k.sh; mpirun -np 40 cp2k.psmp -o D1.outD1.inp'

[0119] After the above script is executed, a cp2k application parallel computing task can be implemented on 40 cores, generating the output result D1.out.

[0120] Specific example application 2:

[0121] To complete a soil microbial metagenomic analysis, six application software are needed: sortmerna (rRNA removal tool based on alignment), megahit (metagenomic sequence assembly tool), bowtie2 (sequence alignment tool), samtools (tool for operating sam and bam files), seqkit (tool for processing fsata / q sequence files), and diamond (tool for protein (blastp, blastx) sequence alignment). These six software are generated according to the repack image production process: sortmerna-4.3.7-aarch64-conda.rpk, megahit-1.2.9-aarch64-kunpeng.rpk, and

[0122] diamond-0.9.14.rpk three repack images, users directly use the three Repack images in the job scheduling system to perform metagenome analysis and calculation. The specific scripts are as follows:

[0123] #! / bin / bash

[0124] #SBATCH --nodes=1

[0125] #SBATCH --ntasks=64

[0126] export REPACK_EXTRA_OPTS='--bind / etc / hosts / etc / hosts '

[0127] / opt / app / repack / sortmerna-4.3.7-aarch64-conda.rpk – sortmerna

[0128] / opt / app / repack / megahit-1.2.9-aarch64-kunpeng.rpk – megahit

[0129] / opt / app / repack / sortmerna-4.3.7-aarch64-conda.rpk -- bowtie2

[0130] / opt / app / repack / sortmerna-4.3.7-aarch64-conda.rpk -- bowtie2-build-s

[0131] / opt / app / repack / sortmerna-4.3.7-aarch64-conda.rpk –- samtools view

[0132] / opt / app / repack / sortmerna-4.3.7-aarch64-conda.rpk –- samtools sort

[0133] / opt / app / repack / sortmerna-4.3.7-aarch64-conda.rpk –- samtools index

[0134] / opt / app / repack / sortmerna-4.3.7-aarch64-conda.rpk –- samtoolsidxstats

[0135] / opt / app / repack / sortmerna-4.3.7-aarch64-conda.rpk –- seqkit

[0136] / opt / app / repack / diamond-0.9.14.rpk -- diamond blastx

[0137] After the script is executed, metagenomic analysis and calculation can be performed on a 96-core server and the final output data can be generated.

[0138] Specific example application three:

[0139] NAMD is a parallel molecular dynamics application software used to quickly simulate macromolecular systems on large-scale parallel computers. NAMD software supports GPU acceleration. NVIDIA's official website provides NAMD's singularity image. A script is used to convert the singularity image into a repack image namd3.0-singlenode.rpk. Users can run this repack image on three different types of GPU servers by following the following commands:

[0140] . / namd3.0-singlenode.rpk -- namd3+p1 +devices 0 +setcpuaffinityapoa1 / apoa1_nve_cuda_soa.namd

[0141] The results of running the three servers are as follows Figure 3As shown, we can see that the NAMD repack image runs normally on three different models of GPU servers and obtains calculation results.

[0142] The above three application examples show that the repack image allows users to run directly on different systems, achieving out-of-the-box and cross-platform operation. The reapck image in Example 1 can run directly on the X86 system, the repack image in Example 2 can run directly on the ARM architecture system, and the repack in Example 3 can run directly on systems with different GPU models.

[0143] The above embodiments of the present invention are merely examples for clearly illustrating the technical solutions of the present invention, and are not intended to limit the specific implementation methods of the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the claims of the present invention shall be included in the protection scope of the claims of the present invention.

Claims

1. A Repack image file packaging method for solving the running dependency of HPC applications, characterized in that: The method comprises: S1. Software analysis: Perform dependency analysis on the target HPC application software using static and dynamic analysis techniques, and identify all direct and indirect dependent libraries, configuration files, compilers, tool chains, and runtime environments through automated scripts; S2. Collection of installation files: Based on the analysis results of step S1, an automated package management script is used to automatically collect all third-party software and source code installation packages that the target HPC application depends on; S3. Image preparation: Generate an installation file script using automated script generation technology based on the target application's installation rules. The generated installation file script is based on the target application's dependencies and installation order, and uses a template generation system and dynamically adjusted configuration files to ensure that all dependencies can be installed in the correct order. The automated script generation technique utilizes a graph algorithm based on target application dependencies: First, create a directed graph containing all software packages and their dependencies, where nodes represent software packages and directed edges represent dependencies, and output an optimized list of software package loading order; The graph is then topologically sorted to determine a linear order so that each package is loaded before its dependencies, which is achieved through the Kahn algorithm or depth-first search DFS. During the topological sorting process, if a circular dependency is detected, that is, the sorting cannot be completed, an error is reported and the user is prompted to resolve these circular dependencies. For dependencies with the same package name but different versions, a version conflict resolution strategy is used. Finally, based on the results of topological sorting and version conflict resolution, the final software package loading order list is generated; S4. Image creation system preparation: In a Linux environment with root privileges, use automated scripting technology to create images. At the same time, use automatic permission verification and configuration mechanisms to automatically detect and repair permission issues during the image creation process. S5. Target HPC application image structure: Using hierarchical directory structure technology, create a hierarchical directory structure of the target HPC application under the image production system, and place the dependencies of each application in a directory isolated from the operating system; S6. Create a basic image: Download the basic operating system image and generate a head file in the process. The head file implements the mounting of the fuse file system and the startup of the container engine. S7. System directory binding: Use intelligent automatic mounting mechanism to mount the current system's / dev / proc file system to the directory rootdir; S8. Create an environment variable file: Generate an environment variable script repack.sh, which automatically loads the environment variables required by the application when the image is started, and makes necessary path adjustments; The configuration of the environment variables is dynamically generated, and the environment variables can be automatically generated and adjusted according to the needs of different applications; S9. Write application installation shell script: Use the automated script generation tool to generate the installation process script of the target HPC application and automatically monitor the system environment; S10. Image entry creation: Encapsulate the application installation shell script into repack.sh, and create an entry script run through automated script generation technology. The script run loads environment variables and determines how to run the application based on the parameters passed in by the specific HPC application. S11. squashfs file encapsulation: Use the mksquashfs tool to package the rootdir directory into a compressed squashfs file system, that is, use the squashfs efficient file system tool to compress the dependent files and application files into a read-only file system; S12. Image merging: Use the versioning management mechanism and modularization mechanism to merge the generated squashfs file system and the head header file generated when creating the base image to generate the final Repack image; The version management mechanism means that each image file is accompanied by complete version information to facilitate version control and update management; The modular management means that the Repack image includes a head header file module and a squashfs file system module, each module can be updated and maintained independently, the head header file module provides the container engine file required for the image runtime, and the squashfs file system module contains all the files and directories required for the system operation; S13. Image naming: Name the image to distinguish the target HPC application version number.

2. A Repack image file encapsulation method for solving the running dependency of HPC applications according to claim 1, characterized in that: Supports importing singularity images, directly converting singularity images into Repack images. The conversion steps of singularity images are as follows: 1) Get the rootfs file system of the singularity image; 2) Unzip the rootfs file system to the directory corresponding to each application; 3) Delete the environment variables in the rootfs file system; 4) Create a Repack image to automatically start the default application; 5) Package the above rootfs file system directory into squashfs; 6) Merge the squashfs and head files to generate a Repack image.

3. A Repack image file encapsulation method for solving the running dependency of HPC applications according to claim 1, characterized in that: The Repack image supports the X86 architecture and is also compatible with the ARM architecture and other specific HPC computing environments.

4. A Repack image file encapsulation method for solving the running dependency of HPC applications according to claim 1, characterized in that: The static and dynamic analysis techniques refer to: first, using static code analysis tools to check unused dependency packages, check whether the dependencies used in the project are compatible, and whether there are version conflicts; at the same time, using runtime monitoring tools to analyze the application's dependencies and dynamically analyze the communication and performance of the MPI program, and track the usage of its dependency packages while the application is running, so as to identify which dependencies are active and which are redundant.

5. A Repack image file encapsulation method for solving the running dependency of HPC applications according to claim 1, characterized in that: The automated package installation script includes error detection and handling mechanisms, configuration settings, and various software dependency package security detection mechanisms, and is used for automated software package installation and configuration.

6. A Repack image file encapsulation method for solving the running dependency of HPC applications according to claim 1, characterized in that: The intelligent automatic mounting mechanism refers to: using system API or script to detect newly connected devices, intelligently allocating mount points and mount options according to the characteristics of the storage device and the current status of the system, and detecting the file system used by the connected storage device. When an abnormality is detected, it attempts to automatically recover or remount.

7. A Repack image file encapsulation method for solving the running dependency of HPC applications according to claim 1, characterized in that: The Repack image has a built-in lightweight container engine.

8. A Repack image file encapsulation method for solving the running dependency of HPC applications according to claim 1, characterized in that: The Repack image runs directly on any supported HPC platform, and users use command line operations to start and use applications.

9. A Repack image file encapsulation method for solving the running dependency of HPC applications according to claim 1, characterized in that: The Repack image file is released as a statically linked binary executable file.

Images