A method for permanent storage of dependencies required during image building

Abstract

The present invention belongs to the technical field of high-performance computing, and more specifically, it relates to a method for permanent storage of dependencies required in the image building process. The method establishes a cross-domain peer-to-peer transmission and storage system based on encrypted coding conversion and distributed wide area network, and divides the original files, construction files and the final image files required in the image building process into content, scrambles them, and finally transmits them to the peer-to-peer network for permanent storage. In the image production process, the dependent files required for application installation are referenced and obtained using the unique identifier generated after transcoding. The application image constructed by this method will never fail to build due to invalid references at any time, thereby achieving the repeatability of container image construction and effectively solving the problem that installation files or third-party files they depend on cannot be permanently referenced during image construction.

Description

Technical Field

[0001] The present invention belongs to the technical field of high performance computing, and more specifically, relates to a method for storing persistent dependencies required in a mirror image building process. Background Art

[0002] In the field of high-performance computing, application software installation is a very complex process. In addition to using the software management and installation tools that come with the operating system, such as rpm, yum, and apt, many software are also installed by compiling the source code. These methods are highly dependent on the operating system, especially when deploying multiple software or multiple versions of a software on a platform. This is a challenging task. The diversity of software dependencies also makes software management extremely difficult.

[0003] To solve these problems, the container method came into being. It has the function of isolating applications and allows a large number of software with different dependencies to be deployed on a single platform. Containers use virtualization technology to allow developers to package applications and their dependencies together, thereby achieving consistent operation in different environments and isolating different processes, so that the entire container can share the kernel at the bottom layer and the application at the upper layer.

[0004] Since Docker containers require root permissions and do not support job scheduling, MPI, GPU, and Infiniband required by high-performance computing scenarios, singularity has become the most commonly used container in high-performance computing scenarios. Containers are usually provided to users in the form of images, and each image of HPC application software needs to be made separately. Singularity defines the image production format, and users can encapsulate the software installation process in a custom def file. This def file contains the operating system, various software, and dependent source code installation packages, patches, etc. required for software installation. Finally, this def file is packaged into a singularity container image with the suffix sif. This file is read-only and can run in any system environment based on the same kernel, thereby resolving the dependency between the software and a certain system.

[0005] However, the HPC application software installation package and its dependent software and base image ISO required by the singularity container image in the construction process are all obtained through network download. If the network address of these installation packages is invalid or the download address is updated, the software package cannot be obtained from the original address. For example, in the singularity def file, the software of the application is often obtained during the construction process by defining a network link. If the download address access is invalid, the software installation package cannot be obtained. Then the image will generate an error during the automatic construction process, and ultimately the image cannot be successfully produced. This defect also makes it impossible to achieve repeatability in image construction, and the repeatability of calculation results is crucial for high-performance computing application research. If these installation and dependent source packages are permanently saved during the image construction process, the above problem can be solved.

[0006] For example, patent document US106487850A mentions a method and device for obtaining images in a cloud environment, but this method also faces the problem of network resource failure; patent document CN107301022A discusses a container-based cloud storage solution, but in actual applications, cloud resources may also be inaccessible due to link failure, resulting in image building failure. The high-performance computing environment is particularly dependent on images because the repeatability of computing results is crucial. All high-performance computing methods involving containers lack the processing of persistent storage. Summary of the invention

[0007] The present invention aims to overcome at least one defect of the above-mentioned prior art, and provides a method for permanent storage of dependencies required in the image building process. The method establishes a cross-domain peer-to-peer transmission and storage system based on encryption code conversion and distributed wide area network, and divides the original files, construction files and the final image files required in the image building process into content, scrambles them, and finally transmits them to the peer-to-peer network for permanent storage. In the image making process, the dependent files required for application installation are referenced and obtained using the unique identifier generated after transcoding. The application image constructed by this method will never fail to build due to invalid references at any time, thereby achieving the repeatability of container image construction and effectively solving the problem that the installation files or the third-party files they depend on cannot be permanently referenced during the image building process. The present invention can be applied to the construction of container files for high-performance computing applications.

[0008] The detailed technical scheme of the present invention is as follows:

[0009] A method for permanently storing dependencies required in an image building process, the method comprising:

[0010] S1. Segment-encode the target application, other third-party files that the target application depends on, and the operating system environment files required to build the target file using an encryption algorithm based on the quadratic remainder problem;

[0011] S2, configuring the distributed wide area network cross-domain peer-to-peer transmission and storage system, and then sending the segmented encoded data to the configured distributed wide area network cross-domain peer-to-peer transmission and storage system;

[0012] S3. When the segmented encoded data is sent to the distributed wide area network cross-domain peer transmission and storage system, the differently encoded data is sent to different storage nodes for storage, and the same encoded data is sent to multiple data storage nodes for storage; the data is sent in a concurrent manner;

[0013] S4. When the segmented encoded data is sent to the distributed wide area network cross-domain peer-to-peer transmission and storage system, the overall check code of the file data is calculated from the different encoded data segments, and the check code is encoded and sent to the distributed wide area network cross-domain peer-to-peer transmission and storage system, thereby ensuring the reproducibility and disaster recoverability of the data;

[0014] S5. The overall checksum of all segmented coded data is the unique identifier of the file. Any data change at any position of the original data will result in a change in the final identifier, which determines the integrity and uniqueness of the data.

[0015] S6. Modify the image definition file, and change the installation package and dependencies originally obtained through the network link to the acquisition of all dependent files involved in the image building process by using the overall verification code of the file data as a unique identifier;

[0016] S7. After completing the modification of the image definition file, start building the image. Obtain all dependent files involved in the image building process from the distributed wide area network cross-domain peer transmission and storage system to perform the image building operation. After the building is completed, test the generated image file.

[0017] Furthermore, the encryption algorithm based on the quadratic remainder problem refers to, given an integer p and an odd prime number a, determining whether there is an integer x such that when the square of x is calculated, the remainder when it is divided by p is exactly a.

[0018] Furthermore, the data in the distributed wide area network cross-domain peer-to-peer transmission and storage system is segmented and distributed and stored on different nodes according to a certain strategy, which can disperse the pressure of data storage. Even if some nodes fail, data can still be restored through other nodes to ensure data security and availability, and fully utilize the upload bandwidth of each node; data transmission and storage in the system are not limited to a single local area network, but span multiple geographical areas or network domains, solving the problem of low transmission efficiency of traditional storage systems in a wide area network environment; data stored in the system uses the encoding of each segment of data after segmentation as a key value to find the storage node corresponding to the data and realize the access of the corresponding segment data; and each storage node in the system has a unique ID, and supports dynamic joining and leaving of nodes through a hash ring algorithm;

[0019] The hash ring algorithm solves the problem of data load balancing among dynamically distributed nodes by mapping data and nodes to a logical ring, and minimizes data migration when adding or removing nodes, thereby improving the scalability and stability of the system.

[0020] Furthermore, in the mirror building process described in S7, the required content stored in the distributed wide area network cross-domain peer transmission and storage system is concurrently transmitted from the storage nodes corresponding to each segment using a unique identifier, thereby speeding up the transmission speed.

[0021] Furthermore, if the storage node is behind a firewall, a pipeline for direct data transmission is established preferentially through UDP hole punching technology. If the pipeline establishment fails, data routing and forwarding are performed through a server with a public network address to obtain data.

[0022] Furthermore, the configuration of the distributed wide area network cross-domain peer-to-peer transmission and storage system specifically refers to:

[0023] Initialize the node: Initialize the distributed wide area network cross-domain peer-to-peer transmission and storage system node, which will create a local node for interaction with the distributed wide area network cross-domain peer-to-peer transmission and storage system network; at the same time, set up a health check mechanism during the node initialization stage to detect hardware or configuration anomalies that the node may encounter during the daemon startup stage;

[0024] Start the daemon process: Start the distributed WAN cross-domain peer-to-peer transmission and storage system daemon process to ensure that the node is always online and can exchange data with other nodes, and ensure the distributed WAN cross-domain peer-to-peer transmission and storage system nodes and the distributed WAN cross-domain peer-to-peer transmission and storage system network connection; at the same time, the daemon process has an abnormal node isolation mechanism. When a node has abnormal data transmission or is offline for a long time, the system can automatically isolate it and notify the administrator to check;

[0025] Network connection test: Test the connectivity between the distributed wide area network cross-domain peer-to-peer transmission and storage system nodes and other nodes in the network, especially the seed server, to ensure the stability of data transmission; a multi-path transmission strategy is also adopted in the network connection test phase, and the distributed wide area network cross-domain peer-to-peer transmission and storage system nodes and other nodes in the network select the best path to transmit data, while supporting multi-path transmission and dynamic path switching.

[0026] Furthermore, each time any modification is made to all dependent files involved in the image building process and the image file finally generated, a new unique identifier is generated and different versions of the file are managed by recording these unique identifiers.

[0027] Furthermore, when the image needs to be updated, the new unique identifier is used to obtain the installation package and the dependent files to build the image and store the old version of the image and the dependent files.

[0028] Compared with the prior art, the present invention has the following beneficial effects:

[0029] (1) The present invention collects the software installation packages and their dependent files required in the image building process, and adds the collected software installation packages and their dependent files to the distributed wide area network cross-domain peer-to-peer transmission and storage system, thereby realizing the permanent preservation of these files, thereby solving the problem of invalid references. By referencing and obtaining dependent files through unique identifiers, it is ensured that the building process will not fail due to invalid references of dependent files at any time and environment, thereby achieving the repeatability of container image building.

[0030] (2) The present invention uses an encryption algorithm based on the quadratic remainder problem for segmented encoding, combined with data check codes and a multi-node storage mechanism, to ensure the security and integrity of data during transmission and storage. At the same time, redundant storage is performed on multiple storage nodes, and data check codes are transmitted simultaneously, ensuring that the system has reproducibility and disaster recoverability in the face of single-point failures or disasters, thus solving the security and integrity of data.

[0031] (3) The distributed wide area network cross-domain peer-to-peer transmission and storage system introduced by the present invention generates and records a new unique identifier each time a file is modified. By recording these unique identifiers, different versions of the file can be managed to ensure that the correct version can be selected when the image is updated. When the container image needs to be updated, it is built using the unique identifier of the new version file to ensure that the image always uses the latest dependent files. For historical versions of images and dependent files, long-term storage and anytime call-up of versions can be achieved through unique identifier management, thereby ensuring the system's backtracking capability.

[0032] (4) By using a distributed wide area network cross-domain peer-to-peer transmission and storage system, the present invention allows users to quickly obtain the required files through a distributed network, no longer relying on the transmission speed and network bandwidth of a single server. This point-to-point transmission mechanism can significantly reduce file transmission delays, especially when processing large-scale, high-bandwidth data files, greatly improving the efficiency of image building. Moreover, in the distributed wide area network cross-domain peer-to-peer transmission and storage system, the application of data segmentation and concurrent transmission and UDP hole punching technology greatly improves data transmission efficiency, solves the problem that the image building process is no longer limited by fluctuations in network conditions, and shortens the overall time of image production.

[0033] (5) The present invention generates and records a new unique identifier each time a file is modified, making file version management clearer and tracing back historical versions. Version control of files is achieved through a distributed wide area network cross-domain peer-to-peer transmission and storage system. Users can track different versions of the same software package or dependent package and easily manage and switch versions. This solves the problem of ensuring that developers and researchers in high-performance computing application environments can maintain system consistency and stability when faced with software updates or upgrades, and avoid operational errors or result deviations caused by version incompatibility.

[0034] (6) The present invention is not only applicable to container image construction in high-performance computing environments, but can also be extended to other computing scenarios that require high reliability, high security, and high repeatability. For example, data processing systems for scientific experiments, big data analysis platforms, virtual machine or container management in cloud computing environments, etc. can all achieve persistent storage of dependent files and efficient data access by introducing a distributed wide area network cross-domain peer-to-peer transmission and storage system, thereby greatly improving the reliability and operating efficiency of these systems. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] Figure 1 It is a flow chart of a method for storing dependencies permanently in a mirror image building process described in the present invention.

[0036] Figure 2 It is a configuration flow chart of the distributed wide area network cross-domain peer-to-peer transmission and storage system described in the present invention. DETAILED DESCRIPTION

[0037] The present disclosure is further described below in conjunction with the accompanying drawings and embodiments.

[0038] Embodiment 1,

[0039] like Figure 1 As shown, this embodiment provides a method for storing dependencies permanently in a mirror image building process, the method comprising:

[0040] S1. Segment-encode the target application, other third-party files that the target application depends on, and the operating system environment files required to build the target file using an encryption algorithm based on the quadratic remainder problem;

[0041] S2, configuring the distributed wide area network cross-domain peer-to-peer transmission and storage system, and then sending the segmented encoded data to the configured distributed wide area network cross-domain peer-to-peer transmission and storage system;

[0042] The distributed wide area network cross-domain peer-to-peer transmission and storage system means that data can be stored in multiple nodes instead of a single server, which can disperse the pressure of data storage; secondly, data transmission and storage are not limited to a single local area network, but can span multiple geographical areas or network domains, solving the problem of low transmission efficiency of traditional storage systems in wide area network environments. It fragments the data according to a certain strategy and distributes it on different nodes. Even if some nodes fail, the data can still be restored through other nodes to ensure data security and availability, and make full use of the upload bandwidth of each node. Moreover, each storage node has a unique ID, and supports dynamic joining and leaving of nodes through a hash ring algorithm. The data stored in the distributed wide area network cross-domain peer-to-peer transmission and storage system uses the encoding of each segment of data after segmentation as the key value to find the storage node corresponding to the data and access the corresponding segment data;

[0043] The hash ring algorithm solves the problem of data load balancing among dynamically distributed nodes (such as servers, storage nodes, etc.) by mapping data and nodes to a logical ring. It can minimize data migration when adding or removing nodes, thereby improving the scalability and stability of the system.

[0044] S3. When the segmented encoded data is sent to the distributed wide area network cross-domain peer transmission and storage system, the differently encoded data is sent to different storage nodes for storage, and the same encoded data is sent to multiple data storage nodes for storage, thereby ensuring data redundancy; the data is sent in a concurrent manner, thereby improving transmission efficiency;

[0045] S4. When the segmented encoded data is sent to the distributed wide area network cross-domain peer transmission and storage system, the overall check code of the file data is calculated from different encoded data segments, and the check code is also encoded and sent to the distributed wide area network cross-domain peer transmission and storage system, so as to ensure the reproducibility and disaster recoverability of the data; the specific encoding algorithm can be selected and switched according to the needs during encoding, and common algorithms include SHA256 or BLAKE2, etc.;

[0046] S5. The overall checksum of all segmented coded data is the unique identifier of the file. Any data change at any position of the original data will result in a change in the final identifier, which determines the integrity and uniqueness of the data.

[0047] S6. Modify the image definition file. Instead of using the traditional file URL address (such as http, ftp and other network addresses) as an identifier to obtain files, use the overall checksum of the file data described in S4 above as a unique identifier to obtain all files involved in the image building process, including but not limited to source code files, installation files, configuration files, data files, etc.;

[0048] S7. After completing the image definition file setting, start building the image. Obtain the source code files, installation files, configuration files, data files and other dependent files required for the image building process described above from the distributed wide area network cross-domain peer transmission and storage system to perform the image building operation. After the building is completed, test the generated image file.

[0049] In the construction process described in S7, the required content stored in the distributed wide area network cross-domain peer transmission and storage system is concurrently transmitted from the storage nodes corresponding to each segment using a unique identifier, thereby speeding up the transmission speed.

[0050] If the storage node is behind a firewall, it will first establish a pipeline for direct data transmission through UDP hole punching technology. If the pipeline establishment fails, the data is routed and forwarded through a server with a public network address to obtain data. UDP hole punching technology is a network penetration technology used to solve the problem of direct communication between two devices behind different NATs or firewalls. It uses the NAT session tracking mechanism to exchange the public network IP and port information of the device through a relay server, allowing both parties to send UDP data packets to each other, thereby establishing a direct connection channel. This technology can bypass NAT restrictions and achieve low-latency point-to-point communication.

[0051] like Figure 2 As shown, the configuration of the distributed wide area network cross-domain peer-to-peer transmission and storage system specifically refers to:

[0052] Initialize the node: Initialize the distributed wide area network cross-domain peer-to-peer transmission and storage system node, which will create a local node for interaction with the distributed wide area network cross-domain peer-to-peer transmission and storage system network; at the same time, set up a health check mechanism during the node initialization stage to detect hardware or configuration anomalies that the node may encounter during the daemon startup stage;

[0053] Start the daemon process: Start the distributed WAN cross-domain peer-to-peer transmission and storage system daemon process to ensure that the node is always online and can exchange data with other nodes, and ensure the distributed WAN cross-domain peer-to-peer transmission and storage system nodes and the distributed WAN cross-domain peer-to-peer transmission and storage system network connection; at the same time, the daemon process has an abnormal node isolation mechanism function. When a node has abnormal data transmission or is offline for a long time, the system can automatically isolate it and notify the administrator to check;

[0054] Network connection test: Test the connectivity between the distributed wide area network cross-domain peer-to-peer transmission and storage system nodes and other nodes in the network, especially the seed server, to ensure the stability of data transmission; a multi-path transmission strategy is also adopted in the network connection test phase, and the distributed wide area network cross-domain peer-to-peer transmission and storage system nodes and other nodes in the network can choose the best path to transmit data, while supporting multi-path transmission and dynamic path switching.

[0055] Each time any modification is made to the source code files, installation files, configuration files, data files, and the resulting image files involved in the image building process, a new unique identifier is generated and different versions of the files are managed by recording these unique identifiers. When the image needs to be updated, the new unique identifier is used to obtain the installation package and dependent files for image building and the old version of the image and dependent files are stored.

[0056] Embodiment 2,

[0057] This embodiment provides a specific implementation process of a method for persistent storage of dependencies required in an image building process, so as to improve the repeatability and reliability of container image building in a high-performance computing (HPC) application environment. The method is divided into a preparation stage, implementation steps, and subsequent management. The method includes:

[0058] 1. Preparation stage:

[0059] 1) Distributed WAN cross-domain peer-to-peer transmission and storage system installation:

[0060] First, a distributed WAN cross-domain peer-to-peer transmission and storage system file system is deployed in the target environment.

[0061] 2) If Figure 2 As shown, the distributed wide area network cross-domain peer-to-peer transmission and storage system configuration includes:

[0062] Initialize the node: Initialize the distributed WAN cross-domain peer-to-peer transmission and storage system node, which will create a local node for interaction with the distributed WAN cross-domain peer-to-peer transmission and storage system network; at the same time, set up a health check mechanism in the initialization phase to detect hardware or configuration anomalies that the node may encounter during the startup phase;

[0063] Start the daemon process: Start the distributed WAN cross-domain peer-to-peer transmission and storage system daemon process to ensure that the node is always online and can exchange data with other nodes, and ensure the distributed WAN cross-domain peer-to-peer transmission and storage system nodes and the distributed WAN cross-domain peer-to-peer transmission and storage system network connection; at the same time, the daemon process has an abnormal node isolation mechanism function. When a node has abnormal data transmission or is offline for a long time, the system can automatically isolate it and notify the administrator to check;

[0064] Network connection test: Test the connectivity between distributed wide area network cross-domain peer transmission and storage system nodes and other nodes in the network, especially the seed server, to ensure the stability of data transmission; a multi-path transmission strategy is also adopted in the network connection test phase, and nodes can choose the best path to transmit data, while supporting multi-path transmission and dynamic path switching.

[0065] 3) Data preparation:

[0066] Data preparation includes the collection of software installation packages and dependent files and the classification and organization of data;

[0067] The collection of software installation packages and dependent files includes collecting all software installation packages, dependent libraries, operating system images, and patch files required in the image building process. These files will be used as objects that need to be persisted in subsequent steps.

[0068] The collected files are classified and organized according to the type and purpose of the files so that they can be efficiently stored and managed in the distributed wide area network cross-domain peer-to-peer transmission and storage system file system.

[0069] (II) Implementation steps:

[0070] 1) Persistent storage of files:

[0071] Upload files to the distributed WAN cross-domain peer-to-peer transmission and storage system: The software installation package and its dependent files are segmented by size and added to the distributed WAN cross-domain peer-to-peer transmission and storage system file system. After each segment file is uploaded, the distributed WAN cross-domain peer-to-peer transmission and storage system will generate a segment code based on the encryption algorithm of the quadratic remainder problem, and this identifier will be used as the unique identifier of the segment file.

[0072] The encryption algorithm based on the quadratic remainder problem can be described as given an integer p and an odd prime number a, determining whether there is an integer x such that when x is squared, the remainder divided by p is exactly a. In the field of cryptography, an encryption algorithm can be constructed based on the difficulty assumption of quadratic remainders, and its pseudo code is:

[0073] / / The function is used to calculate fast power for subsequent calculations. Here a simple fast power algorithm is used

[0074] fn power_mod(base: u64, exponent: u64, modulus: u64) ->u64 {

[0075] let mut result = 1;

[0076] let mut base = base % modulus;

[0077] let mut exponent = exponent;

[0078] while exponent>0 {

[0079] if exponent % 2 == 1 {

[0080] result = (result * base) % modulus;

[0081] }

[0082] exponent / = 2;

[0083] base = (base * base) % modulus;

[0084] }

[0085] Result

[0086] }

[0087] / / Determine whether the given number a is a quadratic residue modulo a prime number p

[0088] fn is_quadratic_residue(a: u64, p: u64) ->bool {

[0089] if p % 2 == 0 {

[0090] / / If p is not an odd prime number, simply return an error here. In actual applications, it can be handled more rigorously.

[0091] return false;

[0092] }

[0093] let result = power_mod(a, (p - 1) / 2, p);

[0094] result == 1

[0095] }

[0096] 2) Calculate the overall checksum of the file data based on the different coded data segments as the unique access identifier for cross-domain peer-to-peer transmission and storage systems in the distributed wide area network. Record the generated unique identifier together with the corresponding file information to ensure that these files can be accurately referenced during the image creation process.

[0097] 3) Download the file:

[0098] The software installation package stored in the distributed wide area network cross-domain peer-to-peer transmission and storage system is downloaded locally using the unique identifier.

[0099] For example, use the following command:

[0100] hget QmT9x8UTBVgDzpyXcwyLFeJ2y9s5K9jwzKMZPS37ymxr25 -o diamond-0.0.14.tar.gz, you can download the file corresponding to the unique identifier from the distributed wide area network cross-domain peer transmission and storage system file system.

[0101] 4) Building container images:

[0102] Modify the image definition file: When building a container image, such as Singularity or Docker, modify the image definition file, such as Singularity's .def file, to change the installation package and dependencies that were originally obtained through a network link to access through a unique identifier. For example, change `wget <url>` is replaced with `distributed WAN cross-domain peer transmission and storage system cat <hash value>` to ensure that the required files are obtained directly from the distributed WAN cross-domain peer transmission and storage system file system during the build process.

[0103] Image building and verification: Perform image building operations to ensure that all files can be successfully obtained from the distributed wide area network cross-domain peer transmission and storage system, and successfully complete the image production. After the build is completed, the generated image file is tested to verify its availability and stability.

[0104] (III) Subsequent management, namely, updating and management of images and data:

[0105] 1) File version control: In the distributed WAN cross-domain peer-to-peer transmission and storage system file system, a new unique identifier is generated every time a file is modified. By recording these unique identifiers, different versions of the file can be managed to ensure that the correct version can be selected when the image is updated.

[0106] 2) Image update: When the container image needs to be updated, it is built using the unique identifier of the new version file to ensure that the image always uses the latest dependency files.

[0107] 3) Storage of old versions of images and data: For historical versions of images and dependent files, unique identification management can achieve long-term storage and call-up at any time, ensuring the system's backtracking capability.

[0108] Specific Example 1

[0109] CP2K is an open source quantum chemistry and solid-state physics software package, widely used to simulate the electronic structure and energy of molecules, liquids, materials and biological systems. The image production of cp2k requires three installation packages. Each installation package first generates a unique hash value through the distributed wide area network cross-domain peer-to-peer transmission and storage system add. The hash value is used to access the file, as follows:

[0110] hcat QmS4JAN8kceQRYWfSJ94gAu8sFVdEb8dPbtAEc7HCKvn8x | tar Jxf - -Crootdir

[0111] # cp2k-2023.2.tar.bz2

[0112] hcat QmYXRdG6AvmBga5vY1YXtre2pGEMq1zZU6MnWhQCmAuFH2 | tar jxf - -Crootdir / opt / repack /

[0113] The command unpacks the cp2k-2023.2.tar.bz2 file to / rootdir / opt / repack / , and QmS4JAN8kceQRYWfSJ94gAu8sFVdEb8dPbtAEc7HCKvn8x and QmYXRdG6AvmBga5vY1YXtre2pGEMq1zZU6MnWhQCmAuFH2 are the hash values ​​generated for the file.

[0114] # build.tar.xz

[0115] hcat QmR1vYhWMtDuu6A1TCJWRMs44gcJ11q4c3nnd72P8SNXgn | tar Jxf - -Crootdir / opt / repack / cp2k-2023.2 / tools / toolchain

[0116] This command decompresses the build.tar.xz file to / rootdir / opt / repack / , and QmR1vYhWMtDuu6A1TCJWRMs44gcJ11q4c3nnd72P8SNXgn is the hash value generated for the file.

[0117] # intel one api

[0118] hcat QmNso6a6j124sn3jwKrpXpdFQKmcDNcm95n8n7xk9akAfj | tar Jxf - -Crootdir / opt /

[0119] This command decompresses the intel oneapi file to / rootdir / opt / repack / . QmNso6a6j124sn3jwKrpXpdFQKmcDNcm95n8n7xk9akAfj is the hash value generated for the file.

[0120] Specific Example 2

[0121] MAPS is an earth system model developed by the National Center for Atmospheric Research (NCAR) in the United States. It supports a variety of grid types and physical processes and is suitable for simulations from global to regional scales. Making a MAPS image requires an operating system base image file, an Intel compiler, and the source code installation file of MAPS. We put all three files in a file directory MAPS, and then add –r MAPS through the distributed wide area network cross-domain peer-to-peer transmission and storage system, slice the three files and store them all in the distributed wide area network cross-domain peer-to-peer transmission and storage system file system, and generate hash value unique identifiers for the three files. We access the three files through these three hash values ​​in the image creation script. The details are as follows:

[0122] hcat QmbzgnXP7kZycSjsZ1Rgoat7sscygA21syFEn2D2KwrF8i | tar Jxf - -Crootdir

[0123] This command decompresses a centos-8.5 operating system base image file to the rootdir directory. QmbzgnXP7kZycSjsZ1Rgoat7sscygA21syFEn2D2KwrF8i is the hash value generated by the centos-8.5 image file.

[0124] hcat QmP7zpNTARVB5UizokbUkWf9hBMiRKznDvhKtbUnRA9c8s | tar Jxf - -Crootdir / opt /

[0125] This command decompresses an Intel oneapi compiler installation file to the rootdir / opt directory. QmP7zpNTARVB5UizokbUkWf9hBMiRKznDvhKtbUnRA9c8s is the hash value generated by the Intel oneapi compiler.

[0126] hcat QmbwP3ZVztD1Ee8YwtLoSnhMWcjn4xaqaixEydDuKLJjm2 | tar xf - -Crootdir / opt / repack /

[0127] This command decompresses the source code installation file of MAPS to the rootdi / opt / repack directory. QmbwP3ZVztD1Ee8YwtLoSnhMWcjn4xaqaixEydDuKLJjm2 is the hash value generated by the MPAS-Model-7.0 source code installation file.

[0128] Specific Example 3

[0129] Diamond is a software tool for aligning large-scale protein and nucleic acid sequences. It can efficiently process high-throughput sequencing data, especially in metagenomics and transcriptomics research. To make a diamond image for the ARM architecture, we need the base image of the ARM operating system and the diamond source code installation package. We save the ARM operating system base image file and the diamond installation file in the distributed wide area network cross-domain peer-to-peer transmission and storage system through slicing, and generate unique hash values ​​for the two files. We call the hash values ​​of the two files to access the two files in the creation of the diamond image. The details are as follows:

[0130] # openEuler-20.03LTS_SP3-aarch64-dev.tar.xz

[0131] hcat Qmdca9tToXcrBF649LFfSdfn5gteCUjm77aeQcutpXnq1p | tar Jxf - -Crootdir

[0132] This command will access the base image file of openEuler-20.03 through the hash value and unzip the file to the rootdir directory.

[0133] # diamond-0.9.14.tar.gz

[0134] hcat QmT9x8UTBVgDzpyXcwyLFeJ2y9s5K9jwzKMZPS37ymxr25 | tar zxf - -Crootdir / opt / repack /

[0135] This command uses the hash value of the diamond installation file stored in the distributed wide area network cross-domain peer transmission and storage system to access the diamond-0.9.14 installation file, and unpacks the installation file to the rootdir / opt / repack directory.

[0136] Through the above process, after the image file is produced in the above three specific examples, all installation packages required for the image are permanently stored in the distributed wide area network cross-domain peer-to-peer transmission and storage system file system. Even if the original official website of the software no longer provides the software installation file or the download link of this file is invalid, it will not affect the production of the software image, thereby ensuring the repeatability of image production.

[0137] If the software packages that image production depends on are invalid, image production will fail. By making these software packages persistent, the image can be rebuilt at any time as long as the file content remains unchanged. In addition, the idea of ​​Git is borrowed to connect different versions of the same file through version control, so that file modifications and updates can be tracked.

[0138] The distributed wide area network cross-domain peer-to-peer transmission and storage system adopted by the present invention is a distributed point-to-point file system, which identifies content by the content itself rather than the location of the content. No matter where the file is stored, it can be accessed by the same unique value. This use of encrypted hash function ensures the integrity of the file, and can also prevent the file from being tampered with during the transmission process, thereby increasing the security of the system. At the same time, the file is stored on multiple nodes in the network. As long as there is a node, the file will not be lost, which improves the reliability and persistence of the data. Moreover, the distributed wide area network cross-domain peer-to-peer transmission and storage system adopts a block-level link mode to store files, and only transmits the requested blocks, thereby reducing the consumption of network bandwidth.

[0139] These key points together form the basis of the distributed wide area network cross-domain peer-to-peer transmission and storage system file system, making it an efficient, reliable and secure distributed file storage and sharing solution.

[0140] It can be seen from the technical solution provided by the above-mentioned embodiments of the present invention that the present invention introduces a distributed wide area network cross-domain peer-to-peer transmission and storage system file system in the image building process, so that all software installation packages and their dependencies required in the image building process are saved in the distributed wide area network cross-domain peer-to-peer transmission and storage system file system, thereby realizing the permanent preservation of these files, thereby ensuring the repeatability of the image building at any time, and providing a permanent and effective image production solution for users' calculations.

[0141] The key technical point of the present invention is how to use the distributed wide area network cross-domain peer-to-peer transmission and storage system file system to perform persistent storage of files, thereby solving the problem of dependent file failure during the container image construction process in a high-performance computing environment. Specifically, the following aspects constitute the core technical points of the present invention:

[0142] 1. Introduction and initialization of distributed WAN cross-domain peer-to-peer transmission and storage system file system:

[0143] The primary key point of the present invention is to introduce a distributed wide area network cross-domain peer-to-peer transmission and storage system as the core storage system in the image building process. By initializing the distributed wide area network cross-domain peer-to-peer transmission and storage system nodes and starting the daemon process, a stable distributed wide area network cross-domain peer-to-peer transmission and storage system network connection is constructed, so that it can realize file distribution and storage in a distributed environment. This step ensures that the system can fully utilize the decentralized characteristics of the distributed wide area network cross-domain peer-to-peer transmission and storage system, providing a solid foundation for subsequent file management.

[0144] 2. Distributed and encrypted storage of files:

[0145] In the distributed wide area network cross-domain peer-to-peer transmission and storage system, the application software installation package and dependent files are segmented into multiple small blocks and distributedly stored in different nodes in the network. The distributed wide area network cross-domain peer-to-peer transmission and storage system generates a unique identifier for each segmented file based on the encryption of the quadratic remainder problem. This unique encrypted identifier is then used to generate the overall hash value check code of the software installation package or dependent file, and this hash check code is used to reference and access the file instead of relying on the physical location of the file in a certain server or path. This hash identifier not only ensures the uniqueness and integrity of the file content, but also prevents the file from being tampered with during transmission, ensuring the security and reliability of image construction.

[0146] 3. File version control and traceability:

[0147] The present invention draws on the version control concept of Git, and through the hash identifier system of the distributed wide area network cross-domain peer-to-peer transmission and storage system, it is possible to track and manage different versions of software installation packages and their dependent files. Each time a file is updated or modified, a new hash value is generated, thereby closely linking the version change with the file content. In this way, users can easily manage and switch different versions of dependent files during the image building process, ensuring that the image building process is traceable and controllable. This version control mechanism effectively avoids build failures or runtime errors caused by version incompatibility.

[0148] 4. Deduplication and storage optimization of distributed WAN cross-domain peer-to-peer transmission and storage systems:

[0149] Another key point of the distributed WAN cross-domain peer-to-peer transmission and storage system file system is its deduplication function, that is, identifying and removing duplicate files across the entire network. In a high-performance computing environment, dependent files of different application software may have the same version or content. The deduplication function of the distributed WAN cross-domain peer-to-peer transmission and storage system can save storage space and network bandwidth and reduce unnecessary resource consumption. This feature not only reduces the operating cost of the system, but also optimizes resource utilization, making the image building process more efficient and economical.

[0150] 5. Persistence and high availability of files during image building:

[0151] During the image building process, all dependent files involved, such as ISO images, source code packages, patches, etc., are added to the distributed wide area network cross-domain peer-to-peer transmission and storage system file system and referenced using the unique hash value check code generated by it, ensuring that these files can still be accessed through the distributed wide area network cross-domain peer-to-peer transmission and storage system network even if the original network link fails or the server is offline. This persistence method greatly improves the repeatability and high availability of image building, especially in scientific computing and research environments with long spans, ensuring the continuous availability of data and files.

[0152] 6. Reliability and speed of distributed file access:

[0153] The distributed WAN cross-domain peer-to-peer transmission and storage system file system adopts a point-to-point distributed file transmission mechanism, allowing users to obtain file data from multiple nodes in parallel. This not only improves the speed of file access, but also enhances the reliability of the system. During the image building process, there is no need to rely on a single server or centralized network resources. Users can efficiently obtain the required files in a distributed environment, reducing the impact of network congestion and bandwidth bottlenecks on image production. This mechanism is particularly suitable for image building scenarios with high bandwidth requirements, effectively improving the overall performance of the system.

[0154] 7. Integration of distributed WAN cross-domain peer-to-peer transmission and storage systems with high-performance computing environments:

[0155] Finally, the key point of the present invention is to successfully integrate the distributed wide area network cross-domain peer-to-peer transmission and storage system file system with the container image building process in the high-performance computing HPC environment. This integration not only ensures the efficiency and stability of high-performance computing applications, but also makes complex multi-version dependency management simpler and controllable. In actual operation, by pre-configuring the distributed wide area network cross-domain peer-to-peer transmission and storage system network and file storage strategies, users can quickly build and deploy reliable container images in the HPC environment, improving the ease of use and scalability of the entire computing platform.

[0156] The above embodiments of the present invention are merely examples for clearly illustrating the technical solutions of the present invention, and are not intended to limit the specific implementation methods of the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the claims of the present invention shall be included in the protection scope of the claims of the present invention.< / url>

Claims

1. A method for permanent storage of dependencies required in an image building process, characterized in that: The method comprises: S1. Segment-encode the target application, other third-party files that the target application depends on, and the operating system environment files required to build the target file using an encryption algorithm based on the quadratic remainder problem; S2, configuring the distributed wide area network cross-domain peer-to-peer transmission and storage system, and then sending the segmented encoded data to the configured distributed wide area network cross-domain peer-to-peer transmission and storage system; S3. When the segmented encoded data is sent to the distributed wide area network cross-domain peer transmission and storage system, the differently encoded data is sent to different storage nodes for storage, and the same encoded data is sent to multiple data storage nodes for storage; the data is sent in a concurrent manner; S4, when sending the segmented encoded data to the distributed wide area network cross-domain peer-to-peer transmission and storage system, calculating the overall check code of the file data from the different encoded data segments, and encoding the check code and sending it to the distributed wide area network cross-domain peer-to-peer transmission and storage system; S5. The overall checksum of all segmented coded data is the unique identifier of the file. Any data change at any position of the original data will result in a change in the final identifier. S6. Modify the image definition file, and change the installation package and dependencies originally obtained through the network link to the acquisition of all dependent files involved in the image building process by using the overall verification code of the file data as a unique identifier; S7. After completing the modification of the image definition file, start building the image. Obtain all dependent files involved in the image building process from the distributed wide area network cross-domain peer transmission and storage system to perform the image building operation. After the building is completed, test the generated image file.

2. A method for permanent storage of dependencies required in a mirror image building process according to claim 1, characterized in that: The encryption algorithm based on the quadratic remainder problem refers to determining whether there is an integer x such that when the square of x is calculated, the remainder of the integer x divided by p is exactly a, given an integer p and an odd prime number a.

3. The method for permanent storage of dependencies required in the image building process according to claim 1, characterized in that: The data in the distributed wide area network cross-domain peer-to-peer transmission and storage system is segmented and distributed and stored on different nodes according to a certain strategy; the data transmission and storage in the system spans multiple geographical areas or network domains; the data stored in the system uses the encoding of each segment of data after segmentation as a key value to find the storage node corresponding to the data and access the corresponding segment of data; and each storage node in the system has a unique ID, and supports dynamic joining and leaving of nodes through a hash ring algorithm; The hash ring algorithm is to solve the problem of data load balancing among dynamically distributed nodes by mapping data and nodes to a logical ring.

4. The method for permanent storage of dependencies required in the image building process according to claim 1, characterized in that: In the process of building the image described in S7, the unique identifier is used to concurrently transmit the required content stored in the distributed wide area network cross-domain peer transmission and storage system from the storage nodes corresponding to each segment.

5. The method for permanent storage of dependencies required in the image building process according to claim 3, characterized in that: If the storage node is behind a firewall, a pipeline for direct data transmission is preferably established through UDP hole punching technology. If the pipeline establishment fails, data routing and forwarding are performed through a server with a public network address to obtain data.

6. The method for permanent storage of dependencies required in the image building process according to claim 1, characterized in that: The configuration of the distributed wide area network cross-domain peer-to-peer transmission and storage system specifically refers to: Initialize the node: Initialize the distributed wide area network cross-domain peer-to-peer transmission and storage system node, which will create a local node for interaction with the distributed wide area network cross-domain peer-to-peer transmission and storage system network; at the same time, set up a health check mechanism during the node initialization stage to detect hardware or configuration anomalies that the node will encounter during the daemon startup stage; Start the daemon process: Start the distributed WAN cross-domain peer-to-peer transmission and storage system daemon process to ensure that the node is always online and can exchange data with other nodes, and ensure the distributed WAN cross-domain peer-to-peer transmission and storage system nodes and the distributed WAN cross-domain peer-to-peer transmission and storage system network connection; at the same time, the daemon process has an abnormal node isolation mechanism. When a node has abnormal data transmission or is offline for a long time, the system can automatically isolate it and notify the administrator to check; Network connection test: Test the connectivity between distributed WAN cross-domain peer-to-peer transmission and storage system nodes and other nodes in the network, as well as the connectivity of seed servers, to ensure the stability of data transmission; During the network connection test phase, a multi-path transmission strategy was adopted. The distributed wide area network cross-domain peer transmission and storage system nodes and other nodes in the network selected the best path to transmit data, while supporting multi-path transmission and dynamic path switching.

7. A method for permanent storage of dependencies required in a mirror image building process according to claim 1, characterized in that: Each time any modification is made to all dependent files involved in the image building process and the final image file, a new unique identifier is generated and different versions of the files are managed by recording these unique identifiers.

8. A method for permanent storage of dependencies required in a mirror image building process according to claim 7, characterized in that: When the image needs to be updated, the new unique identifier is used to obtain the installation package and the dependent files to build the image and store the old version of the image and the dependent files.

Images