A mirror deployment method and system based on mirror acceleration technology

Abstract

The application discloses a mirror deployment method and system based on a mirror acceleration technology, and the mirror deployment method based on the mirror acceleration technology comprises the following steps: performing format conversion on a first mirror file in a target platform to obtain a second mirror file; the format of the first mirror file is an OCI format, and the format of the second mirror file is a Nydus format; comparing mirror files already stored in a mirror pool, performing deduplication on the second mirror file, and storing the deduplicated second mirror file into the mirror pool; in response to an export task, packaging data corresponding to a target mirror file in the mirror pool to form a deployment data packet, the target mirror file being a mirror file that needs to be deployed in a container in a guest environment; and based on a snapshotter, deploying the deployment data packet in the container to deploy the target mirror file in the container, the snapshotter being used for understanding and running the mirror file in the Nydus format. The application aims to improve the container startup speed, optimize the storage efficiency and enhance the security during runtime.

Description

Technical Field

[0001] This invention relates to the field of image technology, and more specifically to an image deployment and system based on image acceleration technology. Background Technology

[0002] In the field of cloud-native computing, container technology has become the standard way to deploy and run applications. Deployment platforms (such as the Sisyphus platform) are responsible for managing various deployment media, including container images, and for completely migrating them from the development environment to the customer environment.

[0003] To optimize container image export efficiency, existing technologies typically employ image pooling. This approach ensures uniqueness by storing images uniformly in a global directory, utilizes hard links for rapid image file reuse, and introduces concurrency control mechanisms to avoid data conflicts during multi-task export. This approach reduces redundant image fetching and storage redundancy to some extent, constituting the current mainstream image distribution and management model based on the OCI (Open Container Initiative) specification.

[0004] However, the aforementioned distribution model based on traditional OCI images has inherent technical flaws. First, regarding container startup performance, this model requires containers to completely download and decompress all data layers of the image before startup, making startup time heavily dependent on image size and network bandwidth, which is insufficient to meet the stringent millisecond-level startup requirements of scenarios like Function Compute. Second, in terms of storage and transmission efficiency, the layered structure of OCI images lacks the ability to deduplicat block-level data across layers or images, resulting in the repeated storage and transmission of identical data blocks, leading to significant waste of storage space and network bandwidth. Furthermore, regarding security and observability, existing technologies cannot provide runtime data access insights, making it difficult to optimize data layout and adjust prefetching strategies. They also lack end-to-end integrity verification mechanisms from data source to runtime, failing to effectively prevent data tampering risks during supply chain attacks. Summary of the Invention

[0005] To address the shortcomings of existing technologies, this invention discloses an image deployment method and system based on image acceleration technology, aiming to improve container startup speed, optimize storage efficiency, and enhance runtime security.

[0006] This invention discloses a method for image deployment based on image acceleration technology, comprising: The first image file within the target platform is converted to a second image file; the first image file is in OCI format, and the second image file is in Nydus format. Compare the second image file with the image files already stored in the image pool, remove duplicates from the second image file, and store the deduplicated second image file into the image pool. In response to the export task, the data corresponding to the target image file in the image pool is packaged into a deployment data package. The target image file is the image file that needs to be deployed in the container in the customer environment. Based on the snapshotter, deployment packages are deployed in containers to deploy target image files within the containers. The snapshotter is used to understand and run Nydus format image files.

[0007] Preferably, the second image file is deduplicated by comparing it with the image files already stored in the image pool, and the deduplicated second image file is stored in the image pool, including: The second image file is split to obtain at least one data block; Determine if each data block exists in the mirror pool; Store data blocks that do not exist in the mirror pool into the mirror pool.

[0008] Preferably, in response to the export task, the data corresponding to the target image file in the image pool is packaged into a deployment data package, including: The snapshotter retrieves the metadata layer and data block index corresponding to the target image file from the image pool. Based on the hard link mechanism of the image pool, the metadata layer, data block index and other deployment media are packaged together to form a deployment data package.

[0009] Preferably, based on a snapshotter, the deployment package is deployed in a container, including: Configure and enable the snapshotter in the container runtime of the guest environment; Mount the metadata layer via a snapshotter; Based on the data block index, the target data block corresponding to the target image file is retrieved from the metadata layer.

[0010] Preferably, configuring and enabling the snapshotter in a container runtime within a guest environment includes: Import the deployment data package into the target platform of the customer environment; Configure the Kubernetes cluster in the guest environment to use Containerd as the container runtime and configure the snapshotter.

[0011] Preferably, based on the data block index, the target data block corresponding to the target image file is retrieved from the metadata layer, including: Determine the data block index corresponding to each target data block; Based on each data block index, download the target data block corresponding to each data block index from the image pool to the container.

[0012] Preferably, the image deployment method based on image acceleration technology also includes: Perform end-to-end integrity verification on the target data block being pulled.

[0013] Preferably, the first image file uploaded to the target platform is converted to a new format to obtain a second image file, including: Integrate the Nydus toolchain within the target platform's internal outsourcing environment; When the upload of the first image file is detected or the export task of the first image file is triggered, the Nydus toolchain is invoked to pull the first image file from the target platform. The first image file pulled is converted into a Nydus format image using the Nydus toolchain to obtain the second image file.

[0014] Preferably, the second image file is compatible with the OCI specification. Image deployment methods based on image acceleration technology also include: Store the second image file in the target platform's OCI-compatible backend storage.

[0015] This invention discloses an image deployment system based on image acceleration technology, used to execute an image deployment method based on image acceleration technology, comprising: The format conversion module is configured to convert the format of the first image file in the target platform to obtain the second image file; the format of the first image file is OCI format, and the format of the second image file is Nydus format. The image deduplication module is configured to: compare the second image file with the image files already stored in the image pool, deduplicate the second image file, and store the deduplicated second image file into the image pool; The image export module is configured to: in response to an export task, package the data corresponding to the target image file in the image pool into a deployment data package, where the target image file is the image file that needs to be deployed in a container in the customer environment; The image deployment module is configured to deploy deployment packages within containers based on a snapshotter to deploy target image files within the containers. The snapshotter is used to understand and run Nydus format image files.

[0016] Compared with the prior art, the beneficial effects of the present invention are as follows: This invention significantly improves the efficiency of image pulling and container startup by introducing Nydus format image storage and an on-demand loading mechanism, especially saving substantial bandwidth and storage resources in large-scale deployment scenarios. Nydus format's block-level deduplication mechanism effectively reduces the storage overhead of the image repository, while the runtime on-demand loading feature allows containers to start without waiting for the complete image to download, achieving second-level cold starts. Furthermore, because Nydus images are compatible with the OCI specification, existing infrastructure can be smoothly integrated without modification, ensuring system compatibility and maintainability. In addition, the end-to-end integrity verification mechanism ensures the reliability of data during transmission and loading, effectively preventing malicious tampering and data corruption risks, and enhancing runtime security. Attached Figure Description

[0017] Figure 1 A flowchart illustrating the image deployment method based on image acceleration technology provided by this invention; Figure 2 A schematic diagram of the accelerated export process of the Nydus-integrated Sisyphus image provided by this invention. Figure 3 A schematic diagram illustrating the runtime acceleration process of the Nydus image provided by this invention in a customer environment; Figure 4 This is a schematic diagram of the structure of the image deployment system based on image acceleration technology provided by the present invention. Detailed Implementation

[0018] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0019] The present invention will now be described in further detail with reference to the accompanying drawings.

[0020] like Figure 1 As shown in the figure, this embodiment of the invention provides a method for image deployment based on image acceleration technology, including the following steps.

[0021] S1. Convert the format of the first image file in the target platform to obtain the second image file.

[0022] The first image file is in OCI format, and the second image file is in Nydus format.

[0023] In this embodiment of the invention, the conversion of the image file format facilitates efficient deduplication of the image file based on the Nydus format in subsequent processes, thereby reducing storage space usage and improving data transmission efficiency. Furthermore, it facilitates the subsequent utilization of the lazy loading and hierarchical indexing features supported by the Nydus format to achieve efficient organization and on-demand reading of the image data.

[0024] For example, the target platform could be the Sisyphus platform.

[0025] In some embodiments, within the target platform's internal outbound environment, the Nydus toolchain is integrated. When the upload of the first image file is detected or an export task for the first image file is triggered, the Nydus toolchain is invoked to pull the first image file from the target platform. The pulled first image file is then converted into a Nydus format image using the Nydus toolchain to obtain the second image file.

[0026] In this way, by integrating the Nydusify toolchain, the original OCI image is converted in the target platform's internal output environment to generate a Nydus image that is compatible with the OCI specification.

[0027] For example, this process utilizes Nydus's hierarchical metadata mechanism to reconstruct the image's file system structure into block-level data units that can be loaded on demand, while maintaining compatibility with existing Registry protocols, ensuring that the converted image can be directly pushed to backend storage such as Nexus or Harbor.

[0028] In some embodiments, the second image file is compatible with the OCI specification, and therefore can be directly stored in the target platform's OCI-compatible backend storage. This allows the converted image file to be stored directly on the target platform without any modification to the existing storage architecture. This process not only improves image storage efficiency but also lays the foundation for subsequent on-demand loading and runtime acceleration, further optimizing overall deployment performance.

[0029] Specifically, within the Sisyphus platform's internal outsourcing environment, before uploading or exporting images, the official Nydus toolchain is integrated, such as the command-line tool Nydusify or the build tool Buildkit. These tools are responsible for efficiently converting standard OCI format container images to Nydus format. Nydusify, as a standalone CLI tool, can directly pull OCI images from the source Registry and complete the conversion. Crucially, the converted Nydus images remain format-compatible with the OCI specification, thus allowing seamless storage on the Sisyphus platform's existing OCI-compatible backend storage, such as Nexus or Harbor, without requiring modifications to the storage infrastructure.

[0030] S2. Compare the image files already stored in the image pool, deduplicate the second image file, and store the deduplicated second image file into the image pool.

[0031] In this embodiment of the invention, the block-level data deduplication mechanism of Nydus format is used to identify and eliminate data blocks in the second image file that are duplicated with existing images in the image pool, retaining only a unique physical copy, thereby significantly reducing storage space usage.

[0032] Leveraging Nydus' advanced "chunk deduplication" feature, duplicate data blocks are identified and eliminated throughout the entire lifecycle of an image (including storage, export, and distribution), thereby significantly reducing storage space, network bandwidth, and runtime memory consumption.

[0033] Understandably, while the layered structure of OCI images in related technologies provides a certain degree of data reuse, it lacks a block-level data deduplication mechanism across layers or images. Therefore, even if multiple images or different layers contain a large number of identical data blocks (e.g., shared library files or basic data), these data blocks are still repeatedly stored in the image pool and repeatedly transmitted during export and distribution. This not only wastes valuable storage resources but also significantly increases network transmission costs and time. This invention, through Nydus's block-level deduplication mechanism, achieves cross-image and cross-layer data block sharing, effectively solving this problem. It not only significantly reduces the storage space occupied by the Sisyphus image pool but also significantly compresses the size of exported data packets.

[0034] Specifically, during the storage phase, Nydus' core feature—"block-level data deduplication"—is fully utilized. When Nydus format images are stored in the image pool, this mechanism can automatically identify and eliminate duplicate data blocks across images and layers, retaining only one copy of the physical data, thereby greatly optimizing the storage efficiency of the image pool.

[0035] For example, this process can be automatically triggered when the image file is written to the image pool after the image file format is converted, without the need for additional scheduling. It supports global deduplication capabilities across images and layers, further improving storage density.

[0036] Specifically, when deduplicating the second image file, it needs to be split into at least one data block. Then, it's determined whether each data block exists in the image pool. If a data block does not exist in the image pool, it is stored in the image pool.

[0037] S3. In response to the export task, package the data corresponding to the target image file in the image pool into a deployment data package. The target image file is the image file that needs to be deployed in a container in the customer environment.

[0038] In this embodiment of the invention, since block-level deduplication across images and layers has been implemented in the image pool, the exported target image file only needs to package its metadata and a unique data block index, significantly reducing the deployment package size. In the component export process, the role of the image-manager changes: it no longer pulls the large traditional OCI image tar package, but instead pulls optimized Nydus format image files (mainly containing metadata and data block indexes). Therefore, the final exported data package directory structure will contain Nydus format files.

[0039] For example, the deployment package can be quickly imported into the guest environment and pushed to the local Registry, and combined with the snapshotter (nydus-snapshotter) to achieve on-demand loading, ultimately achieving a complete closed loop of second-level startup and efficient operation.

[0040] In some embodiments, in response to an export task, the data corresponding to the target image file within the image pool is packaged into a deployment data package, including: pulling the metadata layer and data block index corresponding to the target image file from the image pool using a snapshotter. Based on the hard link mechanism of the image pool, the metadata layer, data block index, and other deployment media are packaged together to form the deployment data package. After packaging, the deployment data package can be distributed to the guest environment through a secure channel.

[0041] For example, by pulling the metadata layer and data block index corresponding to the target image file from the image pool through the snapshotter, on-demand reading and partial loading of image data can be achieved without pre-downloading the complete image. This metadata layer contains file system hierarchy information and data block addressing indexes. Combined with the Nydus format image already pushed in the local Registry, it is resolved and mounted as a lazy-loaded read-only file system by nydus-snapshotter when the container starts. When the application accesses a specific file, the driver pulls the corresponding data blocks from the remote repository on demand, greatly reducing cold start waiting time.

[0042] For example, the system can extract the metadata and data blocks corresponding to the target image file from the image pool and map these contents to a temporary packaging directory using hard link technology, avoiding redundant data copying. Subsequently, other deployment media are aggregated to build a complete deployment package, which is then subjected to integrity verification and digital signature to ensure the security and consistency of the distribution process.

[0043] For example, other deployment media include Linux software installation packages (such as rpm and deb), container images (docker-image), and Helm Charts. The target platform uses a series of managers, such as image-manager, to implement core management functions for container images, including push, load, save, and build. During component export, the platform iterates through the list of installationMedia defined in the deployment orchestration. Subsequently, docker-manager pulls all necessary image dependencies from its internal Nexus repository and packages them into a complete deployment package.

[0044] S4. Based on the snapshotter, deploy deployment packages within the container to deploy the target image file inside the container. The snapshotter is used to understand and run Nydus format image files.

[0045] In this embodiment of the invention, a snapshotter is used to parse and run the Nydus image, so as to load the data required for the deployment target image file on demand, thereby loading only the necessary data blocks when the container starts, avoiding the delay caused by pulling the full image.

[0046] This invention fundamentally changes the way images are loaded, achieving "second-level container startup" or "millisecond-level function computation code package loading speed" in scenarios such as function computation, completely eliminating the constraint of image size on startup performance. By introducing an "on-demand load" mechanism, containers no longer need to download the entire image at startup. Instead, data is downloaded from the remote location only in extremely small chunks when the application actually needs to access specific data, thereby fundamentally accelerating the container startup process.

[0047] Furthermore, it's understandable that the OCI format mechanism requires that all layers of the image be completely downloaded and decompressed to the local file system before the container starts. This means that the container's startup time is directly related to the image size and network bandwidth. For large images or in environments with poor network conditions, startup latency can be as long as several minutes, which is completely unacceptable for scenarios with stringent requirements for high performance and high concurrency, such as Function Compute (which requires millisecond-level response) or rapid scaling of large-scale clusters. The Nydus format, by introducing a lazy loading mechanism, pulls the necessary data blocks only as needed during container runtime, significantly reducing startup latency and achieving cold starts in seconds or even milliseconds.

[0048] Compared to the lengthy wait required for the complete image pull and decompression in the traditional OCI mode, this invention drastically reduces image pull time from minutes to seconds, fundamentally improving container startup speed. Especially in high-concurrency function computation scenarios where startup latency is extremely sensitive, the code package loading speed can even reach millisecond levels, providing robust performance guarantees for applications such as Serverless.

[0049] Specifically, in the guest environment, after the deployment package is imported, the Nydus format image contained therein is pushed to the local Registry / Nexus of the guest environment. In order for containers to understand and run Nydus images, the dedicated snapshotter provided by Nydus needs to be configured and enabled in the container runtime of the guest environment (such as Containerd, Docker / Moby, KataContainers, CRI-O / Podman).

[0050] The core function of nydus-snapshotter, as a plugin, is to intercept image mount requests from container runtimes (such as Containerd), thereby bypassing the traditional layer-by-layer download and decompression process. When a container starts, nydus-snapshotter mounts a virtual file system based on FUSE (user space) or EROFS (kernel space), which contains only the image's metadata. Subsequently, when the application inside the container actually initiates file read / write requests, this virtual file system pulls the corresponding data blocks from the remote Registry on demand, in "chunk units," thus achieving container startup in seconds.

[0051] In some embodiments, deploying a deployment package in a container based on a snapshotter includes: configuring and enabling the snapshotter in a container runtime of a guest environment; mounting a metadata layer via the snapshotter; and pulling the target data block corresponding to the target image file from the metadata layer based on the data block index.

[0052] After mounting the metadata layer, the snapshotter constructs a lazy-loaded virtual file system view. During container startup, it only resolves path and permission information without triggering actual data downloads. When an application accesses a file for the first time, the file system locates the corresponding chunk using the data block index. The snapshotter then pulls the chunk from the local registry on demand and delivers it transparently, achieving I / O-level elastic loading. The entire process requires no modification to application logic, is compatible with the OCI standard, and ensures consistency and security across environments.

[0053] To parse and run the Nydus image, a snapshotter needs to be configured and enabled in the container runtime of the guest environment before image deployment. This process includes: importing the deployment package into the target platform of the guest environment; configuring the Kubernetes cluster of the guest environment to use Containerd as the container runtime; and configuring the snapshotter. After the deployment package is imported, the platform automatically pushes the Nydus image to the local Registry and triggers the container runtime configuration verification process. Once it is confirmed that nydus-snapshotter is enabled in Containerd, the deployment system begins creating Pod instances. At this point, kubelet initiates a container creation request, and Containerd delegates the image root filesystem mounting to nydus-snapshotter. The metadata layer is quickly loaded and built as a read-only view, and the EROFS driver intervenes to intercept file access. When the kube-apiserver process starts and attempts to read the core binary, the on-demand fetch mechanism immediately takes effect, retrieving only the necessary data blocks from the Registry, significantly reducing initialization latency.

[0054] In some embodiments, the process of fetching the database on demand includes: determining the data block index corresponding to each target data block; downloading the target data block corresponding to each data block index from the image pool to the container according to each data block index; and dynamically populating the corresponding location in the virtual file system after the target data block is downloaded, thus achieving transparent reading.

[0055] Furthermore, to enhance data security, end-to-end integrity verification of the retrieved target data blocks is also required. This provides a robust security defense for the deployment media throughout its distribution and operation, ensuring effective detection and prevention of potential supply chain attacks during runtime.

[0056] For example, the verification process uses pre-defined hash digest information to perform cryptographic verification on each data block immediately after it is pulled, ensuring that its content is consistent with that generated at the source, preventing tampering or injection of malicious code during transmission. Furthermore, all verification records are auditable and linked to the Kubernetes event system, triggering alarms and blocking container startup when a mismatch is detected, thereby achieving a secure and controllable runtime environment.

[0057] Understandably, existing file system layers lack the ability to analyze and access images, files, and data blocks at runtime. This makes fine-grained data layout optimization, prefetching strategy adjustments, or incremental build optimizations based on actual usage extremely difficult. More importantly, the traditional model lacks an end-to-end data integrity verification mechanism from the data source to the runtime, making it particularly vulnerable to increasingly sophisticated supply chain attacks and unable to effectively detect and prevent the injection or tampering of malicious data at runtime.

[0058] This invention deeply integrates Nydus, as a high-performance file system solution, into the entire lifecycle of image processing on the Sisyphus platform, covering all key stages from image building and export storage to runtime loading, thereby achieving comprehensive performance, efficiency, and security improvements.

[0059] Specifically, during runtime, the Nydus file system performs end-to-end integrity checks on each data block fetched on demand, ensuring that data remains untampered throughout its journey from source to runtime, effectively preventing supply chain attacks. Furthermore, leveraging Nydus's Data Analyzable feature, access patterns and frequencies of files and data blocks can be recorded. This data can be used for future data layout optimization, intelligent prefetching strategy development, and detection of abnormal access behavior.

[0060] For example, the present invention can be seamlessly integrated with P2P distribution systems such as Dragonfly, thereby further saving the network bandwidth required for image distribution in large cluster environments, effectively reducing the network load of the central Registry, and improving distribution efficiency.

[0061] Understandably, Nydus ensures comprehensive file system compatibility with existing workloads by supporting multiple standard interfaces, including FUSE (user space), VirtioFS (virtualization), and the high-performance In-Kernel EROFS (kernel space). In other words, applications can run on Nydus without any modifications, guaranteeing smooth integration of the solution and seamless migration of applications.

[0062] To illustrate the implementation methods and practical effects of the present invention more specifically, two specific embodiments are given below.

[0063] The first embodiment details the process of image conversion, storage, and export. This embodiment describes how to process a standard OCI image named k8s-deploy / kube-apiserver:v1.21.5 within the Sisyphus internal outbound environment.

[0064] First, upload the standard image: Users upload a standard k8s-deploy / kube-apiserver:v1.21.5 OCI image using the trailer push command on the sisyphus platform.

[0065] Secondly, the uploaded image undergoes format conversion: After the platform detects a new image upload, it automatically calls the integrated Nydusify toolchain. Nydusify pulls the OCI image from its internal Nexus and converts it to Nydus format.

[0066] Furthermore, Nydus efficiently deduplicates and stores the converted files: During the conversion process, Nydus' block-level deduplication algorithm analyzes the image content. It splits the image file system into multiple data blocks and compares them with all existing data blocks in the image pool, storing only new, non-duplicate data blocks into physical storage, while generating metadata pointing to these data blocks.

[0067] Finally, streamlined export packaging: When a user initiates an export task containing this image, docker-manager no longer needs to pull the complete OCI image tar package. Instead, it only needs to pull the corresponding Nydus format artifacts (such as an OCI-compatible layer containing metadata and block indexes) from the image pool and package them into the deployment package, resulting in a smaller export package and faster packaging speed.

[0068] The second embodiment specifically illustrates on-demand accelerated startup in a guest environment. This embodiment describes how to deploy and accelerate the startup of the k8s-deploy / kube-apiserver:v1.21.5 image in a guest environment.

[0069] First, the deployment package is imported: The administrator imports the deployment package containing the Nydus format image into the sisyphus platform in the guest environment. The import process will push the Nydus image to the local Registry / Nexus in the guest environment.

[0070] Secondly, configure the runtime: the Kubernetes cluster in the guest environment has been pre-configured to use Containerd as the container runtime and nydus-snapshotter has been enabled as a custom snapshotter.

[0071] Furthermore, on-demand Pod startup: When deployment orchestration requires starting a Pod using the kube-apiserver image, the internal mechanism is as follows: The kubelet sends a request to Containerd to create a container.

[0072] Containerd calls nydus-snapshotter to prepare the root file system of the image. The container can then begin the startup process without waiting for the entire image data to be downloaded.

[0073] nydus-snapshotter immediately starts the Nydus file system driver (e.g., high-performance EROFS) and mounts the image's metadata layer.

[0074] When the kube-apiserver process inside the container starts running, it attempts to read configuration files, binary files, etc. Whenever a file access occurs, the EROFS driver intercepts the request and pulls the corresponding data from the remote registry on demand (on-demand load) in very small chunks (chunk units).

[0075] Finally, the quantitative results were achieved: because only a small amount of data necessary for startup was downloaded, the startup time of the Pod was successfully reduced from several minutes in the traditional OCI mode to within "several seconds", achieving a significant performance improvement.

[0076] The following combination Figure 2 This document explains the accelerated export process of the Sisyphus image after Nydus integration. Specifically, Figure 2 This demonstrates how to convert standard OCI images to Nydus format for export within the Sisyphus platform's internal export environment, thereby optimizing the traditional image packaging process.

[0077] S201. The user creates an export task on the Sisyphus platform. This begins the export of the image's deployment package.

[0078] S202. The system iterates through the `installationMedia` field associated with the export task to identify a list of all Docker images that need to be exported. This allows the system to obtain the media used for deployment.

[0079] S203. For each OCI image in the list, the system first checks whether the corresponding Nydus format version already exists in the internal Nexus storage. This allows the Nydus format of the image file to be checked.

[0080] S204. Determine if the Nydus format exists. If it exists, proceed directly to the Nydus image retrieval process, i.e., proceed to step S210. If it does not exist, execute steps S205-S209 to convert the file to Nydus format.

[0081] S205, Invoke the Nydusify toolchain.

[0082] S206, Nydusify pulls the original OCI image.

[0083] S207. Convert the original OCI image to Nydus format.

[0084] S208 performs efficient block-level data deduplication. This merges duplicate data blocks across mirrors, retaining only a unique copy, thus significantly reducing storage footprint.

[0085] S209. The result is stored back in Nexus. In this way, the converted and deduplicated Nydus image is pushed to the internal Nexus storage for reuse in subsequent export tasks.

[0086] S210 and docker-manager pull lightweight Nydus format image files (mainly containing metadata and data block indexes) from Nexus. This allows pulling Nydus images.

[0087] S211. Map the retrieved Nydus format image file to the export task directory using a hard link. This preserves the storage optimization features of the original image pool.

[0088] S212. Packaging complete. The system continues to process other types of deployment media (such as Helm Charts and RPM packages).

[0089] S213. Compress all content into a single deployment package.

[0090] It is worth noting that S203-S211 provide the specific process for processing each OCI image. This process can process multiple images in parallel, improving export efficiency.

[0091] The following combination Figure 3 This section describes the runtime acceleration process of the Nydus image in a customer environment. Specifically, Figure 3 This demonstrates how Nydus technology, through its runtime components, enables on-demand loading of image data when containers start in a guest environment, thereby fundamentally optimizing the container startup process.

[0092] S301. When a user deploys an application in a guest environment using Kubernetes, it triggers the creation process of a Pod. This initiates the deployment or startup of the container.

[0093] S302, Kubelet instructs container runtime (such as Containerd) to create containers.

[0094] S303, Containerd requests its configured snapshotter to prepare the container's root file system.

[0095] In this way, container runtime calls can be made via S302 and S303.

[0096] S304, Nyudus Snapshotter intercepts this request.

[0097] S305 and Nydus Snapshotter do not download the complete image. Instead, they use the Nydus runtime to quickly mount the metadata layer of the Nydus image using a POSIX-compliant file system interface (such as FUSE or EROFS).

[0098] S306, The file system is then ready.

[0099] In this way, the Nydus Snapshotter can be activated via S305 and S306.

[0100] S307. The container process begins execution and attempts to access files in its file system (such as reading configuration files, loading dynamic libraries, etc.). This is to start the application.

[0101] S308 and Nydus file system drivers (such as EROFS) detect these file access requests. This allows data blocks to be loaded on demand.

[0102] S309. Based on the file access offset and length, Nydus runtime calculates which data blocks are needed.

[0103] S310, pulls the necessary data blocks from the Nexus / Registry in the guest environment via the network only on a chunk unit basis.

[0104] In this way, data blocks can be remotely obtained via S309 and S310.

[0105] S311. Before providing data blocks to the application, Nydus performs an end-to-end data integrity check on each fetched data block to ensure that the data has not been tampered with. This is how data integrity is verified.

[0106] S312. The application continues execution after obtaining the required data.

[0107] This allows containers to run quickly. Because the entire process happens on demand and instantly, the container starts up and enters a service-ready state almost instantly. It is worth noting that S308-S311 provide the specific process for loading data blocks on demand.

[0108] like Figure 4 As shown, the present invention also provides an image deployment system based on image acceleration technology, used to execute an image deployment method based on image acceleration technology. The image deployment system based on image acceleration technology includes: a format conversion module 401, an image deduplication module 402, an image export module 403, and an image deployment module 404.

[0109] Specifically, the format conversion module 401 is configured to convert the format of the first image file in the target platform to obtain a second image file; the first image file is in OCI format, and the second image file is in Nydus format. The image deduplication module 402 is configured to compare the second image file with the image files already stored in the image pool, deduplicate the second image file, and store the deduplicated second image file in the image pool. The image export module 403 is configured to, in response to the export task, package the data corresponding to the target image file in the image pool into a deployment data package; the target image file is the image file that needs to be deployed in a container in the customer environment. The image deployment module 404 is configured to, based on a snapshotter, deploy the deployment data package in the container to deploy the target image file within the container; the snapshotter is used to understand and run Nydus format image files.

[0110] As can be seen from the above technical solution, the present invention discloses an image deployment method and system based on image acceleration technology. The image deployment method based on image acceleration technology includes: converting the format of a first image file in the target platform to obtain a second image file; the format of the first image file is OCI format, and the format of the second image file is Nydus format; comparing the second image file with the image files already stored in the image pool, deduplicating the second image file, and storing the deduplicated second image file in the image pool; responding to the export task, packaging the data corresponding to the target image file in the image pool into a deployment data package, wherein the target image file is the image file that needs to be deployed in a container in the customer environment; and deploying the deployment data package in the container based on a snapshotter to deploy the target image file in the container, wherein the snapshotter is used to understand and run the Nydus format image file.

[0111] This invention significantly improves the efficiency of image pulling and container startup by introducing Nydus format image storage and an on-demand loading mechanism, especially saving substantial bandwidth and storage resources in large-scale deployment scenarios. Nydus format's block-level deduplication mechanism effectively reduces the storage overhead of the image repository, while the runtime on-demand loading feature allows containers to start without waiting for the complete image to download, achieving second-level cold starts. Furthermore, because Nydus images are compatible with the OCI specification, existing infrastructure can be smoothly integrated without modification, ensuring system compatibility and maintainability. In addition, the end-to-end integrity verification mechanism ensures the reliability of data during transmission and loading, effectively preventing malicious tampering and data corruption risks, and enhancing runtime security.

[0112] The above are merely preferred embodiments of the present invention and are not intended to limit the present invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A method for image deployment based on image acceleration technology, characterized in that, include: The first image file within the target platform is converted to a different format to obtain the second image file; The first image file is in OCI format, and the second image file is in Nydus format; The process of comparing the second image file with the image files already stored in the image pool, deduplicating the second image file, and storing the deduplicated second image file into the image pool includes: splitting the second image file into at least one data block; determining whether each data block exists in the image pool; and storing the data blocks that do not exist in the image pool into the image pool. In response to the export task, the data corresponding to the target image file in the image pool is packaged into a deployment data package, including: pulling the metadata layer and data block index corresponding to the target image file in the image pool through a snapshotter; and packaging the metadata layer, the data block index, and other deployment media together into the deployment data package based on the hard link mechanism of the image pool; the target image file is an image file that needs to be deployed in a container in a customer environment. Based on the snapshotter, the deployment package is deployed in the container to deploy the target image file within the container. The snapshotter is used to understand and run the Nydus format image file.

2. The image deployment method based on image acceleration technology according to claim 1, characterized in that, The deployment of the deployment package in the container based on the snapshotter includes: Configure and enable the snapshotter in the container runtime of the guest environment; The metadata layer is mounted via the snapshotter; Based on the data block index, the target data block corresponding to the target image file is retrieved from the metadata layer.

3. The image deployment method based on image acceleration technology according to claim 2, characterized in that, Configuring and enabling the snapshotter in the container runtime of the guest environment includes: Import the deployment data package into the target platform of the customer environment; Configure the Kubernetes cluster in the guest environment to use Containerd as the container runtime, and configure the snapshotter.

4. The image deployment method based on image acceleration technology according to claim 2, characterized in that, The step of retrieving the target data block corresponding to the target image file from the metadata layer based on the data block index includes: Determine the data block index corresponding to each of the target data blocks; According to each of the data block indices, the target data block corresponding to each data block index is downloaded from the mirror pool to the container.

5. The image deployment method based on image acceleration technology according to claim 4, characterized in that, Also includes: Perform end-to-end integrity verification on the retrieved target data block.

6. The image deployment method based on image acceleration technology according to claim 1, characterized in that, The process of converting the format of the first image file uploaded to the target platform to obtain the second image file includes: Integrate the Nydus toolchain within the target platform's internal outsourcing environment; When the upload of the first image file is detected or an export task for the first image file is triggered, the Nydus toolchain is invoked to pull the first image file from the target platform. The first image file pulled is converted into a Nydus format image using the Nydus toolchain to obtain the second image file.

7. The image deployment method based on image acceleration technology according to claim 6, characterized in that, The second image file is compatible with the OCI specification, and the image deployment method based on image acceleration technology further includes: The second image file is stored in the OCI-compatible backend storage of the target platform.

8. A mirror deployment system based on image acceleration technology, used to execute the image deployment method based on image acceleration technology as described in any one of claims 1-7, characterized in that, include: The format conversion module is configured to convert the format of the first image file in the target platform to obtain the second image file. The first image file is in OCI format, and the second image file is in Nydus format; The image deduplication module is configured to: compare the second image file with the image files already stored in the image pool, deduplicate the second image file, and store the deduplicated second image file into the image pool; The image export module is configured to: in response to an export task, package the data corresponding to the target image file in the image pool into a deployment data package, wherein the target image file is an image file that needs to be deployed in a container in a customer environment; The image deployment module is configured to deploy the deployment package in the container based on a snapshotter to deploy the target image file within the container, wherein the snapshotter is used to understand and run the Nydus format image file.

Images