Edge environment container accelerated supply method, system, electronic device and storage medium

Abstract

The present invention provides an edge environment container accelerated supply method, system, electronic device and storage medium. The supply method includes: S1, Docker Daemon generates a download request; S2, DownloadProxy intercepts the download request; S3, queries whether the file requested for download is the Manifests of the image; if so, forwards the request directly to the cloud image warehouse; if not, sends a request to the central node to query the location of the image layer file based on the SHA256 value of the request, accepts the returned location and rebuilds the download request; S4, queries whether the image layer file is cached on the edge node; if not, forwards the request directly to the cloud image warehouse; if so, forwards the download request to the corresponding edge node; S5, forwards the requested download result to Docker Daemon. This solution integrates the storage resources of all edge nodes in a distributed cache manner, thereby expanding the scope of cache content and further speeding up the image download speed.

Description

Technical Field

[0001] The present invention belongs to the field of edge computing technology, and specifically relates to an edge environment container accelerated supply method, system, electronic device and storage medium. Background Art

[0002] As a new computing model, edge computing has attracted attention from both academia and industry since its introduction. For edge computing, the resource-constrained nature of edge nodes allows for an organic combination of container technology and edge computing. Furthermore, the high cohesion and low coupling characteristics of containers are well-suited to edge environments, making containerization the de facto standard for edge computing platforms. Combining edge computing with container technology offers the following advantages: 1) Container technology enables rapid deployment or termination of services in edge environments; 2) Container technology provides edge devices with convenient service migration and service discovery methods for managing services within edge devices; 3) Container technology can improve the fault tolerance of services and enhance the availability and reliability of edge nodes; 4) The edge can adopt a caching strategy for container images and required data, thereby improving the overall performance of edge services.

[0003] However, while introducing container technology to provide edge computing solutions for deploying edge services also creates new challenges. Containers require image files, which can range in size from tens of megabytes to several gigabytes. Due to limited network bandwidth at edge nodes, image downloads can incur significant latency, impacting the quality of service (QoS).

[0004] In order to improve the download speed of service images, there are many solutions to increase the speed of container provisioning. The solutions currently proposed require customized images, or redesign and implement the download mechanism of the application container engine (Docker) at the source code level. For example, a mechanism called FogDocker increases the download speed of images by building the basic files in the image layer into a special base layer and modifying Docker's container deployment process. However, since it makes customized modifications to each image, this is difficult to implement in edge environments that are closer to the public cloud. Another example is DockerPI, another container deployment acceleration solution. Its main idea is to use multi-threading technology to accelerate the decompression process of Docker images after downloading, and to parallelize the serial process of downloading, decompressing, and storing the image acquisition process, thereby accelerating container deployment. However, this solution makes source code-level modifications to Docker, which destroys the integrity of Docker.

[0005] These non-universal acceleration strategies are difficult to popularize in practical applications. Furthermore, traditional distributed storage strategies primarily target single files like videos and data, failing to consider the critical reuse of image layers. Therefore, accelerating image download speeds at the edge without changing the original Docker and image organization structure is a critical issue that needs to be addressed. Summary of the Invention

[0006] The present invention provides an edge environment container accelerated supply method, system, electronic device and storage medium, so as to accelerate the download speed of the image at the edge without changing the original organizational structure of Docker and the image.

[0007] To solve the above technical problems, the first aspect of the present invention provides a method for accelerating the supply of edge environment containers, including: S1, applying the native daemon of the container engine (English name is DockerDaemon) to generate a download request; S2, a download proxy (English name is DownloadProxy) intercepts the download request; S3, queries whether the file requested for download is a description list of the image (English name is Manifests); if so, forwards the request directly to the cloud image repository; if not, sends a request to the central node to query the location of the image layer file based on the SHA256 value of the request, accepts the returned location and rebuilds the download request; S4, queries whether the image layer file is cached on the edge node; if not, forwards the request directly to the cloud image repository; if so, forwards the download request to the corresponding edge node; S5, forwards the download result of the request to DockerDaemon.

[0008] In some exemplary embodiments, in the step S1, it specifically includes: S101, after receiving the request to build a container, Docker Daemon will search its own cache to find out whether the corresponding image file of the container exists locally; if so, Docker Daemon directly obtains the Manifests form of the image; if not, Docker Daemon generates a GET request to download the Manifests of the corresponding image, and sends it to the cloud image repository to obtain the Manifests form of the image; S102, after receiving the returned Manifests form, Docker Daemon deserializes the Manifests form and polls the fslayers field in the Manifests form; Docker Daemon uses the fslayer field as the SHA256 value of the unique identifier of the image layer, and searches for a reusable layer file in the local image layer file; if so, Docker Daemon forwards the request directly to the cloud image repository; if not, Docker Daemon generates a download request for the layer file that does not exist locally, and sends the request to the cloud image repository.

[0009] In some exemplary embodiments, in the step S2, specifically including: DownloadProxy intercepts the download request, generates a corresponding query cache location request based on the SHA256 value carried in the download request, and forwards the query cache location request to the task processing module of the central node.

[0010] In some exemplary embodiments, in the step S3, it specifically includes: S301, the task processing module of the central node receives the query cache location request and forwards it to the open source lightweight distributed key-value storage database (referred to as ETCD database) to find out whether the corresponding location information of the image layer exists; the ETCD database returns the corresponding result to the task processing module of the central node; wherein, the ETCD database stores the IDs of all edge devices that cache the image layer in the form of key-value pairs, and the cache location of the image layer is stored in the / location / path of the ETCD database; S302, the task processing module of the central node takes the result and returns it to DownloadProxy; if the corresponding location information of the image layer is found in the ETCD database, the Key-Value is returned to DownloadProxy; if the corresponding location information of the image layer is not found in the ETCD database, an empty value is returned to DownloadProxy.

[0011] In some exemplary embodiments, in step S4, it specifically includes: when the DownloadProxy of the edge node receives a response message, if the response message is an empty value, DownloadProxy forwards the download request directly to the cloud image repository and delivers the obtained image layer file to DockerProxy; if the response message is not empty, DownloadProxy constructs a new download request and sends it to the corresponding edge node, thereby obtaining the image layer tar.gz compressed file cached in the edge cluster.

[0012] In some exemplary embodiments, in step S5, it specifically includes: when Docker Daemon receives the tar.gz compressed file of the corresponding request layer, it uses Docker's original decompression method and file system to decompress it and mount it to the same mount point, finally forming a complete image.

[0013] The second aspect of the present invention provides an edge environment container acceleration supply system, which includes: a cloud storage layer, which is used as a centralized image warehouse for the entire cluster and stores all images in the edge system; a redirection layer, which is deployed at the central node end and is used to store the correspondence between edge devices in the edge cluster and their cache layers, and to centrally manage the placement of cache image layers; a local cache layer, which is deployed at the cache node end and is used to store actual cached files and intercept image download requests when Docker downloads images, and redirect Docker's image download requests based on the image layer location information forwarded by the redirection layer.

[0014] In some exemplary embodiments, the redirection layer includes ETCD, a cache placement module and a download task processing module; the ETCD is deployed to the central node and is used as an open source lightweight distributed key-value storage database to store information about the storage location of the image layer; the cache placement module has a built-in cache placement algorithm for calculating the cache location of the image layer, and calculating the cache file download cost of the entire edge cluster based on node requirements and network conditions between nodes, and minimizing the cost through distributed solution; the download task processing module is communicated with the ETCD and is used to receive download requests forwarded by edge nodes, and read the corresponding file cache location information from ETCD based on the request, and return it to the requesting node; the local cache layer includes DockerDaemons on, DownloadProxy and local warehouse (English name is LocalRegistry); the DockerDaemon is the native daemon of Docker, which is used to receive requests sent by users, download the corresponding images, and build corresponding containers based on the images; the DownloadProxy is deployed on the edge node in the form of a container, which is used to intercept the image download request of DockerDaemon, request the corresponding image layer storage location from the download task processing module according to the request, and redirect the download request to the cloud image warehouse or edge cache node according to the returned result; the LocalRegistry is a small image warehouse used to store the allocated edge layer data on the edge device.

[0015] The third aspect of the present invention provides an electronic device, characterized in that it includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to execute the edge environment container accelerated supply method as described above.

[0016] A fourth aspect of the present invention provides a computer-readable storage medium storing a computer program, characterized in that when the computer program is executed by a processor, the above-mentioned edge environment container accelerated supply method is implemented.

[0017] The technical solution provided by the present invention has at least the following advantages:

[0018] The present invention provides an edge environment container acceleration provisioning method, system, electronic device, and storage medium. This method, based on the native Docker image download mechanism, designs and implements a layer-based image caching mechanism while maintaining the integrity of Docker and the image. It also redirects download requests to the corresponding edge cache node, accelerating image download speeds and thus accelerating container deployment. Furthermore, compared to traditional single-node caching, this solution integrates the storage resources of all edge nodes in a distributed cache, thereby expanding the scope of cache content and further accelerating image download speeds. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] One or more embodiments are exemplarily described by the pictures in the corresponding drawings. These exemplifications do not constitute limitations on the embodiments. Unless otherwise stated, the pictures in the drawings do not constitute proportional limitations.

[0020] Figure 1 This is the architecture diagram of the image accelerated download scheme based on distributed cache (Image Accelerated Download Scheme based on Distributed Cache, abbreviated as IADSDC) in the present invention;

[0021] Figure 2 This is a flowchart of the processing of image download requests by IADSDC in the present invention;

[0022] Figure 3 Schematic diagram of the response flow of IADSDC processing image download request in the present invention;

[0023] Figure 4 The figure is a schematic structural diagram of an electronic device provided by the present invention. DETAILED DESCRIPTION

[0024] As can be seen from the background technology, there are problems in the existing technology: edge nodes are limited by network bandwidth and cannot quickly download the required images from the cloud image repository; and the storage capacity of edge nodes is limited, and only a very small part of the image files can be cached locally on the node.

[0025] In order to overcome the problem of slow image download speed mentioned in the background technology and increase the universality of the solution, the inventors have designed an image acceleration download solution IADSDC based on distributed caching based on the original Docker image download process after research. This solution adopts a centralized design, dividing the edge nodes into two roles: central nodes and cache nodes. The cache nodes are responsible for storing images at the layer granularity, and the central nodes are used to manage cache metadata. In addition, the solution deploys all components in a containerized manner on the edge nodes without making code-level changes to Docker, truly achieving high cohesion and low coupling, and enabling plug-and-play. It solves the problems of accelerating the download speed of container images in edge computing and placing image caches in edge clusters with limited storage space.

[0026] The following detailed description of the various embodiments of the present application is provided in conjunction with the accompanying drawings. However, those skilled in the art will appreciate that many technical details are provided in the various embodiments of the present application to facilitate a better understanding of the present application. However, even without these technical details and the various variations and modifications based on the following embodiments, the technical solutions claimed in the present application can still be implemented.

[0027] like Figure 1 、 Figure 3 As shown, the first aspect of the present invention provides a method for accelerating the supply of an edge environment container, comprising:

[0028] S1. Docker Daemon generates a download request.

[0029] S2. DownloadProxy intercepts the download request.

[0030] S3. Query whether the file requested for download is a mirrored Manifests file.

[0031] If yes, the request is forwarded directly to the cloud image repository;

[0032] If not, a request is sent to the central node to query the location of the layer based on the SHA256 value of the request, and the returned location is accepted and the download request is rebuilt.

[0033] S4. Check whether the layer is cached on the edge node; if not, forward the request directly to the cloud image repository; if so, forward the download request to the corresponding edge node; S5. Forward the download result of the request to Docker Daemon.

[0034] In some exemplary embodiments, step S1 specifically includes:

[0035] S101. After receiving the container building request, Docker Daemon searches its own cache to find out whether the image file corresponding to the container exists locally.

[0036] If so, Docker Daemon directly obtains the Manifests form of the image;

[0037] If not, Docker Daemon generates a GET request to download the Manifests of the corresponding image, and sends it to the cloud image repository to obtain the Manifests form of the image.

[0038] S102. After receiving the returned Manifests form, Docker Daemon deserializes the Manifests form and polls the fslayers field in the Manifests form; Docker Daemon uses the fslayer field as the SHA256 value of the image layer's unique identifier to search for a reusable layer file in the local image layer file;

[0039] If so, Docker Daemon forwards the request directly to the cloud image repository;

[0040] If not, Docker Daemon generates a download request for the layer file that does not exist locally and sends the request to the cloud image repository.

[0041] In some exemplary embodiments, step S2 specifically includes:

[0042] DownloadProxy intercepts the download request, generates a corresponding cache location query request based on the SHA256 value carried in the download request, and forwards the cache location query request to the task processing module of the central node.

[0043] In some exemplary embodiments, step S3 specifically includes:

[0044] S301. The task processing module of the central node receives the query cache location request and forwards it to the ETCD database to find out whether there is corresponding location information of the image layer; the ETCD database returns the corresponding result to the task processing module of the central node; wherein, the ETCD database stores the IDs of all edge devices that cache the image layer in the form of key-value pairs, and the cache location of the image layer is stored in the / location / path of the ETCD database.

[0045] S302: The task processing module of the central node obtains the result and returns it to DownloadProxy;

[0046] If the corresponding location information of the image layer is found in the ETCD database, the Key-Value is returned to DownloadProxy;

[0047] If the corresponding location information of the image layer is not found in the ETCD database, a null value is returned to DownloadProxy.

[0048] In some exemplary embodiments, step S4 specifically includes:

[0049] When the DownloadProxy of the edge node receives a response message, if the response message is empty, DownloadProxy forwards the download request directly to the cloud image repository and delivers the obtained image layer file to DockerProxy; if the response message is not empty, DownloadProxy constructs a new download request and sends it to the corresponding edge node to obtain the image layer tar.gz compressed file cached in the edge cluster.

[0050] In some exemplary embodiments, step S5 specifically includes:

[0051] When Docker Daemon receives the tar.gz compressed file of the corresponding request layer, it uses Docker's original decompression method and file system to decompress it and mount it to the same mount point, eventually forming a complete image.

[0052] A second aspect of the present invention provides an edge environment container accelerated supply system, the system comprising:

[0053] The cloud storage layer serves as a centralized image repository for the entire cluster and stores all images in the edge system. This image repository uses the same API as DockerHub, so users won't notice any difference in the image download process compared to downloading from DockerHub. Therefore, in a cloud-edge environment, Harbor is used to establish a private image repository in the cloud to store images for the entire edge cluster.

[0054] The redirection layer is deployed at the central node end and is used to store the correspondence between edge devices and their cache layers in the edge cluster, and to centrally manage the placement of the cache mirror layer.

[0055] The local cache layer is deployed on the cache node end and is used to store the actual cached files and intercept the image download request when Docker downloads the image, and redirect the Docker image download request based on the image layer location information forwarded by the redirection layer.

[0056] In some exemplary embodiments, see Figure 2 The redirection layer includes ETCD, a cache placement module and a download task processing module.

[0057] The ETCD is deployed to the central node and is used as an open source lightweight distributed key-value storage database to store information about the storage location of the image layer.

[0058] The cache placement module has a built-in cache placement algorithm for calculating the cache location of the image layer, and calculates the cache file download cost of the entire edge cluster based on node requirements and network conditions between nodes, and minimizes the cost through distributed solution.

[0059] The download task processing module is in communication with the ETCD, and is configured to receive the download request forwarded by the edge node, and read the corresponding file cache location information from the ETCD according to the request, and return it to the requesting node.

[0060] Continue to see Figure 2 The local cache layer includes DockerDaemon, DownloadProxy and LocalRegistry.

[0061] The Docker Daemon is Docker's native daemon process, responsible for receiving user requests, downloading the corresponding image, and building the corresponding container based on the image. Specifically, the Daemon's main functions include image management, image building, REST API, authentication, security, core networking, and orchestration.

[0062] The DownloadProxy is deployed on the edge node in the form of a container, and is used to intercept the image download request of DockerDaemon, request the corresponding image layer storage location from the download task processing module according to the request, and redirect the download request to the cloud image warehouse or edge cache node according to the returned result.

[0063] The LocalRegistry acts as a small image repository, storing allocated edge-tier data on edge devices. Because the LocalRegistry uses the Registry:v2 image officially released by Docker, its internal core routing mechanism is the same as DockerHub. Therefore, this component uses the same API interface as DockerHub, eliminating the need to redesign download requests.

[0064] refer to Figure 4Another embodiment of the present application provides an electronic device, comprising: at least one processor 110; and a memory 111 communicatively connected to the at least one processor; wherein the memory 111 stores instructions that can be executed by the at least one processor 110, and the instructions are executed by the at least one processor 110 so that the at least one processor 110 can execute any of the above method embodiments.

[0065] The memory 111 and the processor 110 are connected using a bus. The bus may include any number of interconnected buses and bridges. The bus connects various circuits of one or more processors 110 and the memory 111. The bus may also connect various other circuits such as peripheral devices, voltage regulators, and power management circuits. These are all well known in the art and are therefore not described further herein. The bus interface provides an interface between the bus and the transceiver. The transceiver may be a single component or multiple components, such as multiple receivers and transmitters, providing a unit for communicating with various other devices over a transmission medium. Data processed by the processor 110 is transmitted over a wireless medium via an antenna. Furthermore, the antenna receives data and transmits the data to the processor 110.

[0066] The processor 110 is responsible for managing the bus and general processing, and may also provide various functions, including timing, peripheral interfaces, voltage regulation, power management, and other control functions. The memory 111 may be used to store data used by the processor 110 when performing operations.

[0067] Another embodiment of the present application relates to a computer-readable storage medium storing a computer program, which implements the above method embodiment when executed by a processor.

[0068] That is, those skilled in the art will understand that all or part of the steps in the above-mentioned embodiments can be implemented by instructing the relevant hardware through a program, which is stored in a storage medium and includes a number of instructions for causing a device (which may be a single-chip microcomputer, chip, etc.) or a processor to execute all or part of the steps in the above-mentioned methods of each embodiment of the present application. The aforementioned storage medium includes: a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk, etc., various media that can store program code.

[0069] Those skilled in the art will appreciate that the above-described embodiments are specific examples for implementing the present application, and that in actual applications, various changes in form and detail may be made thereto without departing from the spirit and scope of the present application. Any person skilled in the art may make changes and modifications without departing from the spirit and scope of the present application. Therefore, the scope of protection of the present application shall be subject to the scope defined in the claims.

Claims

1. A method for accelerating the supply of edge environment containers, characterized in that: include: S1. The native daemon process of the application container engine generates a download request. S2. The download agent intercepts the download request; S3. Query whether the file requested for download is a description list of an image; If yes, the download request is forwarded directly to the cloud image repository; if not, a request is sent to the central node based on the SHA256 value of the request to query the location of the image layer file, the returned location is accepted and the download request is rebuilt; S4. Check whether the image layer file is cached on the edge node; if not, forward the download request directly to the cloud image repository; if so, forward the download request to the corresponding edge node; S5. Forward the requested download result to the native daemon process of the application container engine; In the step S1, it specifically includes: S101: After receiving a request to build a container, the native daemon process of the application container engine searches its own cache to find out whether the image file corresponding to the container exists locally. If so, the application container engine native daemon directly obtains the image description list form; If not, the application container engine native daemon generates a GET request to download the description list of the corresponding image, and sends it to the cloud image repository to obtain the description list form of the image; S102: After receiving the returned description list form, the native daemon process of the application container engine deserializes the description list form and polls the fslayers field in the description list form; The native daemon process of the application container engine uses the fslayers field as the SHA256 value of the unique identifier of the image layer to search for a reusable layer file in the local image layer file; If so, the application container engine native daemon forwards the download request directly to the cloud image repository; If not, the application container engine native daemon generates a download request for the layer file that does not exist locally and sends the download request to the cloud image repository.

2. The method for accelerating supply of edge environment containers according to claim 1, characterized in that: In the step S2, it specifically includes: The download agent intercepts the download request, generates a corresponding cache location query request according to the SHA256 value carried in the download request, and forwards the cache location query request to the task processing module of the central node.

3. The method for accelerating supply of edge environment containers according to claim 2, characterized in that: In the step S3, it specifically includes: S301: The task processing module of the central node receives the cache location query request and forwards it to the open source lightweight distributed key-value storage database to check whether the corresponding location information of the image layer exists; The open source lightweight distributed key-value storage database returns the corresponding results to the task processing module of the central node; The open-source lightweight distributed key-value storage database stores the IDs of all edge devices that cache the image layer in the form of key-value pairs. The cache location of the image layer is stored in the / location / path of the open-source lightweight distributed key-value storage database. S302: The task processing module of the central node obtains the result and returns it to the download agent; If the corresponding location information of the image layer is found in the open source lightweight distributed key-value storage database, the database storing data in key-value pairs is returned to the download agent; If the corresponding location information of the image layer is not found in the open source lightweight distributed key-value storage database, a null value is returned to the download agent.

4. The method for accelerating supply of edge environment containers according to claim 3, characterized in that: In the step S4, it specifically includes: When the download agent of the edge node receives the response message, if the response message is null, the download agent forwards the download request directly to the cloud image repository and delivers the obtained image layer file to the Docker download agent; If the response message is not empty, the download agent constructs a new download request and sends it to the corresponding edge node, thereby obtaining the image layer tar.gz compressed file cached in the edge cluster.

5. The method for accelerating supply of edge environment containers according to claim 4, characterized in that: In the step S5, it specifically includes: When the native daemon process of the application container engine receives the tar.gz compressed file of the corresponding request layer, it uses the original decompression method and file system of the application container engine to decompress it and mount it to the same mount point, finally forming a complete image.

6. An edge environment container accelerated supply system, the system is used to implement the edge environment container accelerated supply method according to any one of claims 1 to 5, characterized in that: The system includes: The cloud storage layer is used as a centralized image repository for the entire cluster and stores all images in the edge system; The redirection layer is deployed on the central node end and is used to store the correspondence between edge devices in the edge cluster and their cache layers, and to centrally manage the placement of the cache image layer; The local cache layer is deployed on the cache node side and is used to store the actual cached files and intercept the image download request when the application container engine downloads the image, and redirect the image download request of the application container engine based on the image layer location information forwarded by the redirection layer.

7. The edge environment container accelerated supply system according to claim 6, characterized in that: The redirection layer includes an open source lightweight distributed key-value storage database, a cache placement module, and a download task processing module; The open source lightweight distributed key-value storage database is deployed to the central node to store information about the image layer storage location; The cache placement module has a built-in cache placement algorithm for calculating the cache location of the image layer, and calculates the cache file download cost of the entire edge cluster based on node requirements and network conditions between nodes, and minimizes the cost through distributed solution; The download task processing module is in communication with the open source lightweight distributed key-value storage database, and is used to receive the download request forwarded by the edge node, and read the corresponding file cache location information from the open source lightweight distributed key-value storage database according to the request, and return it to the requesting node; The local cache layer includes the application container engine native daemon, download agent and local warehouse; The native daemon process of the application container engine is used to receive requests sent by users, download the corresponding image, and build the corresponding container based on the image; The download agent is deployed on the edge node in the form of a container, and is used to intercept the image download request of the native daemon process of the application container engine, request the corresponding image layer storage location from the download task processing module according to the request, and redirect the download request to the cloud image warehouse or edge cache node according to the returned result; The local warehouse serves as a small mirror warehouse and is used to store the allocated edge layer data on the edge device.

8. An electronic device, characterized in that: include: at least one processor; as well as, A memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to execute the edge environment container accelerated supply method as described in any one of claims 1 to 6.

9. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the edge environment container accelerated supply method described in any one of claims 1 to 6 is implemented.

Images