Mirror image file downloading system and method, storage medium and electronic equipment

Abstract

The embodiment of the invention provides a mirror image file downloading system and method, a storage medium and electronic equipment, and the system comprises a peer node which is used for intercepting a mirror image downloading request initiated by a container engine in a first cluster node and forwarding the mirror image downloading request to a scheduler, the mirror image downloading request is used for requesting to download a first mirror image file from a mirror image warehouse; the scheduler is connected with the peer-to-peer node and used for matching a second cluster node according to a first file identifier of the first mirror image file, generating a first downloading request according to the first file identifier and the second cluster node and sending the first downloading request to the second cluster node, the first downloading request is used for requesting to download a first mirror image file from a second cluster node, and the target cluster comprises the second cluster node; and the peer node is also used for downloading the first mirror image file from the second cluster node according to the first scheduling strategy instruction under the condition that the first scheduling strategy instruction of the second cluster node is received.

Description

Technical Field

[0001] The embodiments of the present application relate to the computer field, and in particular, to a system and method for downloading an image file, a storage medium, and an electronic device. Background Art

[0002] Cloud computing has great potential in the future and will play an important role as we enter a new technological era. Kubernetes is a portable, scalable, open source container orchestration platform for managing containerized workloads and services, facilitating declarative configuration and automation, and gradually becoming the de facto standard for cloud computing.

[0003] When a containerized application is deployed to a Kubernetes cluster, the business must go through the following steps from creation to formal external service provision: Figure 2 The process shown specifically includes: creation, scheduling, cni allocation, starting sidecar, pulling images, and starting services. Among them, after the k8s control plane schedules the Pod to the working node of the cluster, the node agent (Kubelet) running on the working node coordinates with the container runtime (such as docker, containerd, etc.) installed in the node, and pulls the necessary container images from the image repository (such as dockerhub, harbor, etc.). Usually, it takes time to pull the image from the registry to the node, and the resulting delay may become a bottleneck for the rapid startup and rapid expansion of containerized applications in the entire process.

[0004] In the related technology, there is no effective solution to the technical problem of low efficiency in downloading large-scale container image files in a Kubernetes cluster environment. Summary of the invention

[0005] The embodiments of the present application provide a system and method for downloading an image file, a storage medium, and an electronic device to at least solve the problem of low efficiency in downloading large-scale container image files in a Kubernetes cluster environment in the related art.

[0006] According to an embodiment of the present application, a system for downloading an image file is provided, which is arranged in a first cluster node in a target cluster, and includes: a peer node, which is used to intercept an image download request initiated by a container engine in the first cluster node, and forward the image download request to a scheduler, wherein the image download request is used to request to download a first image file from an image repository; the scheduler, which is connected to the peer node, is used to match a second cluster node according to a first file identifier of the first image file, and generates a first download request according to the first file identifier and the second cluster node, and sends the first download request to the second cluster node, wherein the first download request is used to request to download the first image file from the second cluster node, and the target cluster includes the second cluster node; the peer node is also used to download the first image file from the second cluster node according to the first scheduling policy instruction when receiving the first scheduling policy instruction of the second cluster node.

[0007] In an exemplary embodiment, the peer node includes: a hijacker, which is used to intercept the image download request upon receiving the image download request and redirect the image download request to a controller; the controller is connected to the hijacker, and is used to forward the image download request to the scheduler to instruct the scheduler to generate the first download request, and upon receiving the first scheduling policy instruction, generates a download instruction according to the first scheduling policy instruction and sends the download instruction to the downloader; the downloader is connected to the controller, and is used to execute the download instruction to download the first image file from the second cluster node.

[0008] In an exemplary embodiment, the scheduler is further used to: upon receiving a second download request sent by a third cluster node, query the seed cache for a second image file according to a second file identifier carried in the second download request, wherein the second download request is used by the third cluster node to request the third cluster node to download the second image file from the first cluster node, the target cluster includes the third cluster node, the first cluster node includes the seed cache, and the seed cache is used to cache multiple seed block files contained in the image file; and upon querying the second image file, requesting the image repository for the last modification time of the second image file according to the second file identifier, and performing a search on the second image file according to the last modification time. when it is determined that the second image file passes the expiration check, performing an integrity check on the second image file according to the first metadata of the second image file; when it is determined that the second image file does not pass the integrity check, generating a third download request according to the second file identifier, and sending the third download request to the seed cache to instruct the seed cache to download the missing files of the second image file from the image repository, and re-perform an integrity check on the second image file; when it is determined that the second image file passes the integrity check, sending a second scheduling policy instruction to a peer node of the third cluster node to instruct the third cluster node to download the second image file.

[0009] In an exemplary embodiment, the scheduler is further used to: generate a fourth download request based on the second file identifier when the second image file is not found; and send the fourth download request to the seed cache to instruct the seed cache to download the second image file from the image repository.

[0010] In an exemplary embodiment, the scheduler is further used to: when it is determined that the second image file has not passed the expiration check, delete the second image file and generate a fifth download request based on the second file identifier; send the fifth download request to the seed cache to instruct the seed cache to re-download the second image file from the image repository.

[0011] In an exemplary embodiment, the downloader is further used to: obtain second meta information of the first image file from the image repository according to the first file identifier, wherein the second meta information is used to indicate hash values ​​of multiple image layer files contained in the first image file, and the hash value is used to identify the image layer file; when multiple seed block files contained in the first image file are downloaded from the second cluster node, perform integrity verification on the first image file according to the multiple hash values, wherein each image layer file contains multiple seed block files; when the first image file passes the integrity verification, determine that the download of the first image file is complete, and combine the multiple seed block files into the first image file.

[0012] In an exemplary embodiment, the scheduler is further used to: match multiple fourth cluster nodes in the target cluster according to the first file identifier, wherein the multiple fourth cluster nodes all cache the first image file and the multiple fourth cluster nodes include the second cluster node; determine the second cluster node among the multiple fourth cluster nodes according to the network topology connection relationship between the multiple fourth cluster nodes and the first cluster node and the node performance of the multiple fourth cluster nodes, wherein the network topology connection relationship is used to indicate the network connection status and geographical distance between the multiple fourth cluster nodes and the first cluster node.

[0013] According to another embodiment of the present application, a method for downloading an image file is provided, which is applied to the above-mentioned image file downloading system, including: intercepting an image download request initiated by a container engine in the first cluster node, and forwarding the image download request to a scheduler; matching a second cluster node according to a first file identifier of the first image file, and generating a first download request according to the first file identifier and the second cluster node, and sending the first download request to the second cluster node; and downloading the first image file from the second cluster node according to the first scheduling policy instruction when a first scheduling policy instruction of the second cluster node is received.

[0014] According to another embodiment of the present application, a computer-readable storage medium is provided, in which a computer program is stored, wherein the computer program is configured to execute the steps of any of the above method embodiments when run.

[0015] According to another embodiment of the present application, an electronic device is provided, including a memory and a processor, wherein the memory stores a computer program, and the processor is configured to run the computer program to execute the steps in any one of the above method embodiments.

[0016] According to another embodiment of the present application, a computer program product is provided, including a computer program, and when the computer program is executed by a processor, the steps of the method described in each embodiment of the present application are implemented.

[0017] The present application provides a system for downloading an image file, which is arranged in a first cluster node in a target cluster, and includes: a peer node, which is used to intercept an image download request initiated by a container of the first cluster node, and forward the image download request to a scheduler for processing, wherein the image download request is used to request downloading a first image file from an image repository; the scheduler is connected to the peer node, and is used to match a second cluster node in the target cluster according to a first file identifier of the first image file, and generate a first download request according to the first file identifier and the second cluster node, and then send the first download request to the second cluster node, wherein the first download request is used to request downloading the first image file from the second cluster node; the peer node is also used to download the first image file from the second cluster node according to the first scheduling policy instruction after receiving the first scheduling policy instruction of the second cluster node; the above scheme is adopted, and by introducing a P2P network mechanism, the traditional image download architecture is changed, so that nodes of the group can transmit and share image files with each other, and the efficiency of image distribution is greatly improved; thereby solving the problem of low efficiency of large-scale container image file download in a kubernetes cluster environment in the related technology. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] Figure 1 It is a hardware structure block diagram of a computer terminal of a method for downloading an image file according to an embodiment of the present application;

[0019] Figure 2 is a deployment flow chart of a containerized application according to an embodiment of the present application;

[0020] Figure 3 It is a structural block diagram of a system for downloading an image file according to an embodiment of the present application;

[0021] Figure 4 is a structural block diagram of a peer node according to an embodiment of the present application;

[0022] Figure 5 It is a flowchart of a method for downloading an image file according to an embodiment of the present application;

[0023] Figure 6 It is a flowchart of a method for downloading an image file according to an embodiment of the present application. DETAILED DESCRIPTION

[0024] The embodiments of the present application will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0025] It should be noted that the terms "first", "second", etc. in the specification and claims of this application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence.

[0026] The method embodiments provided in the embodiments of the present application can be executed in a computer terminal or a similar computing device. Taking running on a computer terminal as an example, Figure 1 1 is a hardware structure block diagram of a computer terminal for a method for downloading an image file according to an embodiment of the present application. Figure 1 As shown, the computer terminal may include one or more ( Figure 1 Only one is shown in the figure) processor 102 (processor 102 may include but is not limited to a processing device such as a microprocessor MCU or a programmable logic device FPGA) and a memory 104 for storing data, wherein the computer terminal may also include a transmission device 106 and an input / output device 108 for communication functions. It can be understood by those skilled in the art that Figure 1 The structure shown is only for illustration and does not limit the structure of the above-mentioned computer terminal. Figure 1 More or fewer components as shown, or with Figure 1 Different configurations are shown.

[0027] The memory 104 can be used to store computer programs, for example, software programs and modules of application software, such as the computer program corresponding to the method for downloading the image file in the embodiment of the present application. The processor 102 executes various functional applications and data processing by running the computer program stored in the memory 104, that is, to implement the above method. The memory 104 may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some examples, the memory 104 may further include a memory remotely arranged relative to the processor 102, and these remote memories may be connected to the computer terminal via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0028] The transmission device 106 is used to receive or send data via a network. The specific example of the above network may include a wireless network provided by a communication provider of a computer terminal. In one example, the transmission device 106 includes a network adapter (Network Interface Controller, referred to as NIC), which can be connected to other network devices through a base station so as to communicate with the Internet. In one example, the transmission device 106 can be a radio frequency (RF) module, which is used to communicate with the Internet wirelessly.

[0029] The following is an explanation of the professional terms that appear in this application:

[0030] Cluster: A cluster is a group of computers that, as a whole, provide a set of network resources to users. These individual computer systems are nodes of the cluster. Clusters provide the following key features: scalability, high availability, load balancing, and error recovery.

[0031] Kubernetes: Kubernetes is an open source containerized application for managing multiple hosts in a cloud platform, referred to as k8s. The goal of Kubernetes is to make the deployment of containerized applications simple and efficient. Kubernetes provides a mechanism for application deployment, planning, updating, and maintenance. A core feature of Kubernetes is the ability to autonomously manage containers to ensure that containers in the cloud platform run as expected by users.

[0032] Docker: Docker uses container technology to create and independently run one or a group of applications through images. The docker image is like a template that can be used to create container services.

[0033] In actual cloud computing business scenarios, we often encounter the following scenarios:

[0034] 1. A service needs to be started immediately. For example, scheduled services in the financial industry and flash sales on e-commerce platforms require services to be started quickly within a set period of time.

[0035] 2. A service needs to be expanded quickly. For example, testing service stress testing, promotion activities, resource shortages, increased workloads, etc. all require rapid expansion of the number of service instances;

[0036] 3. The network connection between the cluster and the image registry may be affected by insufficient bandwidth, or the node may lose network connection with the local registry image, so the Pod startup will be stuck until the connection is restored. Especially in edge computing, applications must be able to gracefully tolerate all the effects of intermittent network connections;

[0037] 4. With the development of deep learning, model parameters are usually in the hundreds of millions. In image recognition, speech recognition, text recognition and other businesses, they are heavily dependent on GPU environments such as cuda and cudnn, which leads to business image volumes of more than 20G. In addition, with the continuous improvement of the container cloud platform, the number of applications carried is increasing, and the number of running containers can exceed 10,000.

[0038] In actual cloud computing business scenarios, if a containerized application needs to scale out immediately to create additional instances, the K8s control plane will create additional Pod replicas under the action of strategies such as Horizontal Pod Autoscaler (HPA) or Knative Pod Autocaler (KPA). However, these Pods will not be able to handle the increased traffic until the required images are pulled and the containers are started and running. If such applications have strict requirements on the speed of their startup and expansion, they will not be able to achieve their original design purpose. Any excessive delay caused by pulling images from the registry may be unacceptable. In addition, simultaneous requests from different clients to pull images will consume huge network bandwidth, especially public network bandwidth, which will cause network congestion in the image warehouse.

[0039] In the related art, a solution is proposed to load the container image directly on the cluster working node through a pull command such as docker pull during container runtime, so that Kubelet does not need to pull these images, but immediately uses the already pulled images.

[0040] However, when running containers such as Docker, they are deployed on a single machine, and the cloud native ecosystem does not provide friendly operation capabilities for images, lacking a resource, object, and strategy to directly cache images on a large scale on cluster working nodes. Therefore, when the cluster is large, such as containing hundreds or thousands of nodes, executing pull commands on each node to pre-load all required images on the node will consume a lot of manpower and material resources.

[0041] The container runtime on the node (such as docker, containerd, etc.) directly pulls the image from the image registry, which belongs to the cs (client-server) architecture. However, the distribution service system using the traditional C / S (Client-Server) structure has become increasingly unable to adapt to the scenario of large-scale distributed applications during image transmission.

[0042] In order to solve the above problem, in this embodiment, a system for downloading an image file is provided, which is set in a first cluster node in a target cluster. Figure 3 is a structural block diagram of a system for downloading an image file according to an embodiment of the present application, such as Figure 3 As shown, the system includes:

[0043] A peer node 32, configured to intercept an image download request initiated by the container engine in the first cluster node, and forward the image download request to a scheduler, wherein the image download request is used to request downloading of a first image file from an image repository;

[0044] The scheduler 34 is connected to the peer node, and is used to match the second cluster node according to the first file identifier of the first image file, and generate a first download request according to the first file identifier and the second cluster node, and send the first download request to the second cluster node, wherein the first download request is used to request to download the first image file from the second cluster node, and the target cluster includes the second cluster node;

[0045] The peer node 32 is further configured to, upon receiving the first scheduling policy instruction from the second cluster node, download the first image file from the second cluster node according to the first scheduling policy instruction.

[0046] Through the above scheme, a mirror file download system is proposed, which is set in the first cluster node in the target cluster, including: a peer node, which is used to intercept the mirror download request initiated by the container of the first cluster node, and forward the mirror download request to the scheduler for processing, wherein the mirror download request is used to request to download the first mirror file from the mirror warehouse; the scheduler is connected to the peer node, which is used to match the second cluster node in the target cluster according to the first file identifier of the first mirror file, and generate a first download request according to the first file identifier and the second cluster node, and then send the first download request to the second cluster node, wherein the first download request is used to request to download the first mirror file from the second cluster node; the peer node is also used to download the first mirror file from the second cluster node according to the first scheduling strategy instruction after receiving the first scheduling strategy instruction of the second cluster node; adopting the above scheme, by introducing the P2P network mechanism, the traditional mirror download architecture is changed, so that the nodes of the group can transmit and share mirror files with each other, which greatly improves the efficiency of mirror distribution; thereby solving the problem of low efficiency of large-scale container mirror file download in the kubernetes cluster environment in the related technology.

[0047] It should be noted that P2P (Peer-To-Peer) is a distributed and decentralized peer-to-peer network technology that distributes network resources and services on various nodes. Information transmission and service implementation are directly carried out between nodes, avoiding the single-point bottleneck that may occur in the traditional C / S structure.

[0048] In an exemplary embodiment, the structure of the peer node 32 is as follows Figure 4 As shown, the peer node 32 includes:

[0049] The hijacker 42 is used to intercept the image download request when receiving the image download request, and redirect the image download request to the controller;

[0050] The controller 44 is connected to the hijacker, and is used to forward the image download request to the scheduler to instruct the scheduler to generate the first download request, and when receiving the first scheduling policy instruction, generate a download instruction according to the first scheduling policy instruction, and send the download instruction to the downloader;

[0051] The downloader 46 is connected to the controller, and is used to execute the download instruction to download the first image file from the second cluster node.

[0052] Under normal circumstances, the container runtime installed in the node (such as docker, containerd, etc.) or the node agent (kubelet) coordinated to run in the working node will pull the container image from the image repository (such as dockerhub, harbor, etc.). Hijacker (hijacker) as a resident agent will intercept and process those requests related to image layering. For example, hijacker will intercept all HTTP requests directly initiated by the container engine pull command, intercept image pull requests initiated by the container engine hosting when kubernetes is deployed for service, etc. The specific interception method is to modify the daemon.json file of the container runtime and add the first priority registry-mirror value as the listening address of hijacker. It should be noted that container images are different from ordinary files. They are composed of multiple layers of storage, and the download is also a layered download, not a single file. In addition to the layer information, the container runtime sends a request to pull the image meta information (manifest json, recording the sha256 value (i.e. the above hash value) of each layer of the image. Hijacker will only filter out the request to obtain the layer information (i.e. the above image download request) from it, and hijack and forward it.

[0053] After Hijacker intercepts the image download layer information request (i.e., image download request), it redirects the request to the controller and converts it into a download request to the controller.

[0054] After the controller receives the download request for the image layer information, it continues to forward the request to the scheduler component. After receiving the request, the scheduler component will create a download task based on the unique sha256 value in the download request, use the unique value as the identifier of the download task, and register the node that initiated the request under the task. If the task already exists, only the node that initiated the request will be registered under the task.

[0055] Through this embodiment, through the collaborative work of the hijacker, controller and downloader, an intelligent, efficient and secure image file download system is formed. It not only significantly improves the distribution efficiency of image files in the Kubernetes cluster, reduces network latency and costs, but also enhances the stability and security of the system, which is particularly important for large-scale distributed systems.

[0056] It should be noted that the peer node peer also includes another subcomponent uploader, which is a non-resident process on the node and is responsible for processing all image layer seed block file download requests sent by the downloader on the peer node. That is, when receiving a download request sent by the downloader on other cluster nodes, the uploader responds and processes the download request and cooperates with it to complete the transmission of the image file.

[0057] Furthermore, the scheduler 34 is also used for: upon receiving a second download request sent by a third cluster node, querying the second image file in the seed cache according to the second file identifier carried in the second download request, wherein the second download request is used by the third cluster node to request the third cluster node to download the second image file from the first cluster node, the target cluster includes the third cluster node, the first cluster node includes the seed cache, and the seed cache is used to cache multiple seed block files contained in the image file; and upon querying the second image file, requesting the last modification time of the second image file from the image repository according to the second file identifier, and performing modification on the second image file according to the last modification time. performing an expiration check; when it is determined that the second image file passes the expiration check, performing an integrity check on the second image file according to the first metadata of the second image file; when it is determined that the second image file does not pass the integrity check, generating a third download request according to the second file identifier, and sending the third download request to the seed cache to instruct the seed cache to download the missing files of the second image file from the image repository, and re-performing an integrity check on the second image file; when it is determined that the second image file passes the integrity check, sending a second scheduling policy instruction to a peer node of the third cluster node to instruct the third cluster node to download the second image file.

[0058] When the first cluster node receives the second download request sent by the third cluster node, it will first check whether the second image file exists in the seed cache according to the second file identifier; if it exists in the seed cache, it will directly connect to the image warehouse for expiration check, and send the request with the last modification time of the file returned by the image warehouse, and perform expiration check on the second image file according to the last modification time. If the expiration check passes, it means that the second image file in the cache is valid. Then, it will determine whether the file is complete according to the first metadata of the second image file. If it is complete, it means a complete hit, and it will be directly scheduled, and the second scheduling strategy instruction will be issued to instruct the third cluster node where to download the second image file; if it is incomplete, the remaining files will continue to be downloaded in segments according to the breakpoint resumption method.

[0059] Optionally, the missing files of the second image file may be downloaded directly from the image repository, or the down loader on the node (ie, the downloader described above) may be instructed to download the missing files from other cluster nodes.

[0060] Through this embodiment, a series of intelligent strategies and technical means are used to significantly improve the distribution efficiency and security of container images in the Kubernetes cluster, optimize the utilization of network resources, enhance the elasticity and reliability of the system, and reduce operating costs, which greatly promotes the rapid deployment and response of large-scale container applications.

[0061] Furthermore, the scheduler 34 is also used to: generate a fourth download request according to the second file identifier when the second image file is not found; and send the fourth download request to the seed cache to instruct the seed cache to download the second image file from the image repository.

[0062] If the image layer cache does not exist, the seeder & cacher (i.e., seed cache) is notified to connect to the image repository for the first download of the image layer file.

[0063] Optionally, on the one hand, the first cluster node may re-download the second image file, and on the other hand, the third cluster node may be notified to re-match a cluster node (a cluster node other than the first cluster node) for downloading the second image file.

[0064] Optionally, the first cluster node may re-download the second image file directly from the image repository, or may search for other cluster nodes that cache the second image file as a download source in the same manner as the third cluster node, and this application does not impose any restrictions on this.

[0065] Through this embodiment, through the cooperation of the scheduler and the seed cache, the image file management and distribution process in a large-scale cluster environment is optimized, resource utilization efficiency is improved, network costs are reduced, and at the same time, the startup and expansion of services are accelerated, and the performance of the entire system in terms of image management, network traffic control, service reliability and consistency is enhanced.

[0066] Optionally, the scheduler 34 is also used to: when it is determined that the second image file has not passed the expiration check, delete the second image file and generate a fifth download request based on the second file identifier; send the fifth download request to the seed cache to instruct the seed cache to re-download the second image file from the image repository.

[0067] If the second image file does not pass the expiration check, the original cache file is deleted, and the seeder & cacher is notified to connect to the image repository to download the second image file again.

[0068] Through this embodiment, through the intelligent management of the scheduler, effective updating and optimization of image files are achieved, which not only improves the consistency and security of image use in the cluster, but also optimizes the utilization of network and storage resources, enhances the continuity, flexibility and overall performance of services, and is a strategy for improving cluster management efficiency and user experience in many aspects.

[0069] It should be noted that for each cluster node, when downloading each seed block file of each image layer, it can be downloaded from the node where the file block exists in the peer node, or it can be downloaded from the cache&seeder; in addition, when an exception occurs, the scheduler will still instruct the node to directly pull the image layer file directly from the image repository. When the file block exists in a certain node, the node will provide and share the download service of the file block to other nodes, and so on, until all the file blocks are downloaded.

[0070] Optionally, the downloader 46 is further used to: obtain second metadata of the first image file from the image repository according to the first file identifier, wherein the second metadata is used to indicate hash values ​​of multiple image layer files contained in the first image file, and the hash values ​​are used to identify image layer files; when multiple seed block files contained in the first image file are downloaded from the second cluster node, perform integrity verification on the first image file according to the multiple hash values, wherein each image layer file contains multiple seed block files; when the first image file passes the integrity verification, determine that the download of the first image file is complete, and combine the multiple seed block files into the first image file.

[0071] When downloading an image layer, the image layer will be divided into multiple seed block files and downloaded in parallel. The way of segmented download is to set the corresponding range attribute in the HTTP header. Preferably, the size of each seed block file can be selected as 4M. Therefore, the downloader also needs to obtain the second meta information of the first image file according to the first file identifier. The second meta information contains the hash values ​​of multiple image layer files, and then completes the inspection of the complete download of the image file according to the hash value. The specific process may include the following steps:

[0072] 1. Metadata check: Before downloading the image layer file, the downloader will first obtain the image metadata (i.e. the second metadata mentioned above) from the image repository, such as the manifest file. The manifest file contains information about each layer of the image, including the size of the layer, SHA256 checksum, etc.

[0073] 2. Block download and verification: The downloader will split the image layer into multiple block files for parallel download according to the information provided in the manifest file. After each block is downloaded, the downloader will use the SHA256 value of the block for verification to ensure the integrity of the data and that it has not been tampered with.

[0074] 3. Progress tracking: The downloader will track the download progress of each image layer block. Once all block files are downloaded and passed the verification, it will consider the image layer to have been downloaded.

[0075] 4. Layer reorganization: After downloading and verifying all block files, the downloader will reorganize these block files into the original image layer files. This process is usually performed by Docker Daemon.

[0076] Determine whether the entire image file has been downloaded:

[0077] 1. Image layer integrity check: Once all the block files of the image layer are downloaded and reassembled, Docker Daemon will perform an integrity check on the image layer to confirm that the SHA256 value of each layer is consistent with the value recorded in the manifest file.

[0078] 2. Image metadata verification: Image metadata contains information about the image version, layer structure, dependencies, etc. Docker Daemon verifies this information to ensure that the downloaded image matches the requested image.

[0079] 3. Image reassembly and verification: Docker Daemon reassembles all image layers in the order of construction to form a complete image file, and performs a final consistency check. If each layer of the image passes the integrity check and the image metadata is verified to be correct, then Docker Daemon will consider the entire image file downloaded.

[0080] 4. Status update: Once the download of the entire image file is confirmed to be complete, the downloader or Docker Daemon will update the status of the image and notify other components in the system, such as Scheduler and Controller, indicating that the image can now be used to start a new container or Pod.

[0081] This embodiment ensures the integrity and correctness of the image, which is a key step in starting and running the container service.

[0082] Optionally, the scheduler is further used to: match multiple fourth cluster nodes in the target cluster according to the first file identifier, wherein the multiple fourth cluster nodes all cache the first image file and include the second cluster node; determine the second cluster node among the multiple fourth cluster nodes according to the network topology connection relationship between the multiple fourth cluster nodes and the first cluster node and the node performance of the multiple fourth cluster nodes, wherein the network topology connection relationship is used to indicate the network connection status and geographical distance between the multiple fourth cluster nodes and the first cluster node.

[0083] Determining other nodes to pull images mainly depends on the intelligent decision-making of the Scheduler component and the characteristics of the P2P network. The Scheduler will select the most suitable node to pull the seed block file of the image layer based on a series of criteria, including but not limited to:

[0084] 1. Availability of image blocks: Scheduler monitors the image block information cached by the Seeder&Cache component of each node in the cluster. When a node needs to pull an image, Scheduler will query those nodes that already have some or all of the layer blocks of the image, and these nodes will become potential data sources.

[0085] 2. Network topology and connection status: Scheduler will consider the network connection status between nodes, including network bandwidth, latency, packet loss rate, etc., and give priority to nodes with better network conditions for data transmission. In addition, it will also consider the geographical location of the nodes, giving priority to nodes with similar geographical locations for transmission, so as to reduce data transmission across domains or across network devices and optimize network resource utilization.

[0086] 3. Node performance and resource status: Scheduler will evaluate the hardware configuration of the node (such as CPU, memory, storage resources) and the current load. Nodes with good performance, sufficient resources and low load are more suitable as data source nodes because they can provide mirror block data faster without affecting their normal operation.

[0087] 4. Historical transmission performance: Scheduler will record the performance of each node in historical transmission, including transmission speed, transmission success rate, etc. Nodes with excellent performance will be given priority to improve the efficiency and success rate of image pulling.

[0088] 5. Fault recovery and breakpoint resumption: During the transmission process, if a node fails, the Scheduler will select another node to continue transmitting the remaining image blocks based on the breakpoint resumption mechanism to ensure the integrity and continuity of the image pull.

[0089] 6. Resource sharing within the cluster: Scheduler encourages resource sharing within the cluster. Even if a node has only part of the image block, other nodes can obtain the part it has from this node, and then obtain the remaining part from other nodes or other image repositories, thereby building a more compact and efficient data transmission network between nodes.

[0090] 7. Policies and rules: Scheduler components usually follow specific policies and rules. These rules can be performance-based or cost-based. For example, they may give priority to intranet node transmission to reduce public network bandwidth costs, or select nodes with more stable network conditions as data sources in edge computing environments.

[0091] After the seed block file is downloaded and cached, it is registered with the sheduler component and begins to participate in the transmission of block files between nodes. The Scheduler component records the life cycle of each block file, especially when and how it is transmitted to which node. The Scheduler component selects the best parent node (the second cluster node mentioned above) for a node to download based on the distribution of a certain sub-block file in the cluster, the node's network topology, historical performance, network transmission capacity and other factors, and guides the nodes that still lack the file block to where to send requests for download (mirror warehouse or seeder&cache or peer node). It should be mentioned that peer nodes under the same network device can be made to transmit blocks to each other as much as possible to reduce traffic across network devices and computer rooms, thereby further reducing network bandwidth costs.

[0092] Through the above embodiments, the scheduler can dynamically and intelligently decide which nodes provide mirror block data to other nodes, and which specific data blocks to provide. This decision-making process is dynamic and will be adjusted as the cluster status, network conditions and node resources change, ensuring the efficiency and stability of the mirror pulling process.

[0093] It should be noted that the functions of Scheduler include:

[0094] 1. Used to receive and manage client download tasks, and notify the seeder & cacher to download images from the image repository;

[0095] 2. Responsible for managing and maintaining the life cycle of seed blocks;

[0096] 3. Generate and maintain a P2P network consisting of all relevant nodes, select a reasonable download path for each image download task on each node, and schedule block data transmission between nodes.

[0097] In an alternative embodiment, Figure 5The working process of a method for downloading an image file is shown. The method for downloading an image file is applied to the image file downloading system described in the above embodiment. Figure 5 As shown, the image file download method includes the following steps:

[0098] 1. Under normal circumstances, the container runtime installed in the node (such as docker, containerd, etc.) or the node agent (kubelet) running in the working node will pull the container image from the image repository (such as dockerhub, harbor, etc.). Hijacker, as a resident agent, will intercept and process requests related to image layering. For example, hijacker will intercept all HTTP requests directly initiated by the container engine pull command, intercept image pull requests initiated by the container engine hosting when kubernetes is deployed for service, etc. The specific interception method is to modify the daemon.json file of the container runtime and add the first-order registry-mirror value as the listening address of hijacker. It should be noted that container images are different from ordinary files. They are composed of multiple layers of storage, and the download is also layered download, not a single file. In addition to the layer information, the container runtime sends a request to pull the image meta information (manifest json, recording the sha256 value of each layer of the image). Hijacker will only filter out the request to obtain the layer information, hijack and forward it.

[0099] 2. After the hijacker intercepts the image download layer information request, it redirects the request to the controller and converts it into a download request to the controller.

[0100] 3. After the controller receives the download request for the image layer information, it continues to forward the request to the scheduler component. After receiving the request, the scheduler component will create a download task based on the unique sha256 value in the download request, and use the unique value as the identifier of the download task, and register the node that initiated the request under the task. If the task already exists, only the node that initiated the request will be registered under the task. After the image layer download task is created, it will check whether the cache exists. If the cache exists, it will directly connect to the image repository for expiration verification. When sending the request, it will be accompanied by the last modification time of the file returned by the image repository. If the expiration check passes, it means that the cache file is valid. Then, it is determined whether the file is complete based on the metadata of the cache file. If it is complete, the cache is completely hit, and scheduling is performed directly to instruct the node where to download the block file.

[0101] 4. Trigger download. If the image layer cache does not exist, notify the seeder & cacher to connect to the image warehouse for the first download of the image layer file. If it is incomplete, continue to download the remaining files in segments according to the breakpoint resume method. If the expiration check fails, delete the original cache file and notify the seeder & cacher to connect to the image warehouse for another download of the image layer file.

[0102] 5. When the seeder & cacher downloads the image layer from the image repository, it will separate the image layer into multiple seed block files and download them in parallel. The way to download the fragments is to set the corresponding range attribute in the HTTP header. Preferably, the size of each seed block file can be selected as 4M.

[0103] 6. After the seed block file of the image layer is downloaded by the seeder & cacher, it will be cached locally and will serve as the root for the subsequent download of the file by the entire cluster. Moreover, once a block file is downloaded and generated, it can be understood and distributed in the entire distributed network. In order to save storage space, the seed block file can be compressed and transmitted when necessary. It should be mentioned that the seeder & cacher needs to have the ability to clean up garbage regularly to ensure that there is enough space to support long-term services.

[0104] 7. After a certain sub-block file is downloaded and cached, it will register with the sheduler component and begin to participate in the transmission of block files between nodes. The Scheduler component will record the life cycle of each block file, especially when and how it is transmitted to which node. The Scheduler component selects the best parent node for a node to download based on the distribution of a certain sub-block file in the cluster, the node's network topology, historical performance, network transmission capabilities and other factors, and guides the nodes that still lack the file block to where to send requests (mirror warehouse or seeder&cache or peer node) to download. It should be mentioned that peer nodes under the same network device can try to transmit blocks to each other, reducing traffic across network devices and across computer rooms, thereby further reducing network bandwidth costs.

[0105] 8. For each node, when downloading each seed block file of each image layer, it can be downloaded from the node where the file block exists in the peer node, or it can be downloaded from the cache&seeder; in addition, when an exception occurs, the scheduler will still instruct the node to directly pull the image layer file directly from the image repository. When the file block exists in a certain node, the node will provide and share the download service of the file block to other nodes, and so on, until all the file blocks are downloaded.

[0106] 9. Each layer of the image file will be divided into block data blocks and used as seeds for nodes to download. After all the seed blocks of a layer are downloaded, a layer of the image is downloaded and passed to the docker daemon for use. When all layers are downloaded, the docker daemon considers that the entire image is downloaded, and reassembles it into a complete image through the unique ID and sha256 value of each layer of the image to ensure the consistency of the download process, thus completing a distributed image download process.

[0107] The image file downloading method provided in the embodiment of the present application has the following features:

[0108] 1. Reduce service startup and expansion latency. After the K8s control plane dispatches the Pod to the working node, hijacker intercepts the image pull request and performs image layer seed block file mutuality in a distributed and decentralized manner, reusing the transmission capacity on each node, making maximum use of the cluster's network resources, and improving download efficiency, especially in the process of distributing large-volume images, especially in the process of distributing images of large clusters (hundreds or thousands of nodes), especially in the process of starting and expanding response-sensitive services.

[0109] 2. Remove the restrictions on network connectivity and stability in edge scenarios. In edge scenarios, not all nodes can access the image repository, or different nodes have different network connectivity performance to the image repository, and there are certain network stability restrictions. In the distributed image distribution solution described in this patent, in addition to pulling images from the image repository, nodes can also pull image layer seed files from all nodes of the seeder & cache, and can also pull image layer seed files from all peer nodes to finally form an image, removing the restrictions on network connectivity and stability.

[0110] 3. Prevent repeated downloads and relieve the pressure on the image warehouse. When the scheduler receives a request, it will check whether the corresponding image file has been cached in the seeder & cacher. If there is a cache and the cache comparison has not expired, it will not be pulled from the image warehouse again, preventing repeated downloads; therefore, the image warehouse only needs to handle the first pull task in the image life cycle, which greatly relieves the network pressure of the image warehouse.

[0111] 4. Traffic allocation. When scheduling the transmission of image layer seed block files, the scheduler component considers factors such as machine hardware configuration, geographical location, network environment, historical performance, etc. Therefore, when the topological distance between two nodes in the network is far, the data transmission between the two nodes will be reduced, thus ensuring that the blocks are transmitted to each other in the same network area as much as possible, reducing the traffic across network devices.

[0112] 5. Support breakpoint resume. When the scheduler component creates a download task for the container runtime, it uses the sha256 of the image as an identifier, so when a repeated download task is initiated, it can be transferred in seconds. Secondly, the scheduler component records the progress information of the image layer download process and the life cycle of the image layer seed block file on each node. Therefore, when the download is interrupted due to a network crash, the image download task of the container runtime is executed again. The scheduler will schedule the subsequent seed block download under the current record, that is, breakpoint resume.

[0113] 6. No intrusion into container operation. The Hijacker component selects and hijacks the download request of the image layer from the image repository when the container is running, and does not cause any damage to the container runtime. In addition, after all the seed block files of the image layer are downloaded, the image layer is merged and handed over to the container runtime for consistency verification of the image layer and confirmation of the integrity of the image, forming a closed loop of the entire image distribution link.

[0114] Through the description of the above implementation methods, those skilled in the art can clearly understand that the method according to the above embodiment can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk), and includes a number of instructions for a terminal device (which can be a mobile phone, computer, server, or network device, etc.) to execute the methods described in each embodiment of the present application.

[0115] In this embodiment, a method for downloading an image file is also provided, which is applied to the above-mentioned image file downloading system. Figure 6 is a flowchart of a method for downloading an image file according to an embodiment of the present application. Figure 6 As shown, the method comprises the following steps:

[0116] Step S602: intercepting the image download request initiated by the container engine in the first cluster node, and forwarding the image download request to the scheduler;

[0117] Step S604, matching a second cluster node according to the first file identifier of the first image file, generating a first download request according to the first file identifier and the second cluster node, and sending the first download request to the second cluster node;

[0118] Step S606: upon receiving the first scheduling policy instruction of the second cluster node, downloading the first image file from the second cluster node according to the first scheduling policy instruction.

[0119] Through the above method, the image download request initiated by the container engine of the first cluster node is intercepted and forwarded to the scheduler for processing; then the second cluster node is matched according to the first file identifier of the first image file, so as to generate a first download request according to the first file identifier and the second cluster node, and then the first download request is sent to the second cluster node; finally, when the first scheduling policy instruction sent by the second cluster node is received, the first image file is downloaded from the second cluster node according to the first scheduling policy instruction; the above scheme is adopted, and the traditional image download architecture is changed by introducing the P2P network mechanism, so that nodes of the group can transmit and share image files with each other, which greatly improves the efficiency of image distribution; thereby solving the problem of low efficiency of large-scale container image file download in the kubernetes cluster environment in the related technology.

[0120] It should be noted that the above modules can be implemented by software or hardware. For the latter, it can be implemented in the following ways, but not limited to: the above modules are all located in the same processor; or the above modules are located in different processors in any combination.

[0121] An embodiment of the present application further provides a computer-readable storage medium, in which a computer program is stored, wherein the computer program is configured to execute the steps of any of the above method embodiments when running.

[0122] In an exemplary embodiment, the computer-readable storage medium may include, but is not limited to, various media that can store computer programs, such as a USB flash drive, a read-only memory (ROM), a random access memory (RAM), a mobile hard disk, a magnetic disk or an optical disk.

[0123] An embodiment of the present application further provides an electronic device, including a memory and a processor, wherein a computer program is stored in the memory, and the processor is configured to run the computer program to execute the steps in any one of the above method embodiments.

[0124] In an exemplary embodiment, the electronic device may further include a transmission device and an input / output device, wherein the transmission device is connected to the processor, and the input / output device is connected to the processor.

[0125] An embodiment of the present application further provides a computer program product, comprising a non-volatile computer-readable storage medium, wherein the non-volatile computer-readable storage medium stores a computer program product, and when the computer program is executed by a processor, the steps of the method described in each embodiment of the present application are implemented.

[0126] For specific examples in this embodiment, reference may be made to the examples described in the above embodiments and exemplary implementation modes, and this embodiment will not be described in detail herein.

[0127] Obviously, those skilled in the art should understand that the above modules or steps of the present application can be implemented by a general computing device, they can be concentrated on a single computing device, or distributed on a network composed of multiple computing devices, they can be implemented by a program code executable by a computing device, so that they can be stored in a storage device and executed by the computing device, and in some cases, the steps shown or described can be executed in a different order from that herein, or they can be made into individual integrated circuit modules, or multiple modules or steps therein can be made into a single integrated circuit module for implementation. Thus, the present application is not limited to any specific combination of hardware and software.

[0128] The above description is only the preferred embodiment of the present application and is not intended to limit the present application. For those skilled in the art, the present application may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the principles of the present application shall be included in the protection scope of the present application.

Claims

1. A system for downloading an image file, characterized in that: Set in the first cluster node in the target cluster, include: A peer node, used to intercept an image download request initiated by a container engine in the first cluster node, and forward the image download request to a scheduler, wherein the image download request is used to request to download a first image file from an image repository; the scheduler, connected to the peer node, is used to match a second cluster node according to a first file identifier of the first image file, and generates a first download request according to the first file identifier and the second cluster node, and sends the first download request to the second cluster node, wherein the first download request is used to request to download the first image file from the second cluster node, and the target cluster includes the second cluster node; The peer node is further configured to, upon receiving the first scheduling policy instruction of the second cluster node, download the first image file from the second cluster node according to the first scheduling policy instruction.

2. The system according to claim 1, characterized in that The peer node comprises: The hijacker is used to intercept the image download request when receiving the image download request, and redirect the image download request to the controller; The controller is connected to the hijacker and is used to forward the image download request to the scheduler to instruct the scheduler to generate the first download request, and when receiving the first scheduling policy instruction, generate a download instruction according to the first scheduling policy instruction, and send the download instruction to the downloader; The downloader is connected to the controller and is used to execute the download instruction to download the first image file from the second cluster node.

3. The system according to claim 2, characterized in that The scheduler is further used for: In the case of receiving a second download request sent by the third cluster node, querying the second image file in the seed cache according to the second file identifier carried in the second download request, wherein the second download request is used by the third cluster node to request the third cluster node to download the second image file from the first cluster node, the target cluster includes the third cluster node, the first cluster node includes the seed cache, and the seed cache is used to cache multiple seed block files included in the image file; When the second image file is found, requesting the image repository for the last modification time of the second image file according to the second file identifier, and performing an expiration check on the second image file according to the last modification time; If it is determined that the second image file passes the expiration check, performing an integrity check on the second image file according to the first meta information of the second image file; If it is determined that the second image file fails the integrity check, generating a third download request according to the second file identifier, and sending the third download request to the seed cache to instruct the seed cache to download the missing files of the second image file from the image repository, and re-checking the integrity of the second image file; When it is determined that the second image file passes the integrity check, a second scheduling policy instruction is sent to a peer node of the third cluster node to instruct the third cluster node to download the second image file.

4. The system according to claim 3, characterized in that The scheduler is further used for: If the second image file is not found, generating a fourth download request according to the second file identifier; The fourth download request is sent to the seed cache to instruct the seed cache to download the second image file from the image repository.

5. The system according to claim 3, characterized in that The scheduler is further used for: If it is determined that the second image file fails the expiration check, deleting the second image file, and generating a fifth download request according to the second file identifier; The fifth download request is sent to the seed cache to instruct the seed cache to re-download the second image file from the image repository.

6. The system according to claim 3, characterized in that The downloader is also used to: Acquire second meta information of the first image file from the image repository according to the first file identifier, wherein the second meta information is used to indicate hash values ​​of multiple image layer files included in the first image file, and the hash values ​​are used to identify the image layer files; In the case where multiple seed block files included in the first image file are downloaded from the second cluster node, performing integrity check on the first image file according to the multiple hash values, wherein each image layer file includes multiple seed block files; When the first image file passes the integrity check, it is determined that the downloading of the first image file is complete, and the multiple seed block files are combined into the first image file.

7. The system according to claim 1, characterized in that The scheduler is further used for: Matching a plurality of fourth cluster nodes in the target cluster according to the first file identifier, wherein the plurality of fourth cluster nodes all cache the first image file, and the plurality of fourth cluster nodes include the second cluster node; The second cluster node is determined among the multiple fourth cluster nodes based on the network topology connection relationship between the multiple fourth cluster nodes and the first cluster node and the node performance of the multiple fourth cluster nodes, wherein the network topology connection relationship is used to indicate the network connection status and geographical distance between the multiple fourth cluster nodes and the first cluster node.

8. A method for downloading an image file, characterized in that: A system for downloading an image file according to any one of claims 1 to 7, include: Intercepting an image download request initiated by the container engine in the first cluster node, and forwarding the image download request to the scheduler; Match the second cluster node according to the first file identifier of the first image file, generate a first download request according to the first file identifier and the second cluster node, and send the first download request to the second cluster node; upon receiving the first scheduling policy instruction of the second cluster node, download the first image file from the second cluster node according to the first scheduling policy instruction.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, wherein the computer program implements the steps of the method described in claim 8 when executed by a processor.

10. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the steps of the method according to claim 8 are implemented.

Images