Mirror image loading method and device, electronic equipment, storage medium and program product

Abstract

The invention provides a mirror image loading method and device, electronic equipment, a storage medium and a program product, and relates to the technical field of service deployment. The method comprises the steps of obtaining a mirror image packet; reading metadata of each mirror image layer from the configuration file; extracting the diffID of each mirror image layer; according to the diffID of each mirror image layer, determining a chainID of each mirror image layer, snapshot metadata corresponding to each chainID and an SHA256 layer sequence relation chain; the metadata, the chainID of each mirror image layer, the snapshot metadata and the SHA256 layer sequence relation chain are written into a metadatabase of the Container system, and the metadata, the chainID of each mirror image layer, the snapshot metadata and the SHA256 layer sequence relation chain are written into the metadatabase of the Container system; determining a loading strategy of the plurality of mirror image layers; and starting a plurality of goroutines, and loading the plurality of mirror image layers by utilizing the plurality of goroutines according to the loading strategy. The mirror image loading mode is short in time consumption and high in efficiency.

Description

Technical Field

[0001] The present application relates to the field of service deployment technology, and in particular to an image loading method, device, electronic device, storage medium, and program product. Background Art

[0002] With the rapid development of network service architectures, container and image technologies are increasingly being used in service deployment. This technology enables flexible deployment of network services on different computing nodes. During network service deployment, the time required to load container images directly affects the quality of service.

[0003] In related technologies, when a container (such as Containerd) is running, a serial loading strategy is used to load each image layer during the image loading process.

[0004] However, serially loading each image layer in Containerd is time-consuming and inefficient.

[0005] It should be noted that the information disclosed in the above background technology section is only used to enhance the understanding of the background of this application, and therefore may include information that does not constitute prior art known to ordinary technicians in this field. Summary of the Invention

[0006] The present application provides an image loading method, device, electronic device, storage medium and program product, which at least to some extent overcome the problems of long time consumption and low efficiency in loading each image layer in the related art.

[0007] Other features and advantages of the present application will become apparent from the following detailed description, or may be learned in part by practice of the present application.

[0008] According to one aspect of the present application, a method for loading an image is provided, which is applied to a local device in which a Containerd system (an open source container runtime) is deployed, comprising: obtaining an image package, wherein the image package includes a configuration file and multiple image layers, wherein the configuration file includes a manifest.json file and a config.json file; reading metadata of each image layer from the configuration file; reading information of a Layers list from the manifest.json file to obtain a file name for each image layer, and separating the multiple image layers from the configuration file according to the file name of each image layer; extracting a diffID for each image layer from the rootfs / diff_ids list in the config.json file; and extracting a diffID for each image layer according to the diff fID, determine the chainID of each image layer, the snapshot metadata corresponding to each chainID, and the SHA256 layer sequence relationship chain; write the metadata, the chainID of each image layer, the snapshot metadata, and the SHA256 layer sequence relationship chain into the metadata database of the Containerd system to configure the image loading environment in the Containerd system; determine the loading strategy of the multiple image layers; create a snapshot directory for loading image layers in the Containerd system, the snapshot directory includes multiple subdirectories, and the subdirectories correspond to the image layers one by one; start multiple goroutines (threads), and use the multiple goroutines to load the multiple image layers according to the loading strategy, so that each image layer is loaded into the corresponding subdirectory.

[0009] By extracting the file name and diffID of each image layer from the configuration file of the image package, calculating the chainID, and writing the relevant metadata into the metadata database of the Containerd system, the restrictions on the dependencies between image layers are removed and the loading of each image layer is decoupled, so that each image layer can be loaded separately. Then, by starting multiple goroutines to load the multiple image layers, parallel loading of image layers in the Containerd system is achieved. Compared with the serial loading of image layers in the Containerd system, the time required for image loading is greatly reduced and the efficiency of image loading is improved.

[0010] In some embodiments, determining the chainID of each image layer according to the diffID of each image layer includes: calculating the chainID of the image layer according to the following formula based on the diffID of the image layer:

[0011]

[0012] Among them, chainID i is the chainID of the i-th image layer, SHA256(.) is the function corresponding to the SHA256 algorithm, diffID i is the diffID of the i-th image layer.

[0013] In some embodiments, the configuration file also includes an index.json file, and the metadata includes tag metadata and timestamp metadata; reading the metadata of each image layer from the configuration file includes: reading the tag metadata of each image layer from the index.json file, the manifest.json file, and the config.json file; and reading the timestamp metadata of the image from the index.json file.

[0014] In some embodiments, the metadata database includes meta.db and metadata.db; writing the metadata, the chainID of each image layer, the snapshot metadata and the SHA256 hierarchical relationship chain into the metadata database of the Containerd system includes: writing the tag metadata into the v1 / default / content / blob sub-bucket of the meta.db; writing the timestamp metadata into the v1 / default / images sub-bucket of the meta.db; writing the snapshot metadata into the v1 / default / snapshotss / overlayfs sub-bucket of the meta.db; and writing the SHA256 hierarchical relationship chain into the v1 / snapshotss sub-bucket of the metadata.db.

[0015] By reading the configuration file and pre-processing the tag metadata and timestamp metadata, and writing the relevant metadata into the meta.db and metadata.db databases, you can configure an image loading environment in the Containerd system that allows multiple image layers to be loaded in parallel, laying the foundation for parallel loading of multiple image layers.

[0016] In some embodiments, determining the loading strategy of the multiple image layers includes: determining the loading time of each image layer according to the following formula:

[0017]

[0018] Among them, B p (i) is the size of the i-th image layer before decompression, μ(i) is the decompression rate of the i-th image layer, and W is the storage bandwidth of the local device;

[0019] Determine, based on available processing resources of the local device, to start the multiple goroutines for loading the image layer, where the number of the multiple goroutines is N; and construct a Mixed Integer Linear Programming (MILP) problem based on the loading time and the multiple goroutines. The objective function of the MILP problem is:

[0020] min(T max )

[0021] Among them, min(.) is the minimum value function; T max T is the maximum loading time of loading the multiple image layers in the multiple goroutines. max It is expressed as the following formula:

[0022]

[0023] max(.) is the maximum value function; r is used to represent the rth goroutine among N goroutines; Equal to 0 or 1, if the i-th image layer is allocated on the r-th goroutine, Equal to 1, if the i-th image layer is not allocated on the r-th goroutine, =0; L is the number of mirror layers in the plurality of mirror layers;

[0024] The constraints of the MILP problem include allocation constraints and time constraints. The allocation constraints are expressed as the following formula:

[0025]

[0026] The time constraint is expressed as the following formula:

[0027]

[0028] Based on the Benders algorithm, the MILP problem is decomposed into a main problem and sub-problems, and iterative calculation is performed until the preset stopping condition is met. As the loading strategy, the main problem is to minimize T max To solve the allocation scheme of the multiple image layers on the multiple goroutines The subproblem is to calculate the current T below max And generate cutting constraints.

[0029] The problem of loading multiple image layers in parallel is modeled as a MILP problem. Based on the efficient iterative solution algorithm of Benders decomposition, the loading tasks of multiple image layers are evenly distributed to multiple goroutines for parallel processing, thereby minimizing the time required for image loading.

[0030] In some embodiments, creating a snapshot directory for loading the image layer in the Containerd system includes: creating a corresponding subdirectory for each image layer in ascending order under the snapshots directory of the Containerd system to obtain the multiple subdirectories.

[0031] According to another aspect of the present application, there is also provided an image loading device, which is applied to a local device, in which a Containerd system is deployed, including: an acquisition module for obtaining an image package, wherein the image package includes a configuration file and multiple image layers, and the configuration file includes a manifest.json file and a config.json file; a configuration module for reading metadata of each image layer from the configuration file; reading the information of the Layers list from the manifest.json file to obtain the file name of each image layer, and separating the multiple image layers from the configuration file according to the file name of each image layer; extracting the diffID of each image layer from the rootfs / diff_ids list of the config.json file; and determining the image layer according to the diffID of each image layer. Determine the chainID of each image layer, the snapshot metadata corresponding to each chainID, and the SHA256 layer sequence relationship chain; write the metadata, the chainID of each image layer, the snapshot metadata, and the SHA256 layer sequence relationship chain into the metadata database of the Containerd system to configure the image loading environment in the Containerd system; create a snapshot directory for loading image layers in the Containerd system, the snapshot directory includes multiple subdirectories, and the subdirectories correspond to image layers one by one; a loading strategy determination module is used to determine the loading strategy of the multiple image layers; a loading module is used to start multiple goroutines, and use the multiple goroutines to load the multiple image layers according to the loading strategy, so as to load each image layer into the corresponding subdirectory.

[0032] In some embodiments, the configuration module is configured to calculate the chainID of the image layer according to the diffID of the image layer according to the following formula:

[0033]

[0034] Among them, chainIDi is the chainID of the i-th image layer, SHA256(.) is the function corresponding to the SHA256 algorithm, diffID i is the diffID of the i-th image layer.

[0035] In some embodiments, the configuration file also includes an index.json file, and the metadata includes tag metadata and timestamp metadata; the configuration module is used to read the tag metadata of each image layer from the index.json file, the manifest.json file, and the config.json file; and read the timestamp metadata of the image from the index.json file.

[0036] In some embodiments, the metadata database includes meta.db and metadata.db; the configuration module is used to write the tag metadata into the v1 / default / content / blob sub-bucket of the meta.db; write the timestamp metadata into the v1 / default / images sub-bucket of the meta.db; write the snapshot metadata into the v1 / default / snapshotss / overlayfs sub-bucket of the meta.db; and write the SHA256 hierarchical relationship chain into the v1 / snapshotss sub-bucket of the metadata.db.

[0037] In some embodiments, the loading strategy determination module is used to determine the loading time of each image layer according to the following formula:

[0038]

[0039] Among them, B p (i) is the size of the i-th image layer before decompression, μ(i) is the decompression rate of the i-th image layer, and W is the storage bandwidth of the local device;

[0040] Determine, based on available processing resources of the local device, to start the multiple goroutines for loading the image layer, where the number of the multiple goroutines is N; and construct a MILP problem based on the loading time and the multiple goroutines, where the objective function of the MILP problem is:

[0041] min(T max )

[0042] Among them, min(.) is the minimum value function; T max T is the maximum loading time of loading the multiple image layers in the multiple goroutines. maxIt is expressed as the following formula:

[0043]

[0044] max(.) is the maximum value function; r is used to represent the rth goroutine among N goroutines; Equal to 0 or 1, if the i-th image layer is allocated on the r-th goroutine, Equal to 1, if the i-th image layer is not allocated on the r-th goroutine, =0; L is the number of mirror layers in the plurality of mirror layers;

[0045] The constraints of the MILP problem include allocation constraints and time constraints. The allocation constraints are expressed as the following formula:

[0046]

[0047] The time constraint is expressed as the following formula:

[0048]

[0049] Based on the Benders algorithm, the MILP problem is decomposed into a main problem and sub-problems, and iterative calculation is performed until the preset stopping condition is met. As the loading strategy, the main problem is to minimize T max To solve the allocation scheme of the multiple image layers on the multiple goroutines The subproblem is to calculate the current T below max And generate cutting constraints.

[0050] In some embodiments, the configuration module is used to create a corresponding subdirectory for each image layer in the snapshots directory of the Containerd system according to the incremental numbering to obtain the multiple subdirectories.

[0051] According to another aspect of the present application, an electronic device is also provided, comprising: a processor; and a memory for storing executable instructions of the processor; wherein the processor is configured to execute any one of the above-mentioned image loading methods by executing the executable instructions.

[0052] According to another aspect of the present application, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the image loading method described above is implemented.

[0053] According to another aspect of the present application, a computer program product is provided, including a computer program, which implements any of the above-mentioned image loading methods when executed by a processor.

[0054] It should be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0055] The accompanying drawings are incorporated into and constitute a part of the specification, illustrate embodiments consistent with the present application, and together with the specification, are used to explain the principles of the present application. Obviously, the drawings described below are only some embodiments of the present application, and those skilled in the art can derive other drawings based on these drawings without inventive effort.

[0056] Figure 1 A schematic diagram of an image loading system according to an embodiment of the present application is shown;

[0057] Figure 2 A schematic diagram showing an image package in an embodiment of the present application is shown;

[0058] Figure 3 A schematic diagram showing a 5-layer mirror image according to an embodiment of the present application is shown;

[0059] Figure 4 A flow chart of an image loading method according to an embodiment of the present application is shown;

[0060] Figure 5 A schematic diagram of an image loading device according to an embodiment of the present application is shown;

[0061] Figure 6 A structural block diagram of an electronic device according to an embodiment of the present application is shown;

[0062] Figure 7 A schematic diagram showing a comparison of multiple image loading strategies in an embodiment of the present application is shown;

[0063] Figure 8 A schematic diagram showing a loading acceleration ratio compared to the native Containerd strategy in an embodiment of the present application. DETAILED DESCRIPTION

[0064] Example embodiments will now be described more fully with reference to the accompanying drawings. However, example embodiments can be embodied in many forms and should not be construed as limited to the examples set forth herein; rather, these embodiments are provided so that this disclosure will be thorough and complete and will fully convey the concepts of the example embodiments to those skilled in the art. The described features, structures, or characteristics may be combined in any suitable manner in one or more embodiments.

[0065] In addition, the accompanying drawings are merely schematic illustrations of the present application and are not necessarily drawn to scale. Identical reference numerals in the figures denote identical or similar parts, and thus repetitive descriptions thereof will be omitted. Some of the blocks shown in the accompanying drawings are functional entities that do not necessarily correspond to physically or logically separate entities. These functional entities may be implemented in software, in one or more hardware modules or integrated circuits, or in different networks and / or processor devices and / or microcontroller devices.

[0066] The specific implementation of the embodiment of the present application is described in detail below with reference to the accompanying drawings.

[0067] Figure 1 A schematic diagram of the image loading system in the embodiment of the application is shown. Figure 1 As shown, the system may include a server 11 and a local device 12 .

[0068] The server 11 has a mirror package (mirror compressed package), and the local device 12 can download the mirror package from the server 11 .

[0069] like Figure 2 As shown, the image package 20 includes a configuration file 201 and multiple image layers 202.

[0070] The local device 12 may separate the configuration file 201 and the plurality of image layers 202 , and read metadata of each image layer from the configuration file 201 .

[0071] The local device 12 can also read the information of the Layers list from the manifest.json file included in the configuration file, obtain the file name of each image layer (a SHA256 value), and separate the configuration file 201 and multiple image layers 202 according to the file name.

[0072] Furthermore, the rootfs / diff_ids list is read from the config.json file to extract the diffID of each image layer. Based on the diffID of each image layer, the chainID of each image layer, the snapshot metadata corresponding to each chainID, and the sha layer sequence relationship chain are determined. The snapshot metadata corresponding to the chainID includes: the chainID of the current layer, the chainID of the parent layer, the chainID of the child layer, and the timestamp.

[0073] like Figure 3As shown, taking the example of multiple image layers 202 including five image layers, the dependency relationship between the five image layers is as follows: image layer L1 is the parent image layer of image layer L2, image layer L2 is the parent image layer of image layer L3, image layer L3 is the parent image layer of image layer L4, and image layer L4 is the parent image layer of image layer L5. Accordingly, for image layer L1, the current layer chain ID is L1-chainID, the parent layer chain ID is empty, and the child layer chain ID is L2-chainID. For image layer L2, the current layer chain ID is L2-chainID, the parent layer chain ID is L1-chainID, and the child layer chain ID is L3-chainID.

[0074] Each image layer has a unique SHA256 identifier (diffID and chainID), and the SHA256 layer sequence chain describes the dependency order of these SHA256 identifiers, such as the snapshot metadata corresponding to each chainID.

[0075] A Containerd system is deployed in the local device 12, and metadata, chainID of each image layer, snapshot metadata and SHA256 layer sequence relationship chain can be written into the metadata database of the Containerd system to configure the image loading environment in the Containerd system.

[0076] The local device 12 may also determine the number of goroutines that can be started based on its currently available processing resources, and further determine a loading strategy for loading the multiple image layers 202 in parallel.

[0077] And create a snapshot directory for loading the image layer in the Containerd system, and load the image layer to the corresponding subdirectory according to the loading strategy.

[0078] The server 11 and the local device 12 communicate with each other via a network, which may be a wired network or a wireless network.

[0079] Optionally, the above-mentioned wireless network or wired network uses standard communication technologies and / or protocols. The network is typically the Internet, but it can also be any network, including but not limited to a local area network (LAN), a metropolitan area network (MAN), a wide area network (WAN), a mobile, wired or wireless network, a private network or any combination of a virtual private network). In some embodiments, technologies and / or formats including Hypertext Markup Language (HTML), Extensible Markup Language (XML), etc. are used to represent data exchanged over the network. In addition, conventional encryption technologies such as Secure Socket Layer (SSL), Transport Layer Security (TLS), Virtual Private Network (VPN), Internet Protocol Security (IPSec), etc. can be used to encrypt all or some links. In other embodiments, customized and / or dedicated data communication technologies can also be used to replace or supplement the above-mentioned data communication technologies.

[0080] The local device 12 may be any electronic device, including but not limited to a smartphone, a tablet computer, a desktop computer, and the like.

[0081] Server 11 can be a server that provides various services. Optionally, the server can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers. It can also be a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN (Content Delivery Network), and big data and artificial intelligence platforms.

[0082] Under the above system architecture, an embodiment of the present application provides an image loading method that can be executed by any electronic device with computing processing capabilities. For example, the electronic device is a local device that has a Containerd system deployed on it.

[0083] Figure 4 A flow chart of an image loading method according to an embodiment of the present application is shown as follows: Figure 4 As shown, the image loading method provided in the embodiment of the present application includes the following S401 to S408.

[0084] S401, obtain an image package, which includes a configuration file and multiple image layers. The configuration file includes a manifest.json file and a config.json file.

[0085] The embodiment of the present application does not limit which network service the image package is specifically for. For example, the network service is Nginx (a high-performance web server commonly used to host static web pages (e.g., HTML, CSS, JS files, etc.)).

[0086] Obtaining the mirror package may include downloading the mirror package from a server storing the mirror package.

[0087] S402: Read metadata of each image layer from the configuration file.

[0088] In one embodiment, the configuration file further includes an index.json file, and the metadata includes tag metadata and timestamp metadata. Reading the metadata of each image layer from the configuration file includes: reading the tag metadata of each image layer from the index.json file, the manifest.json file, and the config.json file; and reading the timestamp metadata of the image from the index.json file.

[0089] The tag metadata may include information such as the image layer number, size, and tag. The timestamp metadata may include the timestamps of when the image layer and related information were created and modified.

[0090] S403, read the information of the Layers list from the manifest.json file to obtain the SHA256 value of each image layer, and read the rootfs / diff_ids list from the config.json file to extract the diffID of each image layer.

[0091] S404 , determining the chainID of each image layer and the snapshot metadata corresponding to each chainID according to the diffID of each image layer, and determining the SHA256 layer sequence relationship chain of each image layer according to the SHA256 value of each image layer.

[0092] In one embodiment, the chainID of each image layer and each chainID are determined according to the diffID of each image layer. The chainID of the image layer can be calculated according to the following formula 1 based on the diffID of the image layer:

[0093]

[0094] Among them, chainID i is the chainID of the i-th image layer, SHA256(.) is the function corresponding to the SHA256 algorithm, diffID i is the diffID of the i-th image layer.

[0095] The snapshot metadata corresponding to each chainID includes the chainID of the current layer, the chainID of the parent layer, the chainID of the child layer, and the timestamp of each image layer.

[0096] S405: Write the metadata, the chainID of each image layer, the snapshot metadata, and the SHA256 layer sequence relationship chain into the metadata repository of the Containerd system to configure the image loading environment in the Containerd system.

[0097] In one embodiment, the metadata database includes meta.db and metadata.db, and the metadata, the chainID of each image layer, the snapshot metadata and the SHA256 hierarchical relationship chain are written into the metadata database of the Containerd system, including: writing the tag metadata into the v1 / default / content / blob sub-bucket of the meta.db; writing the timestamp metadata into the v1 / default / images sub-bucket of the meta.db; writing the snapshot metadata into the v1 / default / snapshotss / overlayfs sub-bucket of the meta.db; and writing the SHA256 hierarchical relationship chain into the v1 / snapshotss sub-bucket of the metadata.db.

[0098] By reading the configuration file and pre-processing the tag metadata and timestamp metadata, and writing the relevant metadata into the meta.db and metadata.db databases, you can configure an image loading environment in the Containerd system that allows multiple image layers to be loaded in parallel, laying the foundation for parallel loading of multiple image layers.

[0099] S406: Determine a loading strategy for the multiple image layers.

[0100] The loading strategy may be to arbitrarily assign the loading tasks corresponding to the multiple image layers to multiple goroutines for loading.

[0101] In one embodiment, determining the loading strategy for the multiple image layers includes:

[0102] Determine the loading time of each image layer according to the following formula 2:

[0103]

[0104] Among them, B p (i) is the size of the i-th image layer before decompression, μ(i) is the decompression rate of the i-th image layer, and W is the storage bandwidth of the local device.

[0105] According to the available processing resources of the local device, it is determined to start the multiple goroutines for loading the image layer, and the number of the multiple goroutines is N.

[0106] Based on the loading time and the multiple goroutines, a MILP problem is constructed. The objective function of the MILP problem is shown in the following formula 3:

[0107] min(T max )(3)

[0108] Among them, min(.) is the minimum value function; T max The maximum loading time for loading the multiple image layers in the multiple goroutines, T max It is expressed as the following formula 4:

[0109]

[0110] max(.) is the maximum value function; r is used to represent the rth goroutine among N goroutines; Equal to 0 or 1, if the i-th image layer is allocated on the r-th goroutine, Equal to 1, if the i-th image layer is not allocated on the r-th goroutine, is equal to 0; L is the number of mirror layers in the multiple mirror layers.

[0111] The constraints of the MILP problem include allocation constraints and time constraints. The allocation constraint is expressed as follows:

[0112]

[0113] The time constraint is expressed as the following formula 6:

[0114]

[0115] Based on the Benders algorithm (a method for solving complex mathematical programming problems), the MILP problem is decomposed into a main problem and sub-problems, and iterative calculations are performed until the preset stopping condition is met. As the loading strategy, the main problem is to minimize T max To solve the allocation scheme of the multiple image layers on the multiple goroutines This subproblem is to calculate the current T below max And generate cutting constraints.

[0116] The problem of loading multiple image layers in parallel is modeled as a MILP problem. Based on the efficient iterative solution algorithm of Benders decomposition, the loading tasks of multiple image layers are evenly distributed to multiple goroutines for parallel processing, thereby minimizing the time required for image loading.

[0117] S407: Create a snapshot directory for loading the image layer in the Containerd system. The snapshot directory includes multiple subdirectories, and the subdirectories correspond to the image layers in a one-to-one manner.

[0118] In one embodiment, a snapshot directory for loading the image layer is created in the Containerd system, including: creating a corresponding subdirectory for each image layer in ascending order under the snapshots directory of the Containerd system to obtain the multiple subdirectories.

[0119] For example, if there are 6 image layers in total and the largest existing subdirectory number under the snapshots directory is 9, then the numbers of the newly created directories should be 10 to 15.

[0120] S408: Start multiple goroutines and use the multiple goroutines to load the multiple image layers according to the loading strategy, so as to load each image layer into a corresponding subdirectory.

[0121] By extracting the file name and diffID of each image layer from the configuration file of the image package, calculating the chainID, and writing the relevant metadata into the metadata database of the Containerd system, the restrictions on the dependencies between image layers are removed and the loading of each image layer is decoupled, so that each image layer can be loaded separately. Then, by starting multiple goroutines to load the multiple image layers, parallel loading of image layers in the Containerd system is achieved. Compared with the serial loading of image layers in the Containerd system, the time required for image loading is greatly reduced and the efficiency of image loading is improved.

[0122] like Figure 7As shown, for the six popular images on Dockerhub (an online platform provided by Docker): python:3.9.3, nginx:1.19.10, redis:6.2.1, cassandra:3.11.9, ghost:3.42.5-alpine and httpd:2.4.43, the method of the present application is compared with the existing native Containerd strategy and LOPO (LongestChain Out-of-order Layer Pulling Orchestration Strategy, an out-of-order layer pull orchestration strategy based on the longest chain) algorithm. The COLE algorithm (the method of the present application) has less loading time. The COLE-R algorithm is an ablation experiment of the method of the present application (using randomly assigned image layers). Figure 7 In the figure, R is the concurrency and Extraction Times is the extraction time (loading time).

[0123] For python:3.9.3, the loading time using the native Containerd strategy is 8.65s, the loading time using the LOPO algorithm at a concurrency of 2 is 7.78s, the loading time using the COLE algorithm is 5.90s, and the loading time using the COLE-R algorithm is 7.91s. The other information in the accompanying drawings can be understood with reference to this description and will not be repeated here. It can be seen that the image loading method (COLE algorithm) in this application can complete the image loading faster and has higher dumpling loading efficiency.

[0124] Figure 8 The figure shows the loading acceleration ratio of the image loading method in this application compared with the native Containerd strategy for the six popular images at different concurrency levels. Figure 8 In the figure, Speedup Ratio is the acceleration ratio and Concurrency Level is the concurrency level.

[0125] It can be seen that the image loading method in this application has faster loading speed and loading efficiency than the native Containerd strategy in image loading.

[0126] Based on the same inventive concept, the present application also provides an image loading device, as described in the following embodiment. Since the principle of solving the problem in the device embodiment is similar to that in the above method embodiment, the implementation of the device embodiment can refer to the implementation of the above method embodiment, and the repeated parts will not be repeated.

[0127] Figure 5 A schematic diagram of an image loading device in an embodiment of the present application is shown as follows: Figure 5As shown, the device is applied to a local device, in which a Containerd system is deployed, including: an acquisition module 51 for acquiring an image package, the image package including a configuration file and multiple image layers, the configuration file including a manifest.json file and a config.json file; a configuration module 52 for reading metadata of each image layer from the configuration file; reading the information of the Layers list from the manifest.json file to obtain the file name of each image layer, and separating the multiple image layers from the configuration file according to the file name of each image layer; extracting the diffID of each image layer from the rootfs / diff_ids list of the config.json file; determining the image layer of each image layer according to the diffID of each image layer. chainID, snapshot metadata corresponding to each chainID, and SHA256 layer sequence relationship chain; write the metadata, chainID of each image layer, snapshot metadata, and SHA256 layer sequence relationship chain into the metadata database of the Containerd system to configure the image loading environment in the Containerd system; create a snapshot directory for loading image layers in the Containerd system, the snapshot directory includes multiple subdirectories, and the subdirectories correspond to the image layers one by one; a loading strategy determination module 53 is used to determine the loading strategy of multiple image layers; a loading module 54 is used to start multiple goroutines, and use multiple goroutines to load multiple image layers according to the loading strategy, so as to load each image layer into the corresponding subdirectory.

[0128] In some embodiments, the configuration module 52 is configured to calculate the chainID of the image layer according to the diffID of the image layer according to the following formula:

[0129]

[0130] Among them, chainID i is the chainID of the i-th image layer, sha256(.) is the function corresponding to the SHA256 algorithm, diffID i is the diffID of the i-th image layer.

[0131] In some embodiments, the configuration file also includes an index.json file, and the metadata includes tag metadata and timestamp metadata; the configuration module 52 is used to read the tag metadata of each image layer from the index.json file, the manifest.json file, and the config.json file; and read the timestamp metadata of the image from the index.json file.

[0132] In some embodiments, the metadata database includes meta.db and metadata.db; the configuration module 52 is used to write the tag metadata into the v1 / default / content / blob sub-bucket of meta.db; write the timestamp metadata into the v1 / default / images sub-bucket of meta.db; write the snapshot metadata into the v1 / default / snapshotss / overlayfs sub-bucket of meta.db; and write the SHA256 hierarchical relationship chain into the v1 / snapshotss sub-bucket of metadata.db.

[0133] In some embodiments, the loading strategy determination module 53 is used to determine the loading time of each image layer according to Formula 2; determine to start multiple goroutines for loading the image layer based on the available processing resources of the local device, and the number of multiple goroutines is N; based on the loading time and the number of goroutines, construct a MILP problem, and the objective function of the MILP problem is Formula 3; T max It is expressed as formula 4; the constraints of the MILP problem include allocation constraints and time constraints. The allocation constraint is expressed as formula 5; the time constraint is expressed as the following formula 6; Based on the Benders algorithm, the MILP problem is decomposed into the main problem and sub-problems, and iterative calculations are performed until the preset stopping condition is met. As a loading strategy, the main problem is to minimize T max To solve the allocation scheme of multiple image layers on multiple goroutines The subproblem is to calculate the current T below max And generate cutting constraints.

[0134] In some embodiments, the configuration module 52 is used to create a corresponding subdirectory for each image layer in the snapshots directory of the Containerd system according to the incremental numbering to obtain the multiple subdirectories.

[0135] By extracting the file name and diffID of each image layer from the configuration file of the image package, calculating the chainID, and writing the relevant metadata into the metadata database of the Containerd system, the restrictions on the dependencies between image layers are removed and the loading of each image layer is decoupled, so that each image layer can be loaded separately. Then, by starting multiple goroutines to load the multiple image layers, parallel loading of image layers in the Containerd system is achieved. Compared with the serial loading of image layers in the Containerd system, the time required for image loading is greatly reduced and the efficiency of image loading is improved.

[0136] Those skilled in the art will appreciate that various aspects of the present application can be implemented as systems, methods, or program products. Therefore, various aspects of the present application can be specifically implemented in the following forms: a complete hardware implementation, a complete software implementation (including firmware, microcode, etc.), or an implementation that combines hardware and software aspects, which may be collectively referred to herein as a "circuit," "module," or "system."

[0137] Refer to the following Figure 6 hereinafter, an electronic device 600 according to this embodiment of the present application is described. Figure 6 The electronic device 600 shown is merely an example and should not limit the functions and scope of use of the embodiments of the present application.

[0138] like Figure 6 As shown, electronic device 600 is implemented as a general-purpose computing device. Components of electronic device 600 may include, but are not limited to, the aforementioned at least one processing unit 610, the aforementioned at least one storage unit 620, and a bus 630 connecting different system components (including storage unit 620 and processing unit 610).

[0139] The storage unit stores program code that can be executed by the processing unit 610, causing the processing unit 610 to perform the steps of various exemplary embodiments of the present application described in the "Exemplary Method" section above. For example, the processing unit 610 can perform the following steps of the above method embodiment: S401 to S408.

[0140] The storage unit 620 may include a readable medium in the form of a volatile storage unit, such as a random access memory unit (RAM) 6201 and / or a cache memory unit 6202 , and may further include a read-only memory unit (ROM) 6203 .

[0141] The storage unit 620 may also include a program / utility 6204 having a set (at least one) of program modules 6205, such program modules 6205 including but not limited to: an operating system, one or more application programs, other program modules, and program data, each of which or some combination may include an implementation of a network environment.

[0142] Bus 630 may represent one or more of several types of bus structures, including a memory bus or memory controller, a peripheral bus, an accelerated graphics port, a processing unit, or a local bus using any of a variety of bus architectures.

[0143] The electronic device 600 can also communicate with one or more external devices 640 (e.g., a keyboard, a pointing device, a Bluetooth device, etc.), one or more devices that enable a user to interact with the electronic device 600, and / or any device that enables the electronic device 600 to communicate with one or more other computing devices (e.g., a router, a modem, etc.). Such communication can occur via an input / output (I / O) interface 650. Furthermore, the electronic device 600 can communicate with one or more networks (e.g., a local area network (LAN), a wide area network (WAN), and / or a public network such as the Internet) via a network adapter 660. As shown, the network adapter 660 communicates with other modules of the electronic device 600 via a bus 630. It should be understood that, although not shown in the figure, other hardware and / or software modules can be used in conjunction with the electronic device 600, including but not limited to microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.

[0144] Through the description of the above embodiments, it is easy for those skilled in the art to understand that the example embodiments described herein can be implemented by software or by combining software with necessary hardware. Therefore, the technical solution according to the embodiments of the present application can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (which can be a CD-ROM, a USB flash drive, a mobile hard disk, etc.) or on a network, and includes several instructions to enable a computing device (which can be a personal computer, a server, a terminal device, or a network device, etc.) to execute the method according to the embodiments of the present application.

[0145] In particular, according to an embodiment of the present application, the process described above with reference to the flowchart may be implemented as a computer program product, which includes: a computer program, which implements the above-mentioned image loading method when executed by a processor.

[0146] In an exemplary embodiment of the present application, a computer-readable storage medium is further provided, which may be a readable signal medium or a readable storage medium. The computer-readable storage medium stores a program product capable of implementing the above-mentioned method of the present application.

[0147] In some possible implementations, various aspects of the present application may also be implemented in the form of a program product, which includes program code. When the program product is run on a terminal device, the program code is used to enable the terminal device to execute the steps of various exemplary implementations of the present application described in the above "Exemplary Method" section of this specification.

[0148] More specific examples of computer-readable storage media in the present application may include, but are not limited to, an electrical connection having one or more conductors, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0149] In this application, a computer-readable storage medium may include a data signal transmitted in baseband or as part of a carrier wave, which carries readable program code. Such a transmitted data signal may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A readable signal medium may also be any readable medium other than a readable storage medium that can transmit, propagate, or transfer a program for use by or in conjunction with an instruction execution system, apparatus, or device.

[0150] Alternatively, the program code contained on the computer-readable storage medium may be transmitted using any appropriate medium, including but not limited to wireless, wired, optical cable, RF, etc., or any suitable combination thereof.

[0151] In a specific implementation, the program code for performing the operations of the present application can be written in any combination of one or more programming languages, including object-oriented programming languages such as Java, C++, etc., and conventional procedural programming languages such as "C" or similar programming languages. The program code can be executed entirely on the user computing device, partially on the user device, as a separate software package, partially on the user computing device and partially on a remote computing device, or entirely on a remote computing device or server. In the case of a remote computing device, the remote computing device can be connected to the user computing device through any type of network, including a local area network (LAN) or a wide area network (WAN), or can be connected to an external computing device (for example, using an Internet service provider to connect via the Internet).

[0152] It should be noted that, although several modules or units of the device for action execution are mentioned in the above detailed description, this division is not mandatory. In fact, according to the embodiment of the application, the features and functions of two or more modules or units described above can be concretized in one module or unit. On the contrary, the features and functions of one module or unit described above can be further divided into multiple modules or units to be concretized.

[0153] Furthermore, although the steps of the method of the present application are described in a particular order in the accompanying drawings, this does not require or imply that the steps must be performed in this particular order, or that all steps shown must be performed to achieve the desired results. Additionally or alternatively, some steps may be omitted, multiple steps may be combined into one step, and / or one step may be decomposed into multiple steps.

[0154] Through the description of the above embodiments, it is easy for those skilled in the art to understand that the example embodiments described here can be implemented by software or by combining software with necessary hardware. Therefore, the technical solution according to the embodiments of the present application can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (which can be a CD-ROM, a USB flash drive, a mobile hard disk, etc.) or on a network, and includes several instructions to enable a computing device (which can be a personal computer, a server, a mobile terminal, or a network device, etc.) to execute the method according to the embodiments of the present application.

[0155] Those skilled in the art will readily appreciate other embodiments of the present invention after considering the specification and practicing the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of the present invention that follow the general principles of the present invention and include common knowledge or customary techniques in the art not disclosed herein. The description and examples are to be considered merely as exemplary, and the true scope and spirit of the present application are indicated by the appended claims.

Claims

1. A method for image loading, characterized in that: Applied to local devices where the Containerd system is deployed, including: Obtain an image package, which includes a configuration file and multiple image layers. The configuration file includes a manifest.json file and a config.json file. Read metadata of each image layer from the configuration file; Read the information of the Layers list from the manifest.json file to obtain the file name of each image layer, and separate the multiple image layers from the configuration file according to the file name of each image layer; Extract the diffID of each image layer from the rootfs / diff_ids list in the config.json file; Based on the diffID of each image layer, determine the chainID of each image layer, the snapshot metadata corresponding to each chainID, and the SHA256 layer sequence relationship chain; Write the metadata, the chainID of each image layer, the snapshot metadata, and the SHA256 layer sequence relationship chain into the metadata database of the Containerd system to configure the image loading environment in the Containerd system; Determining a loading strategy for the multiple image layers; Create a snapshot directory for loading the image layer in the Containerd system, wherein the snapshot directory includes multiple subdirectories, and each subdirectory corresponds to the image layer in a one-to-one manner; Start multiple goroutines, and use the multiple goroutines to load the multiple image layers according to the loading strategy, so as to load each image layer into a corresponding subdirectory.

2. The image loading method according to claim 1, characterized in that: Determining the chainID of each image layer according to the diffID of each image layer includes: According to the diffID of the image layer, the chainID of the image layer is calculated according to the following formula: Among them, chainID i is the chainID of the i-th image layer, SHA256(.) is the function corresponding to the SHA256 algorithm, diffID i is the diffID of the i image layer.

3. The image loading method according to claim 1, wherein: The configuration file also includes an index.json file, and the metadata includes tag metadata and timestamp metadata; The step of reading metadata of each image layer from the configuration file includes: Read the tag metadata of each image layer from the index.json file, the manifest.json file, and the config.json file; Read the image's timestamp metadata from the index.json file.

4. The image loading method according to claim 3, characterized in that: The metadata database includes meta.db and metadata.db; Writing the metadata, the chainID of each image layer, the snapshot metadata, and the SHA256 layer sequence relationship chain into the metadata database of the Containerd system includes: Write the tag metadata into the v1 / default / content / blob sub-bucket of the meta.db; Write the timestamp metadata into the v1 / default / images sub-bucket of the meta.db; Write the snapshot metadata to the v1 / default / snapshotss / overlayfs sub-bucket of the meta.db; Write the SHA256 hierarchical relationship chain into the v1 / snapshotss sub-bucket of the metadata.db.

5. The image loading method according to any one of claims 1 to 4, characterized in that: Determining the loading strategy of the multiple image layers includes: The loading time of each image layer is determined according to the following formula: Among them, B p (i) is the size of the i-th image layer before decompression, μ(i) is the decompression rate of the i-th image layer, and W is the storage bandwidth of the local device; Determine, based on available processing resources of the local device, to start the multiple goroutines for loading the image layer, where the number of the multiple goroutines is N; Based on the loading time and the multiple goroutines, a mixed integer linear programming (MILP) problem is constructed. The objective function of the MILP problem is: min(T max ) Among them, min(.) is the minimum value function; T max T is the maximum loading time of loading the multiple image layers in the multiple goroutines. max It is expressed as the following formula: max(.) is the maximum value function; r is used to represent the rth goroutine among N goroutines; Equal to 0 or 1, if the i-th image layer is allocated on the r-th goroutine, Equal to 1, if the i-th image layer is not allocated on the r-th goroutine, =0; L is the number of mirror layers in the plurality of mirror layers; The constraints of the MILP problem include allocation constraints and time constraints. The allocation constraints are expressed as the following formula: The time constraint is expressed as the following formula: Based on the Benders algorithm, the MILP problem is decomposed into a main problem and sub-problems, and iterative calculation is performed until the preset stopping condition is met. As the loading strategy, the main problem is to minimize T max To solve the allocation scheme of the multiple image layers on the multiple goroutines The subproblem is to calculate the current T below max And generate cutting constraints.

6. The image loading method according to claim 1, wherein: Creating a snapshot directory for loading the image layer in the Containerd system includes: Under the snapshots directory of the Containerd system, a corresponding subdirectory is created for each image layer according to the incremental number to obtain the multiple subdirectories.

7. An image loading device, characterized in that: Applied to local devices where the Containerd system is deployed, including: An acquisition module is used to obtain an image package, wherein the image package includes a configuration file and multiple image layers, wherein the configuration file includes a manifest.json file and a config.json file; A configuration module is configured to read the metadata of each image layer from the configuration file; read the information of the Layers list from the manifest.json file to obtain the file name of each image layer, and separate the multiple image layers from the configuration file according to the file name of each image layer; extract the diffID of each image layer from the rootfs / diff_ids list of the config.json file; determine the chainID of each image layer, the snapshot metadata corresponding to each chainID, and the SHA256 layer sequence relationship chain according to the diffID of each image layer; write the metadata, the chainID of each image layer, the snapshot metadata, and the SHA256 layer sequence relationship chain into the metadata database of the Containerd system to configure the image loading environment in the Containerd system; create a snapshot directory for loading image layers in the Containerd system, the snapshot directory including multiple subdirectories, and the subdirectories correspond one to one to the image layers; A loading strategy determination module, configured to determine the loading strategies for the multiple image layers; The loading module is used to start multiple goroutines and use the multiple goroutines to load the multiple image layers according to the loading strategy, so as to load each image layer into a corresponding subdirectory.

8. An electronic device, characterized in that: include: processor; as well as a memory for storing executable instructions of the processor; The processor is configured to execute the image loading method according to any one of claims 1 to 6 by executing the executable instructions.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the image loading method according to any one of claims 1 to 6 is implemented.

10. A computer program product, comprising a computer program, wherein when the computer program is executed by a processor, the image loading method according to any one of claims 1 to 6 is implemented.

Images