Docker container mirror image management method

Abstract

The invention provides a scheme for efficiently managing and pulling a container mirror image. In the scheme, on the mirror image warehouse side, the mirror image file storage architecture is deeply improved, related data of the mirror image files are carefully sorted by taking data blocks as units, and repeated data blocks are accurately identified and removed, so that the storage pressure of the mirror image warehouse is greatly reduced, and storage resources are more reasonably utilized. On the basis, the technical advantages of the Hash bucket and the Bloom filter are ingeniously fused, an efficient fingerprint retrieval structure is constructed, and the retrieval efficiency of the data blocks is remarkably improved; on the client side, the container mirror image is pulled from the mirror image warehouse with the data block as the unit to start the container, the accurate pulling mode ingeniously avoids transmission of unnecessary data, a shortcut is opened up for data transmission, network resource occupation is effectively reduced, the container mirror image pulling efficiency is practically improved, and the method is suitable for popularization and application. And the progress of the container technology in data processing and application is comprehensively promoted.

Description

Technical Field

[0001] The present invention relates to the fields of cloud computing and microservices, and more specifically to the field of container image management, and more specifically to a Docker container image management method. Background Art

[0002] In modern cloud computing and microservice architecture, Docker container technology has become the main tool for rapid application deployment, isolation and migration. Among them, Docker images, containers and image repositories are the three basic components of Docker container technology and are the key elements for container image management and transmission. Among them, Docker images are lightweight, executable software packages, image repositories are servers for managing container images, and containers are instances of container images running on clients. In traditional Docker container technology, image repositories use image hierarchical structures to store container images. Each layer represents a snapshot of a file system. These layers together constitute a complete container image system. The hierarchical storage design makes Docker images reusable. The same image layer can be shared by multiple images, saving storage space and reducing the image transmission time, thereby improving the image transmission efficiency. However, the existing container images are managed using a hierarchical structure. On the one hand, the efficiency of accessing deeper data is low. On the other hand, the image repository has a large storage pressure due to the large amount of redundant data between layers in the image hierarchical structure. The client pulls image files in units of image layers, which seriously affects the transmission efficiency of image files. Existing Docker container technology generally focuses on optimizing and adapting the image layered architecture to alleviate the storage pressure of the image repository and improve the transmission efficiency of image files. It does not use more fine-grained data units to manage container images, making it difficult for existing technology to completely solve related problems from the underlying architecture.

[0003] In summary, Docker container technology faces many difficulties in existing applications. Traditional Docker container technology uses a hierarchical image structure to manage container images, which results in a large amount of redundant data between layers, leading to high storage pressure on the image warehouse. Although some researchers have optimized and improved the hierarchical image architecture, they have not been able to go deep into the level of refined management of data units, making it impossible to resolve storage pressure and transmission bottlenecks from the root. Therefore, there is an urgent need for a Docker container system that can greatly improve the storage and transmission efficiency of container images.

[0004] It should be noted that this background technology is only used to introduce the relevant information of the present invention to help understand the technical solution of the present invention, but it does not mean that the relevant information is necessarily the prior art. If there is no evidence that the relevant information has been disclosed before the application date of the present invention, the relevant information shall not be regarded as the prior art. Summary of the invention

[0005] Therefore, the purpose of the present invention is to overcome the defects of the above-mentioned prior art and provide a Docker container image management method.

[0006] The objective of the present invention is achieved through the following technical solutions:

[0007] According to a first aspect of the present invention, a Docker container system is proposed, the system comprising an image repository and a client, the image repository being used to manage data related to the container image in units of data blocks and to search for data blocks based on fingerprint information sent by the client and return the data blocks to the client, the system comprising: a data block management module being used to manage data blocks related to the container image and being configured to manage the data blocks in the following manner: obtaining multiple existing container images; dividing each container image into multiple data blocks according to a preset size, and configuring fingerprint information and metadata for recording the data block information for each data block; performing a deduplication operation on all data blocks based on the fingerprint information of the data blocks to delete duplicate data blocks; a data block retrieval module being used to search for corresponding data blocks based on the fingerprint information; the client being used to send multiple fingerprint information required for pulling the container image to the image repository, splicing the multiple data blocks returned by the image repository with its own existing data blocks to obtain the container image, and starting the container based on the container image.

[0008] Preferably, the preset size is any size within the range of 2KB-64KB.

[0009] Preferably, the metadata corresponding to each data block includes fingerprint information, storage address and number of repetitions of the data block, wherein the fingerprint information in each metadata is the hash value of the data block corresponding to the metadata, the storage address in each metadata is the storage address of the data block corresponding to the metadata in the image repository, and the number of repetitions in each metadata is the number of data blocks that are consistent with the data block corresponding to the metadata.

[0010] Preferably, the data block management module is configured to: calculate a hash value of each data block using a Gear hash function based on data in each data block, wherein data blocks with consistent data have the same hash value.

[0011] Preferably, the data block retrieval module is configured with a data block retrieval library containing metadata of all data blocks, wherein the data block retrieval library is configured with: a low-frequency metadata cache area, including multiple hash buckets, wherein each hash bucket corresponds to a specific fingerprint information range and is dedicated to storing metadata of all data blocks whose fingerprint information is within the specific fingerprint information range and whose number of repetitions is lower than a preset threshold; a high-frequency metadata cache area, including multiple hash buckets, wherein each hash bucket corresponds to a specific fingerprint information range and is dedicated to storing metadata of all data blocks whose fingerprint information is within the specific fingerprint information range and whose number of repetitions is higher than or equal to a preset threshold; a fingerprint information cache table, used to cache fingerprint information to be queried; a dynamic Bloom filter unit, including multiple Bloom filters, which is used to determine whether metadata of a data block corresponding to the fingerprint information exists in the data block retrieval library according to given fingerprint information; a metadata temporary storage area, used to temporarily cache metadata to be saved, and regularly save the cached metadata to the corresponding hash bucket in the low-frequency metadata cache area; a hash bucket address table, used to determine the fingerprint information range where the specified fingerprint information is located, and output the address of the hash bucket corresponding to the fingerprint information range in the low-frequency metadata cache area and the high-frequency metadata cache area.

[0012] Preferably, the image warehouse is configured with an external memory and an internal memory, wherein the low-frequency metadata cache area is configured in the external memory of the image warehouse, and the high-frequency metadata cache area, the fingerprint information cache table, the dynamic Bloom filter unit, the metadata temporary storage area and the hash bucket address table are configured in the internal memory of the image warehouse.

[0013] Preferably, the dynamic Bloom filter unit is configured to: add a new Bloom filter whenever the total amount of fingerprint information exceeds the capacity of the current dynamic Bloom filter unit.

[0014] Preferably, the data block retrieval library is configured to be updated in the following manner: the fingerprint information of each data block is stored in a fingerprint information cache table, and the following operations are performed on each fingerprint information in the fingerprint information cache table: Step S1: Use a dynamic Bloom filter unit to determine whether there is metadata corresponding to the current fingerprint information in the data block retrieval library. If so, execute step S2; if not, execute step S4; Step S2: Search the addresses of all hash buckets corresponding to the current fingerprint information through a hash bucket address table, and confirm whether the hash buckets at these addresses have metadata corresponding to the current fingerprint information. If so, execute step S3; if not, execute step S4; Step S3: Add 1 to the number of repetitions of the current fingerprint information in the metadata corresponding to the hash bucket, and delete the data block corresponding to the current fingerprint information; Step S4: Add the metadata corresponding to the current fingerprint information to the metadata temporary storage area.

[0015] Preferably, the data block retrieval module is configured to: periodically scan all metadata in the data block retrieval library at preset time intervals, migrate the metadata of data blocks with a repetition number greater than or equal to a preset threshold to the corresponding hash bucket in the high-frequency metadata cache area, and migrate the metadata of data blocks with a repetition number lower than the preset threshold to the corresponding hash bucket in the low-frequency metadata cache area.

[0016] Preferably, when the client needs to start a container, the image repository and the client are configured to interact in the following manner: the client sends a pull request for a specified container image to the image repository; the image repository responds to the client's container image pull request and sends a fingerprint information list to the client, wherein the fingerprint information list includes fingerprint information of all data blocks related to the specified container image; the client searches for fingerprint information of all missing data blocks based on the fingerprint information list sent by the image repository and its own existing data blocks, and feeds back the fingerprint information of all missing data blocks found to the image repository; the image repository searches for the data blocks corresponding to each fingerprint information fed back by the client through a data block retrieval module, and returns all the found data blocks to the corresponding client; the client splices the data blocks returned by the image repository and its own existing data blocks to obtain the container image, and starts the container based on the container image.

[0017] Compared with the prior art, the advantages of the present invention are:

[0018] The present invention proposes a new solution to achieve efficient management and retrieval of container images. On the image warehouse side, the storage architecture is improved, and the container image-related data is managed and deduplicated in units of data blocks. A fingerprint retrieval system is built in combination with hash buckets and Bloom filters to relieve storage pressure and improve retrieval efficiency. On the client side, the image is pulled in units of data blocks to start the container, avoiding redundant data pulling, effectively enhancing the efficiency of image pulling, and solving existing problems in an all-round way. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] The embodiments of the present invention are further described below with reference to the accompanying drawings, in which:

[0020] Figure 1 A schematic diagram of a Docker image life cycle according to an embodiment of the present invention;

[0021] Figure 2 A schematic diagram of mirror layering according to an embodiment of the present invention;

[0022] Figure 3 A schematic diagram showing a comparison of MySQL image upgrades according to an embodiment of the present invention;

[0023] Figure 4 A schematic diagram of a B+ tree fingerprint index structure according to an embodiment of the present invention;

[0024] Figure 5 A schematic diagram of the structure of a data block retrieval library according to an embodiment of the present invention;

[0025] Figure 6 A schematic diagram of the steps of updating the data block retrieval library according to an embodiment of the present invention;

[0026] Figure 7 A schematic diagram of steps for updating a metadata storage location according to an embodiment of the present invention;

[0027] Figure 8 A schematic diagram of a data block retrieval step according to an embodiment of the present invention;

[0028] Fig. 9 A schematic diagram of the structure of a Docker container system according to an embodiment of the present invention;

[0029] Fig.10 A schematic diagram of a process of pulling an image by a client according to an embodiment of the present invention;

[0030] Fig.11 The figure is a schematic diagram of a process of pulling an image by a client according to an embodiment of the present invention. DETAILED DESCRIPTION

[0031] In order to make the purpose, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail by specific embodiments below. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.

[0032] Before specifically introducing the embodiments of the present invention, in order to facilitate understanding of the Docker container technology, a brief introduction to the existing Docker container technology is first given.

[0033] Docker container technology covers three core basic concepts, namely images, containers and image repositories. Images can be regarded as a special file system that will not change after the container is built. Image construction follows a layered construction mechanism, and construction operations are performed layer by layer during the construction process. The previous layer is the basis for building subsequent layers, and each layer remains fixed after the construction is completed, and only the current layer will change. In addition, container images can also be built with the help of Dockerfile. After executing the container image, a Docker container can be obtained, and the Docker container can also be packaged and converted into an image. The life cycle of a Docker image is as follows Figure 1As shown in the figure, Docker images are static in nature, while containers are dynamic. Docker containers are entities that images correspond to at runtime. Containers are essentially processes, which are different from host processes. Container processes run in their own independent namespaces. The life cycle of the container layer is consistent with that of the container. When the container stops running, the container layer also stops existing. The Docker registry (image warehouse) is responsible for providing centralized storage and distribution services for images, which is similar to the code warehouse in the development process.

[0034] The Docker image in the prior art is constructed by a series of layered storage, such as Figure 2 As shown in the figure, it shows the hierarchical structure of Docker images. Each Docker image is formed by multiple layers, where each image layer is represented as a snapshot of a file system, and all image layers together constitute a complete image file system. Each image layer in the image layer structure has a read-only attribute. During the operation of the container, an additional writable layer will be added to each image layer to adapt to the container's write operation requirements. In this way, multiple containers can share the same read-only image layer, and each has an independent writable layer, thereby achieving a high degree of resource sharing and effective isolation. The design of layered storage makes Docker images reusable, and the same image layer can be shared by multiple images. Taking the layered storage of MySQL images as an example, Figure 3 As shown in the figure, the image layer changes when the application MySQL is upgraded from version 8.2.0 to version 8.3.0. The image size increases from 619MB to 632MB. Among them, only the bottom image layer has not changed. If the local cache has the corresponding image layer of oraclelinux:8-slim, then this layer can be reused.

[0035] Although the layered storage design makes Docker images reusable and the same image layer can be shared by multiple images, since the image layer is read-only, there is a large amount of identical data between different image layers, and it is impossible to achieve finer-grained resource sharing. The redundant data between different image layers cannot be eliminated by sharing images in a layered manner. The inventors analyzed the storage properties of existing image data sets (such as the size and compression ratio distribution of layers and images, directory and file distribution, and Docker-specific properties) and found that approximately 90% of the image layers are referenced by only one image, and another 5% of the image layers are referenced by two images. In addition, through a more fine-grained analysis of the image layers, it was found that more than 99.4% of the files have more than one copy, approximately 50% of the files have 4 copies, and 90% of the files have less than 10 copies. It can be seen that although image layering can reuse data to a certain extent, there is still a lot of data redundancy between image layers. Still with Figure 3 Taking the MySQL image layered storage as an example, the local cache has the image layer corresponding to oraclelinux:8-slim, but this layer only accounts for 19% of the total image volume. From layer2 to layer4, the data changes in each layer are very small. Taking layer4 as an example, the difference between the two images is only the different versions of individual software packages. In addition, the 8.3.0 version only has one more pygments-github-lexers 0.0.5 package. Even if there are only very minor changes in the two image layers, the image layers need to be pulled again. Therefore, even if the image layered reuse method is adopted, there is still a lot of redundant data in certain scenarios.

[0036] The inventors have found through in-depth research that the introduction of deduplication technology into the field of container image management can effectively reduce the redundant data in the image file, thereby improving the performance of image warehouse storage and management of container images. The so-called deduplication technology is a technical means to achieve streamlined data scale and optimize data storage efficiency by identifying and removing duplicate data at the file level. In this technology, fingerprint information index plays a very important role. By comparing fingerprint information, duplicate data can be quickly and efficiently identified. When searching for duplicate data blocks, the system matches the duplicate data through the fingerprint information index, and associates the duplicate new data with the stored data that is the same as the new data, thereby establishing a reference relationship, so that the new data does not need to be stored again. Compared with storing complete data information, only storing fingerprint information index can save a lot of storage space.

[0037] The existing fingerprint information index is an index structure built based on the B+ tree to accelerate fingerprint information retrieval. The B+ tree is a multi-branch balanced search tree, which is usually used in databases and file systems to accelerate file data queries. The B+ tree has the characteristics of ensuring data stability and order, and presents logarithmic time complexity during data insertion and modification operations. Generally speaking, the data of the B+ tree is stored in the leaf nodes. For global data scanning operations, it is only necessary to implement linear scanning operations on the leaf nodes, and there is no need to start the query from the root node, thereby improving the scanning efficiency of the global data. If the order of this tree is m, then each node except the root node covers at least m / 2 elements and at most m-1 elements. For any node, the number of pointers is at most m. The setting status of the subnode information of the general B+ tree is as follows: Figure 4As shown in the figure, it records the hash value, actual storage location, data size, offset and other information corresponding to each data. The hash value is used as the keyword of the node and is generally calculated using the SHA-1 and SHA-256 algorithms. However, in the process of building a B+ tree, it is necessary to frequently insert and delete nodes. Such operations are prone to cause node splitting, merging and rotation, resulting in continuous accumulation of system overhead, which ultimately has an adverse effect on the indexing performance of the fingerprint information retrieval architecture. In addition, if the amount of fingerprint information data is too large and the B+ tree needs to be rebuilt, it often takes a high amount of time and resource costs. In extreme cases, it may even endanger the availability of the system, and is not suitable for scenarios with massive data. In order to solve the performance bottleneck problem encountered by the index structure built based on B+ tree when processing massive data, some researchers proposed to organize and search the fingerprint information index by configuring a fixed number of Bloom filters. The Bloom filter only needs to calculate a few simple hash functions and check whether the corresponding bit is set to complete the query operation. Compared with the tree-based index structure, it avoids the overhead generated by tree traversal, so the query speed is very fast. In addition, the Bloom filter also supports efficient incremental updates, which is very suitable for the dynamically changing fingerprint information index scenario. However, the existing deduplication technology uses a fixed number of Bloom filters for query screening, which has limitations in load, accuracy and overhead. On the one hand, when the amount of data increases, the load of the Bloom filter increases, resulting in reduced query efficiency. On the other hand, setting too many Bloom filters will lead to resource waste, which is not conducive to resource utilization and cost control. Therefore, if deduplication technology is to be applied to container image management, it is also necessary to improve the existing deduplication technology to adapt to the massive data management needs of container images and the query efficiency and resource control needs when data is continuously added.

[0038] In summary, traditional Docker container technology uses a layered image structure to manage container images. This method causes a large amount of redundant data between layers, which in turn causes the image warehouse to be under great storage pressure. Although some researchers have optimized and improved the layered image architecture, they have not yet gone deep into the refined management level of data units, so they cannot fundamentally eliminate storage pressure and transmission bottlenecks. In addition, data deduplication technology can handle the image management problems of existing Docker container technology to a certain extent. However, since the current data deduplication technology is generally built on the fingerprint information retrieval architecture of the B+ tree, and only uses a fixed number of Bloom filters to perform query and screening operations, in the image file management scenario with extremely large data volumes, the cost of building a fingerprint information retrieval architecture is high, and the query accuracy is limited. Therefore, the existing data deduplication technology cannot be applied to the container image management scenario of the present invention.

[0039] In order to overcome the problems faced by the existing Docker container technology, the present invention proposes a solution for efficiently managing and pulling container images. In this solution, the image file storage architecture of the image warehouse is improved, and the associated data of the image file is managed with data blocks as the basic unit, and deduplication operations are performed on the data blocks to reduce the storage pressure of the image warehouse. On this premise, the hash bucket and Bloom filter are integrated to build a fingerprint retrieval system, thereby improving the retrieval efficiency of the data block. In addition, the client pulls the container image from the image warehouse with the data block as the unit to start the container, avoiding pulling redundant data, thereby effectively improving the efficiency of container image pulling.

[0040] In order to better understand the present invention, the present invention is described in detail below in conjunction with specific embodiments.

[0041] According to an embodiment of the present invention, in the image warehouse, the present invention uses data blocks to store image file data. Specifically, all acquired container images are divided into multiple data blocks of preset sizes to manage container images in units of data blocks. The preset size is any size within the range of 2KB-64KB. It should be understood that this size range is a preferred embodiment of the present solution, not an absolute setting, and it can be flexibly configured according to actual conditions. Compared with the existing hierarchical structure management of image files, dividing the container image into data blocks for management has a smaller granularity of data segmentation, which enables more accurate management of each data block during data storage and transmission. In addition, when an abnormality occurs in a data block, it can also be more accurately located and repaired, thereby reducing the impact on the entire container image, thereby improving the overall reliability and fault tolerance of the system.

[0042] According to one embodiment of the present invention, the image repository also needs to configure fingerprint information for each data block for deleting duplicate data blocks. Specifically, the Gear hash algorithm (Gear hash function) is used to calculate the hash value of each data block based on the data in each data block, and this is used as the fingerprint information of each data block, wherein the hash values ​​calculated by the Gear hash algorithm for data blocks with consistent data are the same. The Gear hash algorithm is an efficient hash function that can be used for a variety of purposes such as data segmentation and data verification. It should be understood that the Gear hash algorithm is a well-known technology for those skilled in the art and will not be described in detail here.

[0043] According to one embodiment of the present invention, in the image warehouse, metadata describing the information of each data block is configured for each data block based on the relevant information of each data block so as to quickly obtain the relevant information of each data block. Each metadata contains the fingerprint information, storage address, and number of repetitions of the data block corresponding to the metadata. Specifically, the fingerprint information in each metadata is the hash value of the data block corresponding to the metadata, the storage address in each metadata is the storage address of the data block corresponding to the metadata in the image warehouse, and the number of repetitions in each metadata is the number of data blocks consistent with the data of the data block corresponding to the metadata. Configuring metadata for each data block can organize and manage massive data blocks more efficiently, and facilitates the rapid positioning of target data blocks in operations such as retrieval, calling, and data update. In addition, the key information covered by the metadata cooperates with each other, providing a solid foundation for building a complete data block management system, making the entire Docker container system more organized, efficient, and stable when processing container image-related data, and enhancing the overall operating efficiency of the system.

[0044] The present invention divides the image file into many data blocks for management. While this improvement brings many innovative possibilities, it also raises severe challenges in the management of massive data blocks, especially in terms of ensuring the efficiency of data block queries and accurately grasping resource control. According to one embodiment of the present invention, based on the inspiration of deduplication technology, the present invention improves the existing deduplication technology by combining hash buckets and Bloom filters to collaboratively construct a data block retrieval library (data block retrieval structure) for storing relevant information of all data blocks, so as to improve data block retrieval efficiency and resource utilization. Figure 5As shown in the figure, the structure of the data block retrieval library is shown. The data block retrieval library includes multiple components, which are configured in the internal memory or external memory of the image warehouse. Specifically, a low-frequency metadata cache is configured in the external memory of the image warehouse, and a high-frequency metadata cache, a fingerprint information cache table, a dynamic Bloom filter unit, a metadata temporary storage area, and a hash bucket address table are configured in the internal memory of the image warehouse. The low-frequency metadata cache includes multiple hash buckets, each hash bucket corresponds to a specific fingerprint information range and is dedicated to storing the metadata of all data blocks whose fingerprint information is within the specific fingerprint information range and the number of repetitions is lower than a preset threshold; the high-frequency metadata cache includes multiple hash buckets, each hash bucket corresponds to a specific fingerprint information range and is dedicated to storing the metadata of all data blocks whose fingerprint information is within the specific fingerprint information range and the number of repetitions is higher than or equal to the preset threshold. The high-frequency metadata cache pre-fetches metadata with a high repetition rate into the memory in advance to improve query efficiency; the fingerprint information cache table is used to Cache the fingerprint information to be queried. Since this solution adopts a more fine-grained data block management container image, the number of data blocks is huge, and the system will generate a large number of fingerprints. The use of a fingerprint information cache table can alleviate the pressure of fingerprint query to a certain extent; the dynamic Bloom filter unit includes multiple Bloom filters, which are used to determine whether the metadata of the data block corresponding to the fingerprint information exists in the data block retrieval library according to the given fingerprint information; the dynamic Bloom filter unit includes multiple Bloom filters, which are used to determine whether the metadata of the data block corresponding to the fingerprint information exists in the data block retrieval library according to the given fingerprint information; the metadata temporary storage area is used to temporarily cache the metadata to be saved, and regularly save the cached metadata to the corresponding hash bucket in the low-frequency metadata cache area, and update the hash bucket address table at the same time; the hash bucket address table is used to cache the mapping relationship between the fingerprint information range and the hash bucket address, and is used to determine the fingerprint information range where the specified fingerprint information is located, and output the address of the hash bucket corresponding to the fingerprint information range in the low-frequency metadata cache area and the high-frequency metadata cache area. The data block retrieval library in this solution is constructed by combining hash buckets and Bloom filters, and the number of Bloom filters is dynamically adjusted according to the scale of fingerprint information to adapt to the massive data management needs of container images and the query efficiency and resource control needs when data is continuously added.

[0045] The existing technology uses a fixed number of Bloom filters to query and filter fingerprint information, which has limitations in load, accuracy and overhead. On the one hand, when the amount of data increases, the load of the Bloom filter increases, causing the system response to slow down; on the other hand, the fixed setting will lead to resource waste, which is not conducive to resource utilization and cost control. In order to solve the problems existing in the existing technology that uses a fixed number of Bloom filters for query and filtering, according to an embodiment of the present invention, this solution dynamically adjusts the number of Bloom filters in the Bloom filter unit based on the number of fingerprints. Specifically, whenever the number of fingerprints exceeds the capacity of the Bloom filter unit, a new Bloom filter will be created, and the mapping relationship between the Bloom filter and the hash bucket will be set at the same time. In addition, according to experimental results, when 5 hash functions are set and 1 million fingerprint data are used, the Bloom filter only needs 25MB of space to reduce the false positive probability to one in 100 million. Therefore, only a small amount of space overhead is required to greatly reduce the false positive probability. This scheme uses the strategy of dynamically adjusting the number of Bloom filters in the Bloom filter unit according to the number of fingerprints, which can effectively alleviate the load pressure of the Bloom filter. At the same time, at the expense of a small amount of storage space, it prevents false positive problems in the Bloom filter, effectively guarantees the accuracy and efficiency of data processing, and builds a more stable and reliable support mechanism for data block retrieval and screening operations.

[0046] In the operating environment of the image warehouse, the container image will be in a state of continuous dynamic change due to various operations such as upgrading and uploading. Given that the accuracy and real-time nature of data are crucial to the stable operation and effective use of the entire system, the data block retrieval library also needs to be closely linked and updated in time with each change of the container image to ensure that the data is always kept up to date. According to one embodiment of the present invention, in the image warehouse, the data block retrieval library (data block retrieval structure) is updated based on the fingerprints and metadata of all data blocks. During the update of the data block retrieval library, duplicate data blocks are removed, such as Figure 6As shown in FIG. 1 , the figure shows the detailed steps of updating the data block retrieval library. Specifically, the fingerprint information of all data blocks is cached in the fingerprint information cache table, and then the fingerprint information is retrieved from the fingerprint information cache table in turn, and the following operations are performed on each fingerprint information: Step M1, use the dynamic Bloom filter unit to query the current fingerprint information to determine whether the current fingerprint information has been processed (that is, whether the metadata corresponding to the fingerprint information exists in the hash bucket in the low-frequency metadata cache area or the high-frequency metadata cache area). If it has been processed, execute step M2; if it has not been processed, execute step M4; Step M2, search for all hash addresses corresponding to the current fingerprint information based on the hash bucket address table. Bucket address, and find out whether there is metadata corresponding to the current fingerprint information in these hash buckets. If so, execute step M3, if not, execute step M4; step M3, determine the data block corresponding to the current fingerprint information as a duplicate data block, delete the data block, do not store the data block, and add 1 to the number of repetitions of the metadata corresponding to the current fingerprint in the hash bucket to update the number of repetitions of the metadata, and process the next fingerprint information; step M4, store the metadata of the data block corresponding to the current fingerprint information into the metadata temporary storage area, and the metadata temporary storage area regularly saves the metadata to the corresponding hash bucket in the low-frequency metadata cache area, and processes the next fingerprint information.

[0047] It should be noted that the more times a data block is repeated, the greater the probability that the data block is searched. According to an embodiment of the present invention, this solution pre-fetches metadata of frequently repeated data blocks into the memory in advance to improve the query efficiency of the data blocks. Specifically, Figure 7 As shown, the image repository periodically scans all metadata at preset time intervals, migrates the metadata of data blocks with a repetition number higher than or equal to a preset threshold to the corresponding hash bucket in the high-frequency metadata cache area of ​​the internal memory, and migrates the metadata of data blocks with a repetition number lower than the preset threshold to the corresponding hash bucket in the low-frequency metadata cache area of ​​the external memory.

[0048] According to an embodiment of the present invention, the image warehouse can retrieve data blocks through the constructed data block retrieval library. Specifically, when the image warehouse needs to search for a data block, the corresponding data block is searched through the data block retrieval library based on the fingerprint information of the data block to be searched, such as Figure 8As shown in the figure, the detailed process of searching for data blocks in the image repository includes: step A1, obtaining the fingerprint information to be searched; step A2, caching the fingerprint information to be searched into the fingerprint information table; step A3, taking out the fingerprint information to be searched from the fingerprint cache table in sequence; step A4, searching the hash bucket address table for the hash bucket corresponding to the current fingerprint information to be searched, if the number of repetitions of the metadata corresponding to the fingerprint information to be searched is higher than or equal to the preset threshold, the hash bucket found is the hash bucket in the high-frequency metadata cache area, otherwise, the hash bucket found is the hash bucket in the low-frequency metadata cache area; step A5, querying the metadata corresponding to the fingerprint information to be searched in the hash bucket; obtaining the data block based on the storage address in the metadata; processing the next fingerprint information to be searched.

[0049] According to an embodiment of the present invention, the present invention proposes a new Docker container system based on the improvement of the image repository, such as Fig. 9 As shown, the figure shows the basic structure of the Docker container system. The Docker container system consists of an image repository and a client. It should be understood that the number of image repositories and clients can be single or multiple. As for the structure of the image repository and the client, on the image repository side, a data block management module and a data block query module are configured to manage and query container images in units of data blocks, wherein the data block management module is mainly used to divide the acquired image file into data blocks and perform deduplication operations on all data blocks, and the data block retrieval module is used to search the corresponding database according to the fingerprint information, and is also responsible for maintaining the data block retrieval library constructed based on the hash bucket and the Bloom filter. The data block retrieval library is used to store the relevant information of all data blocks. On the client side, the client sends the fingerprint information of all required data blocks to the image repository before starting the container, and splices the image file based on the data blocks returned by the image repository and the existing data blocks to start the container.

[0050] In the Docker container system, the image repository adopts a mode of managing image files in units of data blocks. Thanks to this management mode, when the client obtains the container image, it can perform a pull operation from the image repository with the data block as the smallest unit. Compared with the conventional solution of pulling images in units of image layers commonly used in the prior art, the innovative strategy of pulling container images in units of data blocks used in the present invention has significant advantages. It can pull container images with a finer and smaller granularity, accurately locate and obtain the required data, thereby effectively avoiding the problem of a large amount of redundant data transmission that may be caused by the image layer pulling method, greatly reducing unnecessary data traffic consumption, and effectively and effectively improving the overall efficiency of image pulling.

[0051] According to an embodiment of the present invention, in a Docker container system, a client pulls a required container image from an image repository in units of data blocks, splices the container image with its own existing data blocks, and starts a container based on the spliced ​​container image. Specifically, Fig.10 As shown, the figure shows the interaction process between the client and the image repository when the client needs to start a container, including: step B1, the client sends a pull request for the specified container image to the image repository; step B2, the image repository responds to the client's container image pull request and sends a fingerprint information list to the client, wherein the fingerprint information list contains the fingerprint information of all data blocks of the specified container image; step B3, the client searches for the fingerprint information of all missing data blocks based on the fingerprint information list sent by the image repository and its own existing data blocks, and feeds back the fingerprint information of all missing data blocks found to the image repository; step B4, the image repository searches for the data blocks corresponding to each fingerprint information fed back by the client through the data block retrieval module, and returns all the found data blocks to the corresponding client; step B5, the client splices the data blocks returned by the image repository and its own existing data blocks to obtain the container image, and starts the container based on the image.

[0052] According to another embodiment of the present invention, an intermediate proxy is set between the client and the image repository, and the intermediate proxy parses the image request content to determine the data block that the image repository needs to transmit to the corresponding client, and the image repository only needs to find the corresponding data block and send the data block to the corresponding client. Specifically, Fig.11As shown, the figure shows the detailed process of another way for the client to pull the container image, including: step B1, the client sends a pull request for the specified container image to the intermediate agent; step B2, the intermediate agent responds to the client's container image pull request and sends a fingerprint information list to the client, wherein the fingerprint information list contains the fingerprint information of all data blocks of the specified container image; step B3, the client searches for the fingerprint information of all missing data blocks based on the fingerprint information list sent by the intermediate agent and its own existing data blocks, and feeds back the fingerprint information of all missing data blocks found to the intermediate agent; step B4, the intermediate agent sends all the fingerprint information fed back by the client to the image warehouse; step B5, the image warehouse searches for the data block corresponding to each fingerprint information fed back by the client through the data block retrieval module, and returns all the found data blocks to the corresponding client; step B6, the client splices the data blocks returned by the image warehouse and its own existing data blocks to obtain the container image, and starts the container based on the container image. This method migrates the client's request processor business to the intermediate proxy, effectively alleviating the pressure on the image repository. At the same time, since only the missing data blocks need to be transmitted over the network, it can maximize the utilization of network bandwidth and significantly reduce the time required for image transmission, thereby improving the data transmission efficiency and resource utilization efficiency of the entire system, providing a more advantageous solution for large-scale container image distribution and management.

[0053] The present invention proposes a new container image management and pulling solution to achieve efficient control and pulling of container images. On the image warehouse side, the storage architecture is improved, the associated data of the container image is managed in units of data blocks and duplicate data blocks are removed, and a fingerprint retrieval system is constructed in combination with hash buckets and Bloom filters to relieve storage pressure and improve retrieval efficiency; on the client side, images are pulled in units of data blocks to avoid pulling redundant data, effectively enhancing the efficiency of image pulling.

[0054] It should be noted that although the above describes the various steps in a specific order, it does not mean that the various steps must be executed in the above specific order. In fact, some of these steps can be executed concurrently or even in a different order as long as the required functions can be achieved.

[0055] The present invention may be a system, a method and / or a computer program product. The computer program product may include a computer-readable storage medium carrying computer-readable program instructions for causing a processor to implement various aspects of the present invention.

[0056] A computer-readable storage medium may be a tangible device that holds and stores instructions used by an instruction execution device. Computer-readable storage media may include, for example, but are not limited to, an electrical storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination thereof. More specific examples (non-exhaustive list) of computer-readable storage media include: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a static random access memory (SRAM), a portable compact disk read-only memory (CD-ROM), a digital versatile disk (DVD), a memory stick, a floppy disk, a mechanical encoding device, such as a punch card or a protruding structure in a groove on which instructions are stored, and any suitable combination thereof.

[0057] The embodiments of the present invention have been described above, and the above description is exemplary, not exhaustive, and is not limited to the disclosed embodiments. Many modifications and variations will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the described embodiments. The selection of terms used herein is intended to best explain the principles of the embodiments, practical applications, or technical improvements in the market, or to enable other persons of ordinary skill in the art to understand the embodiments disclosed herein.

Claims

1. A Docker container system, comprising an image repository and a client, characterized in that: The image repository is used to manage container image-related data in units of data blocks and to search for data blocks based on fingerprint information sent by the client and return them to the client, which includes: The data block management module is used to manage data blocks related to container images and is configured to manage data blocks in the following manner: Get multiple existing container images; Each container image is divided into multiple data blocks according to the preset size, and fingerprint information and metadata for recording the data block information are configured for each data block; Deduplication is performed on all data blocks based on the fingerprint information of the data blocks to delete duplicate data blocks; A data block retrieval module is used to search for corresponding data blocks according to fingerprint information; The client is used to send multiple fingerprint information required to pull the container image to the image repository, splice multiple data blocks returned by the image repository with its own existing data blocks to obtain the container image, and start the container based on the container image.

2. The system according to claim 1, characterized in that The preset size is any size within the range of 2KB-64KB.

3. The system according to claim 1, characterized in that The metadata corresponding to each data block includes the fingerprint information, storage address and number of repetitions of the data block, wherein the fingerprint information in each metadata is the hash value of the data block corresponding to the metadata, the storage address in each metadata is the storage address of the data block corresponding to the metadata in the image repository, and the number of repetitions in each metadata is the number of data blocks that are consistent with the data block corresponding to the metadata.

4. The system according to claim 3, characterized in that The data block management module is configured as follows: The Gear hash function is used to calculate the hash value of each data block based on the data in each data block, wherein the hash values ​​of data blocks with consistent data are the same.

5. The system according to claim 1, characterized in that The data block retrieval module is configured with a data block retrieval library containing metadata of all data blocks, wherein the data block retrieval library is configured with: A low-frequency metadata cache area, including a plurality of hash buckets, wherein each hash bucket corresponds to a specific fingerprint information range and is dedicated to storing metadata of all data blocks whose fingerprint information is within the specific fingerprint information range and whose number of repetitions is lower than a preset threshold; A high-frequency metadata cache area, including a plurality of hash buckets, wherein each hash bucket corresponds to a specific fingerprint information range and is dedicated to storing metadata of all data blocks whose fingerprint information is within the specific fingerprint information range and whose number of repetitions is greater than or equal to a preset threshold; Fingerprint information cache table, used to cache fingerprint information to be queried; A dynamic Bloom filter unit, including a plurality of Bloom filters, is used to determine whether metadata of a data block corresponding to the fingerprint information exists in a data block search library according to given fingerprint information; The metadata temporary storage area is used to temporarily cache the metadata to be saved, and to periodically save the cached metadata to the corresponding hash bucket in the low-frequency metadata cache area; The hash bucket address table is used to determine the fingerprint information range where the specified fingerprint information is located, and output the address of the hash bucket corresponding to the fingerprint information range in the low-frequency metadata cache area and the high-frequency metadata cache area.

6. The system according to claim 5, characterized in that The image warehouse is configured with an external memory and an internal memory, wherein the low-frequency metadata cache area is configured in the external memory of the image warehouse, and the high-frequency metadata cache area, the fingerprint information cache table, the dynamic Bloom filter unit, the metadata temporary storage area and the hash bucket address table are configured in the internal memory of the image warehouse.

7. The system according to any one of claims 5 and 6, characterized in that: The dynamic Bloom filter unit is configured as follows: Whenever the total amount of fingerprint information exceeds the capacity of the current dynamic Bloom filter unit, a new Bloom filter is added.

8. The system according to claim 5, characterized in that The data block retrieval library is configured to be updated in the following manner: The fingerprint information of each data block is stored in the fingerprint information cache table, and the following operations are performed on each fingerprint information in the fingerprint information cache table: Step S1: Use the dynamic Bloom filter unit to determine whether there is metadata corresponding to the current fingerprint information in the data block search library. If yes, execute step S2; if no, execute step S4; Step S2: Search the addresses of all hash buckets corresponding to the current fingerprint information through the hash bucket address table, and confirm whether the hash buckets at these addresses have metadata corresponding to the current fingerprint information. If yes, execute step S3; if not, execute step S4; Step S3: adding 1 to the number of repetitions of the current fingerprint information in the metadata corresponding to the hash bucket, and deleting the data block corresponding to the current fingerprint information; Step S4: adding metadata corresponding to the current fingerprint information to the metadata temporary storage area.

9. The system according to claim 5, characterized in that The data block retrieval module is configured to: Periodically scan all metadata in the data block retrieval library at preset time intervals, migrate the metadata of data blocks with a repetition number greater than or equal to a preset threshold to the corresponding hash bucket in the high-frequency metadata cache, and migrate the metadata of data blocks with a repetition number less than the preset threshold to the corresponding hash bucket in the low-frequency metadata cache.

10. The system according to claim 1, characterized in that When the client needs to start a container, the image repository and the client are configured to interact in the following manner: The client sends a pull request for the specified container image to the image repository; The image repository responds to the client's container image pull request and sends a fingerprint information list to the client, where the fingerprint information list includes fingerprint information of all data blocks related to the specified container image; The client searches for the fingerprint information of all missing data blocks based on the fingerprint information list sent by the image repository and its own existing data blocks, and feeds back the fingerprint information of all missing data blocks found to the image repository; The image repository searches for the data blocks corresponding to each fingerprint information fed back by the client through the data block retrieval module, and returns all the found data blocks to the corresponding client; The client obtains the container image by splicing the data blocks returned by the image repository and its own existing data blocks, and starts the container based on the container image.

Images