A TOC structure, system, and method for delayed container image pull.

Abstract

This invention relates to the field of IT and software development technology, specifically disclosing a TOC structure, system, and method for delayed container image pull. The TOC structure includes a Manifest, a TOC file, and a data file. The Manifest stores the image information of the container image. The TOC file contains a bloom filter for mapping the data blocks. The data file contains the data blocks of the image information. This invention addresses the problem of exposed metadata in the TOC index file in eStargz. The purpose of this invention is to propose a data verification method for delayed container image pull that reduces metadata exposure. It eliminates the need to directly store sensitive information such as the filename and file type of the chunk in the TOC index file, while also allowing the image puller to verify the pulled chunk data using the downloaded TOC file, thereby ensuring the secure transmission of container image data.

Description

Technical Field

[0001] This invention relates to the fields of IT and software development technology, and in particular to a TOC structure, system, and method for delayed container image fetching. Background Technology

[0002] Containers are started from container images, which requires pulling the container image from a remote image repository to the local machine before running the container. However, during the startup process, the container only reads a very small portion of the image content, but it must wait for all container image data to be downloaded before it can start, indicating significant room for optimization in this process.

[0003] Therefore, in recent years, the industry has proposed several delayed fetch solutions, such as stargz and eStargz (an extended version of stargz). These solutions divide the container mirror layer into fine-grained data blocks and prepare an additional index file TOC to record metadata and data summaries. During the container startup phase, the necessary data blocks are fetched from the image repository on demand, eliminating the need to wait for all image data to be downloaded before starting, thus significantly improving container startup speed.

[0004] However, because eStartgz further divides the image layer into finer-grained data chunks, the on-demand pull phase during container startup involves verifying this chunk data using the TOC (Time of Criteria). The TOC index file records the chunk's metadata (such as filename, file type, etc.) and corresponding summary information; therefore, storing the TOC in the image repository poses a risk of exposing the container image's metadata. Summary of the Invention

[0005] To address the issue of exposed metadata in the TOC index file in eStargz, this invention aims to propose a data verification method for pulling container images that reduces the latency of exposed metadata. This method eliminates the need to directly store sensitive information such as the file name and file type of the chunk in the TOC index file. Furthermore, it allows the image puller to verify the pulled chunk data using the downloaded TOC file, thereby ensuring the secure transmission of container image data.

[0006] The purpose of this invention is to provide a TOC structure, including a Manifest, a TOC file, and a data file; the TOC file is linked to both the Manifest and the data file.

[0007] The Manifest is used to store the image information of the container image; the TOC file contains a bloom filter; and the data file is a data block containing the image information.

[0008] The bloom filter is used to map the data block.

[0009] Furthermore, the data block includes several chunks and corresponding sensitive metadata for each chunk.

[0010] Furthermore, the sensitive metadata includes filenames and / or file sizes.

[0011] Furthermore, the bloom filter is used to map the chunk to the sensitive metadata corresponding to the chunk, specifically as follows:

[0012] Calculate the hash value of the chunk and the corresponding sensitive metadata, and then map the hash value to the bits of the bloom filter.

[0013] Furthermore, several of these chunks constitute the complete image information.

[0014] Furthermore, the image information includes image layers, snapshots, and toc.digest.

[0015] Furthermore, the TOC structure also includes Stargz; Stargz is used to split the complete container image into several independent layers, and the independent layers are the image layers.

[0016] The present invention also provides a system for delayed fetching of container images, including any of the TOC structures described above.

[0017] Furthermore, the system also includes: an image repository, a container engine, and containers;

[0018] The mirror repository is used to store the TOC file and the data file;

[0019] The container engine is used to obtain the TOC file and the data file from the image repository; the container engine is also used to extract the image information of the container image of the container and place the container image into the Manifest; the container engine is also used to combine the TOC file, the data file and the Manifest into any of the TOC structures described above.

[0020] This invention also provides a data verification method for reducing the latency of pulling container images by exposing metadata, including:

[0021] Obtain standard image information of a container image of a standard container, obtain the chunk of the standard image information, obtain the sensitive metadata of the chunk, and establish a TOC structure based on the standard image information, the chunk, and the sensitive metadata; the TOC structure is the TOC structure according to any one of claims 1-7;

[0022] Obtain the chunk to be verified and the metadata to be verified, perform hash calculation on the chunk to be verified and the metadata to be verified to obtain a hash value, then compare the hash value with the bits in the bloom filter in the TOC structure, and determine whether the chunk belongs to the standard container based on the comparison result.

[0023] The present invention also provides a method for reducing the delay in pulling container image data by exposing metadata, including constructing a TOC structure and data verification; wherein, the TOC structure is any of the aforementioned TOC structures established based on the image information of a standard container image;

[0024] The data verification includes obtaining the chunk to be verified and the metadata to be verified, performing a hash calculation on the chunk to be verified and the metadata to be verified to obtain a hash value, then comparing the hash value with the bits in the bloomfilter in the TOC structure, and determining whether the chunk belongs to the standard container based on the comparison result.

[0025] The main beneficial effects of this invention are:

[0026] This invention discloses a TOC structure, system, and method for delayed fetching of container images.

[0027] Compared to the existing container image delayed retrieval solution eStargz, the TOC structure, system, and method disclosed in this invention no longer expose sensitive file metadata (such as file name, file size, etc.) in plaintext in the TOC index file. Instead, they are mapped one-way to the bloom filter along with the chunk data via hashing. This can effectively reduce the effective information exposed to the image repository and potential eavesdroppers on the communication channel, thereby improving the security of container image data.

[0028] The technical solution of the present invention will be further described in detail below with reference to the accompanying drawings and embodiments. Attached Figure Description

[0029] Figure 1 Schematic diagrams of the eStargzTOC structure and the TOC structure of this invention;

[0030] Figure 2This is a schematic diagram illustrating the process of using the TOCC structure of this invention to verify the retrieved chunk data. Detailed Implementation

[0031] The technical solution of the present invention will be further described below with reference to the accompanying drawings and embodiments.

[0032] It should be noted that the following detailed description is illustrative and intended to provide further explanation of the invention. Unless otherwise specified, all technical and scientific terms used herein have the ordinary meaning as commonly understood by one of ordinary skill in the art to which this invention pertains.

[0033] It should be noted that the terminology used herein is for the purpose of describing particular implementations only and is not intended to limit the exemplary implementations of the present invention.

[0034] Figure 1 The diagram shows the TOC structure of eStargz and the TOC structure of this invention.

[0035] This invention provides a TOC structure, including a Manifest, a TOC file, and a data file; the TOC file is linked to both the Manifest and the data file.

[0036] The Manifest is used to store the image information of the container image; the TOC file contains a bloom filter; and the data file is a data block containing the image information.

[0037] The bloom filter is used to map the data block.

[0038] Understandably, this embodiment achieves fast and efficient mapping of data blocks by using a bloom filter. A bloom filter is a highly space-efficient probabilistic data structure that uses bit arrays and hash functions to check whether an element is likely a member of a target set. This makes finding specific data blocks in large amounts of data extremely fast. Due to the characteristics of bloom filters, they can also help check whether data blocks have been modified. If the original data block has been modified, the bloom filter will produce different results, thus allowing for immediate detection of data changes. The manifest file is used to store image information for the container image, including version information and other metadata. This makes version control easy, as different versions of the image can be easily viewed and managed. This structure allows for the easy addition of new data blocks or new images. Since bloom filters can map to new data blocks, data storage can be easily expanded. Bloom filters also have the advantage of preventing malicious attacks, as they can help detect potential fraud or unauthorized data access. Compared to traditional file systems, this embodiment uses a bloom filter to significantly reduce storage space usage because the bloom filter is a very compact data structure.

[0039] In some embodiments of this application, the data block includes several chunks and sensitive metadata corresponding to the several chunks.

[0040] In some embodiments of this application, the data block includes several chunks and corresponding sensitive metadata for each chunk. Such a design may offer several advantages.

[0041] Understandably, this embodiment firstly divides the data block into multiple chunks and adds corresponding sensitive metadata, increasing the flexibility of data management. This design allows users to more easily process and analyze different parts of the data block, which is particularly beneficial for large-scale data processing and analysis. Secondly, by subdividing the data block into multiple chunks, data backup and recovery can be performed more effectively. In the event of data loss, only the lost chunk needs to be recovered, rather than the entire data block, saving time and resources. Furthermore, this design helps improve data security. Sensitive metadata can include sensitive information such as Personally Identifiable Information (PII) and Personal Health Information (PHI). By managing sensitive metadata separately from other parts of the data block, this sensitive information can be better protected from unauthorized access. Finally, the mapping between chunks and sensitive metadata enables more granular access control. For example, different access permissions can be set based on the data type or amount of data contained in each chunk, thereby more precisely controlling data access and usage.

[0042] In some embodiments of this application, the sensitive metadata includes filenames and / or file sizes.

[0043] Understandably, in this embodiment, filenames and file sizes often provide clues about file content. For example, a file named "payroll.xlsx" likely contains information about salaries. Treating this type of information as sensitive metadata and encrypting or anonymizing it can protect user privacy. For files containing sensitive information, such as financial records or personally identifiable information, protecting their filenames and sizes can prevent this information from being accessed by malicious attackers or unauthorized third parties. Many data breaches are caused by the accidental disclosure of sensitive information; encrypting or anonymizing sensitive metadata can reduce this risk. In some industries or regions, regulations require special processing of sensitive data to protect user privacy and security. For example, the EU's General Data Protection Regulation (GDPR) stipulates that the processing of personal data must obtain the user's explicit consent and that sensitive data must be additionally protected. In some cases, encrypting or anonymizing filenames and / or file sizes can reduce data processing time and resource consumption, improving data processing efficiency.

[0044] In some embodiments of this application, the bloom filter is used to map the chunk to the sensitive metadata corresponding to the chunk, specifically as follows:

[0045] Calculate the hash value of the chunk and the corresponding sensitive metadata, and then map the hash value to the bits of the bloom filter.

[0046] Understandably, a Bloom filter is a highly space-efficient probabilistic data structure that uses a bit array and a series of hash functions to map input data to a series of positions within the bit array. This allows it to provide fast data retrieval with relatively low space costs when processing large amounts of data. By mapping data to the bits of a Bloom filter, it's possible to quickly check if a data block exists in a set. This provides real-time checking efficiency when processing large amounts of data, especially sensitive data. Bloom filters use bit arrays and hash functions to compress data, allowing them to store large amounts of data in a small amount of storage space. This can significantly save storage space when processing large-scale datasets. Bloom filters allow for a certain false positive rate, meaning they may mistakenly identify non-existent data as present. However, they also have a low false negative rate, meaning they won't miss existing data. This allows them to tolerate a certain error rate when processing large-scale datasets. The hash functions in Bloom filters make them highly scalable. As the dataset increases, Bloom filters simply require adding more hash functions and bit arrays. This makes them highly scalable when processing large-scale datasets.

[0047] In some embodiments of this application, a plurality of said chunks constitute the complete image information.

[0048] Understandably, this embodiment divides the mirror information into multiple chunks, which reduces the amount of data transmitted each time, lowers network transmission pressure, and improves transmission efficiency. If a chunk is lost or corrupted during transmission, only that chunk can be retransmitted instead of the entire mirror information, thereby reducing the amount of data retransmitted and improving error recovery efficiency. Chunks can be added or removed as needed to adapt to different requirements, such as increasing storage space or reducing storage pressure. Dividing the mirror information into multiple chunks ensures that each chunk is of appropriate size, avoiding the problems of low storage efficiency or wasted storage space caused by excessively large or small data blocks. Multiple chunks can be operated and managed independently, such as for backup, recovery, and migration, improving the flexibility of data management.

[0049] In some embodiments of this application, the image information includes image layers, snapshots, and toc.digest.

[0050] Understandably, firstly, this embodiment, by using a mirror layer, can quickly and accurately obtain the state of a file or directory at a specific point in time. This is extremely useful for backup, recovery operations, or troubleshooting. For example, if a system failure occurs, it can be quickly restored to its previous state by rolling back to the mirror layer. Secondly, the snapshot function can record the state of a file or directory at a specific point in time and easily compare the states at different points in time. This helps users quickly identify which files or directories have changed within a specific time period. Finally, `toc.digest` is a checksum that can be used to verify data integrity. It helps users determine whether the mirror information has been tampered with or corrupted. In summary, using mirror information can improve data security and reliability, while also increasing the efficiency of data management.

[0051] In some embodiments of this application, the TOC structure further includes Stargz; Stargz is used to split the complete container image into several independent layers, the independent layers being the image layers.

[0052] Understandably, this embodiment splits the complete container image into several independent layers, allowing only the changed parts to be transferred or stored, rather than the entire image. This significantly improves storage and transfer efficiency. When using Stargz, only the changed parts need to be read or decompressed from storage, not the entire image. This can significantly reduce startup time. Because Stargz splits the image into independent layers, each layer can be version controlled independently. This makes version control more granular and flexible. If it is necessary to update or roll back to a certain layer, only the data of that layer needs to be operated on, without operating the entire image. This makes update and rollback operations simpler and more efficient. Due to Stargz's layered storage mechanism, only the changed parts can be stored, which can save a lot of storage space for large container images, thereby reducing storage costs. Due to Stargz's independent layer design, each layer can be used independently by other containers, improving container portability. Since each layer can be independently version controlled and access managed, sensitive data can be better protected and unauthorized access can be avoided.

[0053] Figure 2 This is a schematic diagram illustrating the process of using the TOCC structure of this invention to verify the retrieved chunk data.

[0054] The present invention also provides a system for delayed fetching of container images, including any of the TOC structures described above.

[0055] In some embodiments of this application, the system further includes: an image repository, a container engine, and containers;

[0056] The mirror repository is used to store the TOC file and the data file;

[0057] The container engine is used to obtain the TOC file and the data file from the image repository; the container engine is also used to extract the image information of the container image of the container and place the container image into the Manifest; the container engine is also used to combine the TOC file, the data file and the Manifest into any of the TOC structures described above.

[0058] It is understandable that the system used in this embodiment for delayed container image pull appears to have the following advantages:

[0059] Flexibility: This system allows users to delay pulling container images when needed, rather than pulling them immediately during the build process. This provides users with greater flexibility and control, allowing them to choose when to fetch container images based on actual needs.

[0060] Efficiency: By storing the container image's metadata (TOC file and data file) in an image repository and retrieving it when needed, repeated image building can be avoided, thus saving time and resources.

[0061] Scalability: The system can be easily scaled to handle multiple container images and can be configured and customized as needed.

[0062] Standardization: By using a common TOC structure and Manifest, the standardization and consistency of container images can be ensured, thereby simplifying system management and maintenance.

[0063] Security: The system may also have security advantages, such as protecting sensitive information in data files and manifests through the use of encryption technology.

[0064] This invention also provides a data verification method for reducing the latency of pulling container images by exposing metadata, including:

[0065] Obtain standard image information of a container image of a standard container, obtain the chunk of the standard image information, obtain the sensitive metadata of the chunk, and establish a TOC structure based on the standard image information, the chunk, and the sensitive metadata; the TOC structure is the TOC structure according to any one of claims 1-7;

[0066] Obtain the chunk to be verified and the metadata to be verified, perform hash calculation on the chunk to be verified and the metadata to be verified to obtain a hash value, then compare the hash value with the bits in the bloom filter in the TOC structure, and determine whether the chunk belongs to the standard container based on the comparison result.

[0067] Understandably, this embodiment uses data verification to detect the presence of sensitive metadata in the container image, thereby reducing the risk of exposing this data. By comparing the hash value with the bits of the bloom filter in the TOC structure, it can verify whether a chunk belongs to a standard container, increasing data credibility. This method uses hash calculation and bloom filter bit comparison to quickly and efficiently complete data verification, reducing processing time and optimizing performance. The method has a clear structure, is easy to understand and maintain, and facilitates iteration and optimization in subsequent development. In summary, this embodiment's method improves security, credibility, performance, and maintainability through data verification, hash calculation, and bloom filter bit comparison.

[0068] The present invention also provides a method for reducing the delay in pulling container image data by exposing metadata, including constructing a TOC structure and data verification; wherein, the TOC structure is any of the aforementioned TOC structures established based on the image information of a standard container image;

[0069] The data verification includes obtaining the chunk to be verified and the metadata to be verified, performing a hash calculation on the chunk to be verified and the metadata to be verified to obtain a hash value, then comparing the hash value with the bits in the bloomfilter in the TOC structure, and determining whether the chunk belongs to the standard container based on the comparison result.

[0070] Understandably, this embodiment improves performance by reducing the time required to directly pull complete image data through the construction of the TOC structure and the use of data verification. If the image exists in a local cache, the step of downloading the complete image data from the source location can be skipped, thereby reducing network congestion and unnecessary network traffic. Through hash calculation and comparison, it can be ensured that the retrieved data blocks belong to the expected container image, thus enhancing data security. If some data blocks are lost or corrupted, this can be determined by checking the hash value and the bits of the bloom filter in the TOC structure, allowing appropriate fault tolerance measures to be taken. This method can be easily adapted to different container images and environments, and can also be extended and optimized as needed. Therefore, this method of reducing the latency of pulling container image data by exposing metadata can improve efficiency, security, fault tolerance, and flexibility.

[0071] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and not to limit them. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can still be made to the technical solutions of the present invention, and these modifications or equivalent substitutions cannot cause the modified technical solutions to deviate from the spirit and scope of the technical solutions of the present invention.

[0072] The system provided in the above embodiments is only illustrated by the division of the above functional modules. In practical applications, the above functions can be assigned to different functional modules as needed, that is, the modules or steps in the embodiments of the present invention can be further decomposed or combined. For example, the modules in the above embodiments can be merged into one module, or further divided into multiple sub-modules to complete all or part of the functions described above. The names of the modules and steps involved in the embodiments of the present invention are only for distinguishing the various modules or steps and are not considered as an improper limitation of the present invention.

[0073] Those skilled in the art will recognize that the modules and method steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. The programs corresponding to the software modules and method steps can be placed in random access memory (RAM), main memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, hard disks, removable disks, CD-ROMs, or any other form of storage medium known in the art. To clearly illustrate the interchangeability of electronic hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in electronic hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of the invention.

Claims

1. A system for delayed fetching of container images, characterized in that, The document includes a TOC structure, which comprises a Manifest, a TOC file, and a data file; the TOC file is linked to both the Manifest and the data file. The Manifest is used to store the image information of the container image; the TOC file contains a bloom filter; and the data file is a data block containing the image information. The bloom filter is used to map the data block. The data block includes several chunks and several sensitive metadata corresponding to the chunks; Furthermore, the sensitive metadata includes filenames and / or file sizes.

2. The system for delayed container image pull according to claim 1, characterized in that, The bloomfilter is used to map the chunk to the sensitive metadata corresponding to the chunk, specifically as follows: Calculate the hash value of the chunk and the corresponding sensitive metadata, and then map the hash value to the bits of the bloom filter.

3. The system for delayed container image fetching according to claim 2, characterized in that, Several of these chunks together form the complete image information.

4. The system for delayed container image fetching according to claim 3, characterized in that, The image information includes image layers, snapshots, and toc.digest.

5. The system for delayed container image pull according to claim 4, characterized in that, The TOC structure also includes Stargz; Stargz is used to split the complete container image into several independent layers, and the independent layers are the image layers.

6. The system for delayed container image pull according to any one of claims 1-5, characterized in that, The system also includes: an image repository, a container engine, and containers; The mirror repository is used to store the TOC file and the data file; The container engine is used to obtain the TOC file and the data file from the image repository; the container engine is also used to extract the image information of the container image of the container and place the container image into the Manifest; the container engine is also used to combine the TOC file, the data file and the Manifest to form the TOC structure in the system for delayed retrieval of container images as described in any one of claims 1-5.

7. A method for data verification of container images that reduces the latency of exposing metadata, characterized in that, include: Obtain the standard image information of the container image of the standard container, obtain the chunk of the standard image information, obtain the sensitive metadata of the chunk, and establish a TOC structure based on the standard image information, the chunk and the sensitive metadata; The TOC structure is the TOC structure in the system for delayed fetching of container images as described in any one of claims 1-5; Obtain the chunk to be verified and the metadata to be verified, perform hash calculation on the chunk to be verified and the metadata to be verified to obtain a hash value, then compare the hash value with the bits in the bloom filter in the TOC structure, and determine whether the chunk belongs to the standard container based on the comparison result.

8. A method for reducing the latency of pulling container image data by exposing metadata, characterized in that, The method includes constructing a TOC structure and data verification; wherein, the TOC structure is a TOC structure in the system for delayed container image retrieval as described in any one of claims 1-5, established based on the image information of the container image of the standard container; the data verification includes obtaining the chunk to be verified and the metadata to be verified, performing hash calculation on the chunk to be verified and the metadata to be verified to obtain a hash value, then comparing the hash value with the bits in the bloom filter in the TOC structure, and determining whether the chunk belongs to the standard container based on the comparison result.

Images