SELF-OPTIMIZING CONTAINER IMAGE FILE SYSTEM

Abstract

A data platform is provided that builds a container image file system from a container image. The data platform runs an application that accesses the container image file system and records the directory and file access activity of the application during execution. Using this directory and file access activity, the data platform dynamically rebuilds the container image file system and mounts it as a newly rebuilt system during subsequent executions of the application.

Description

PRIORITY CLAIM

[0001] This application claims priority over the preliminary US patent application Serial No. 63 / 718,403, which was filed on November 8, 2024, the contents of which are incorporated herein by reference. TECHNICAL AREA

[0002] Examples of disclosure generally relate to data platforms and, in particular, to methods for deploying and optimizing container applications. BACKGROUND

[0003] Data platforms are frequently used for data storage and access in computing and communications contexts. In terms of architecture, a data platform can be a local data platform, a network-based data platform (e.g., a cloud-based data platform), a combination of both, and / or another type of architecture. Regarding the type of data processing, a data platform might implement online transaction processing (OLTP), online analytics processing (OLAP), a combination of both, and / or another type of data processing. Furthermore, a data platform can be or include a relational database management system (RDBMS) and / or one or more other types of database management systems. Cloud-based data platforms can exchange data between databases. BRIEF DESCRIPTION OF THE DRAWINGS

[0004] The present revelation will be better understood with the help of the detailed description below and the accompanying drawings with various examples of the revelation. Fig. Figure 1 shows an example computer environment that includes a network-based data platform communicating with a cloud storage provider user system, as shown in some examples. Fig. Figure 2 is a block diagram showing the components of a Compute Service Manager according to some examples. Fig. Figure 3 is a block diagram illustrating the components of an execution platform according to some examples. Fig. Figure 4 shows an optimized container image file system pipeline according to some examples. Fig. Figure 5 illustrates an optimized container image file system computing environment according to some examples. Fig. Figure 6 illustrates an optimized container image file system procedure according to some examples. Fig. Figure 7 illustrates metadata and content blobs (“binary large objects”) of a container image file system according to some examples. Fig. Figure 8 illustrates a sequence of operations of components of an optimized container image file system, according to some examples. Fig. Figure 9 illustrates a sequence of operations of components of an optimized container image file system, according to some examples. Fig. Figure 10 shows a schematic representation of a machine in the form of a computer system in which a set of instructions can be executed to cause the machine to perform one or more of the methods discussed here, according to some examples. DETAILED DESCRIPTION

[0005] Containerized applications are becoming increasingly popular for packaging and deploying software, particularly in the fields of AI and machine learning. However, with the growing complexity and size of these applications, a significant problem has emerged: slow startup times due to the need to download and unpack large container images before execution. This process can take several minutes, even on high-speed networks, resulting in a poor user experience and inefficient resource utilization. The problem is especially pronounced with AI and ML workloads, where image sizes often exceed 10 GB.

[0006] When running an application packaged as a container image on a host, the traditional approach is to first download and unpack the entire image. Due to the sheer size of the artifacts and the time required for downloading and decompression, this can add minutes to the application's startup time. Existing solutions have attempted to address this issue, but they often lack a comprehensive and efficient approach. Some implementations rely on alternative representations of image contents or network-mounted file systems, but these solutions may not fully optimize the startup process or may introduce additional complexity.Furthermore, many of these solutions do not adequately address the need for continuous optimization based on actual application usage patterns, leaving room for improvement in reducing startup times for frequently run workloads.

[0007] Another challenge in the current landscape is the lack of efficient deduplication and storage optimization for container images within a single account or organization. This leads to unnecessary data duplication and increased storage costs, especially when multiple applications share common base layers or dependencies. Furthermore, existing solutions often lack adequate security measures, such as content encryption, which are beneficial for protecting sensitive data in containerized environments.

[0008] In some examples, a data platform, as described in this disclosure, optimizes container image file systems to improve application startup times and performance. The data platform builds a container image file system from a container image by creating a Binary Large Object (Blob) with metadata that includes a directory hierarchy, file names, sizes, and attributes, while partitioning the file contents into variable-sized content blobs. As an application accessing the container image file system runs, the data platform records directory and file access patterns to understand usage. This information is used to dynamically rebuild the container image file system by categorizing content blobs into "hot" and "cold" segments based on access frequency, grouping frequently accessed data for efficient loading.The data platform implements preloading of frequently accessed content into the kernel page cache and supports inline decompression of compressed content blobs during assembly. A version control mechanism allows the creation of new, optimized versions while maintaining access to the currently active version.

[0009] The problem is exacerbated by the inefficient handling of files that are only partially accessed, such as large binaries or shared libraries. Current approaches often require downloading entire files, even when only small portions are needed to start the application. This leads to wasted bandwidth and longer startup times, especially for applications with large dependencies that are only partially used. A comprehensive solution that addresses these challenges while offering flexibility and adaptability to different deployment scenarios is desirable to unlock the full potential of containerized applications in modern computing environments.

[0010] In some examples, the data platform builds a container image file system from a container image, runs an application that accesses the container image file system, records the directory and file access activity of the container image file system by the application during execution, dynamically rebuilds the container image file system using the directory and file access activity, and mounts the container image file system as newly built during a subsequent execution of the application.

[0011] In some examples, dynamically restoring the container image file system involves creating a new version of a metadata blob and one or more content blobs containing the container image file system, with the one or more content blobs containing files grouped based on directory and file access activity.

[0012] In some examples, dynamically restoring the container image file system involves partitioning two or more content blobs into a first set of content blobs that are expected to be fully used during application startup, and a second set of content blobs that are not expected to be fully used during application startup.

[0013] In some examples, dynamically restoring the container image file system involves compressing the contents of the first set of content blobs and decompressing the contents of the first set of content blobs using inline decompression during application startup.

[0014] In some examples, the data platform performs inline decompression of the compressed content blobs during the mounting of the container image file system.

[0015] In some examples, decompression is performed selectively based on expected usage patterns derived from directory and file access activities.

[0016] In some examples, large files are split into separate content blobs of one or more content blobs, targeted to historically accessible page areas.

[0017] In some examples, one or more content blobs are stored in an object store per account, which deduplicates the content blobs of the one or more content blobs within an account connected to the object store.

[0018] In some examples, the metadata blob is compressed and contains at least one of the following elements: directory hierarchy, file names, sizes, modification times, permissions, unique identifiers, extended attributes, link targets, and similar information.

[0019] In some examples, the one or more content blobs have a variable size and include a header followed by one or more file segments, where the header provides an index for all segments in a corresponding content blob of the one or more content blobs.

[0020] In some examples, the one or more content blobs are divided into categories based on usage patterns.

[0021] In some examples, the directory and file contents of one or more content blobs are preloaded into a kernel page cache based on directory and file access activity.

[0022] In some examples, a newly created container image file system is saved with an updated version number, while an earlier version of the container image file system remains active.

[0023] Specific examples of the embodiment of the invention are now described in detail. Examples of these specific examples are illustrated in the accompanying drawings, and specific details are set forth in the following description to enable a comprehensive understanding of the invention. It is understood that these examples are not intended to limit the scope of the claims to the examples shown. On the contrary, they are intended to cover such alternatives, modifications, and equivalents as may fall within the scope of the disclosure.

[0024] Fig. Figure 1 shows an example computer environment 100 comprising a data platform 102 communicating with a client system 112, according to some examples. To avoid obscuring the subject matter of the invention with unnecessary details, various functional components that are not essential for understanding the subject matter of the invention have been omitted. Fig. 1 omitted. However, the person skilled in the art will readily recognize that various additional functional components may be included as part of the data processing environment 100 to enable additional functions which are not specifically described here.

[0025] As shown, the data platform 102 comprises a data storage system 106, a compute service manager 104, an execution platform 110, and a metadata system 116. The data storage system 106 comprises a variety of computers and provides computer system resources, such as data storage and computing power, to the data platform 102 on demand. As shown, the data storage system 106 includes several data storage devices, such as the data storage device 108-1, the data storage device 108-2, the data storage device 108-3, and the data storage device 108-N. In some examples, the data storage devices 1 through N are cloud-based storage devices located in one or more geographic locations. For example, the data storage devices 1 through N can be part of a public cloud infrastructure or a private cloud infrastructure.The data storage devices 1 through N can be hard disk drives (HDDs), solid-state drives (SSDs), storage clusters, Amazon S3™ storage systems, or any other data storage technology. Furthermore, the data storage system can include distributed file systems (such as Hadoop Distributed File Systems (HDFS)), object storage systems, and the like.

[0026] In some examples, one or more of the data storage devices 108-1 through 108-N are cloud-based data stores configured as Virtual Private Clouds (VPCs). In some examples, a VPC is a secure, isolated virtual network within a public cloud environment, enabling organizations to operate and manage their cloud resources with enhanced control and privacy. A VPC can provide the functionality of a traditional data center without the overhead of physical management and maintenance, allowing users to define their own network scope. This includes selecting IP address ranges, creating subnets, configuring route tables, and setting up network gateways.VPCs are beneficial for organizations that want a partitioned portion of the cloud to ensure their applications and data are isolated from other users of the same public cloud platform. This isolation helps guarantee security and regulatory compliance while enabling scalable and flexible resource management.

[0027] In some examples, data objects are stored in structured data files. These structured data files can be in various structured file formats, such as Comma-Separated Values ​​(CSV), JavaScript Object Notation (JSON), Apache Avro (Avro), Apache Parquet (Parquet), Optimized Row Columnar (ORC), Extensible Markup Language (XML), and the like.

[0028] In some examples, the Data Platform 102 organizes data storage using micropartitions of a database table, employing a suitable structured data file format specifically designed for optimal performance and security within the Computing Environment 100, such as Flocon De Neige (FDN) and similar formats. Each time new data is added to a table, new micropartition files are created. This approach ensures that the data is stored in an immutable format, with the addition of a new record resulting in the creation of a new micropartition file.

[0029] Data Platform 102 is used for reporting and analyzing integrated data from one or more disparate sources, including storage devices 1 through N within Data Storage System 106. Data Platform 102 hosts multiple user accounts and provides them with data reporting and analysis services. Administrative users can create and manage identities (such as users, roles, and groups) and use permissions to allow or deny identities access to resources and services. Generally, Data Platform 102 maintains numerous consumer accounts for numerous respective consumers. Data Platform 102 manages each consumer account in one or more storage devices of Data Storage System 106. Additionally, Data Platform 102 can manage metadata associated with the consumer accounts in the metadata database 114 of Metadata System 116.Each consumer account comprises multiple objects, such as users, roles, permissions, data stores, or other data storage locations.

[0030] The Compute Service Manager 104 coordinates and manages the operations of the Data Platform 102. The Compute Service Manager 104 also performs query optimization and compilation and manages clusters of compute services that provide compute resources (also known as "virtual warehouses"). The Compute Service Manager 104 can support any number and type of clients, such as end users who make data storage and retrieval requests, system administrators who manage the systems and procedures described herein, and other components / devices that interact with the Compute Service Manager 104. For example, the Compute Service Manager 104 communicates with the Client System 112. The Client System 112 can be used by a user of one of the many consumer accounts supported by the Data Platform 102 to interact with and utilize the Data Platform 102's features.

[0031] In some examples, the Compute Service Manager 104 does not receive direct messages from the Client System 112 and only receives messages about jobs from a queue within the Data Platform 102.

[0032] The Compute Service Manager 104 is also coupled with the Metadata System 116. The Metadata System 116 includes a Metadata Database 114, which stores metadata about various functions and examples related to the Data Platform 102 and its users. In some examples, the Metadata Database 114 contains a summary of data stored in remote data storage systems as well as data available in a local cache. In some examples, the Metadata Database 114 may contain information about how the data is organized in remote data storage systems (such as the Data Storage System 106) and the local caches. In some examples, the Metadata Database 114 may contain metric data that describes the usage and access by providers and users of the data stored on the Data Platform 102.In some examples, the metadata database 114 allows systems and services to determine whether a portion of the data needs to be accessed without having to load or retrieve the actual data from a storage device.

[0033] The Compute Service Manager 104 is also coupled with the Execution Platform 110, which provides multiple compute resources that perform various data storage and data retrieval tasks. The Execution Platform 110 is coupled with the Data Storage System 106. The Execution Platform 110 comprises a variety of compute nodes. A set of processes on a compute node executes a query plan compiled by the Compute Service Manager 104.The set of processes can include: a first process to execute the query plan; a second process to monitor and delete micropartition files using an LRU (least recently used) policy and to implement an OOM (out of memory) error limitation process; a third process that extracts state information from process logs and the status to send it back to the Compute Service Manager 104; a fourth process that establishes communication with the Compute Service Manager 104 after a system startup; and a fifth process that handles communication with a compute cluster for a specific job provided by the Compute Service Manager 104 and sends information back to the Compute Service Manager 104 and other compute nodes of the execution platform 110.

[0034] In some examples, communication links between the elements of computer environment 100 are implemented via one or more data communication networks. These data communication networks can use any communication protocol and any type of communication medium. In some examples, the data communication networks are a combination of two or more interconnected data communication networks (or subnetworks). In alternative examples, these communication links are implemented using any communication medium and any communication protocol.

[0035] As in Fig. As shown in Figure 1, the data storage devices 108-1 through 108-N are decoupled from the compute resources associated with the execution platform 110. This architecture supports dynamic changes to the data platform 102 based on changing data storage / retrieval requirements, as well as changing user and system needs. Support for dynamic changes enables rapid scaling of the data platform 102 in response to changing system and component requirements. Decoupling the compute resources from the data storage devices supports the storage of large amounts of data without requiring a correspondingly large amount of compute resources.Similarly, this decoupling of resources supports a significant increase in the computing resources used at any given time, without requiring a corresponding increase in available data storage resources.

[0036] The Compute Service Manager 104, the Metadata System 116, the Execution Platform 110, and the Data Storage System 106 are in Fig. 1. The Compute Service Manager, Metadata System, Execution Platform, and Data Storage System can each be implemented as a distributed system (e.g., distributed across multiple systems / platforms in multiple geographic locations). Furthermore, the Compute Service Manager, Metadata System, Execution Platform, and Data Storage System can all scale up or down (independently) depending on changes in incoming requests and the evolving requirements of Data Platform 102. In the examples described, Data Platform 102 is therefore dynamic and supports regular changes to meet current data processing needs.

[0037] During operation, the Data Platform 102 processes several jobs defined by the Compute Service Manager 104. These jobs are scheduled and managed by the Compute Service Manager 104 to determine when and how the job should be executed. For example, the Compute Service Manager 104 can divide the job into several individual tasks and determine what data is needed to execute each individual task. The Compute Service Manager 104 can assign each of the several discrete tasks to one or more nodes of the Execution Platform 110 for processing. The Compute Service Manager 104 can determine what data is needed to process a task and further determine which nodes within the Execution Platform 110 are best suited for processing the task.Some nodes may already have cached the data required to process the task and are therefore good candidates for task processing. The metadata stored in the metadata database 114 helps the Compute Service Manager 104 determine which nodes in the execution platform 110 have already cached at least some of the data required to process the task. One or more nodes in the execution platform 110 process the task using data cached by the nodes and, if necessary, data retrieved from the data storage system 106. It is desirable to retrieve as much data as possible from the caches within the execution platform 110, as the retrieval speed is generally faster than retrieving data from the data storage system 106.

[0038] As in Fig. As shown in Figure 1, the computing environment 100 separates the execution platform 110 from the data storage system 106. In this arrangement, the processing resources and cache resources in the execution platform 110 operate independently of the database storage devices 108-1 to 108-N in the data storage system 106. Thus, the computing resources and cache resources are not restricted to a specific data storage device 108-1 to 108-N. Instead, the computing resources and cache resources can retrieve and store data from any of the data storage resources in the data storage system 106.

[0039] Fig. Figure 2 is a block diagram showing the components of Compute Service Manager 104, illustrated with some examples. As shown in Fig. As shown in Figure 2, the Compute Service Manager 104 comprises an Access Manager 202 and a Key Manager 204. The Access Manager 202 performs authentication and authorization tasks for the systems described here. The Key Manager 204 manages the storage and authentication of keys used during these authentication and authorization tasks. For example, the Access Manager 202 and the Key Manager 204 manage the keys used to access data stored on remote storage devices (such as data storage devices within the Data Storage Device 206). As used here, the remote storage devices can also be referred to as "permanent storage devices" or "shared storage devices."

[0040] In some examples, the Access Manager 202 operates within a data platform to control access to various objects within the data platform using role-based access control (RBAC). The Access Manager 202 is a component that manages authentication and authorization tasks, allowing authorized entities access to specific resources within the data platform. This component plays a role in maintaining the security and integrity of the data platform by enforcing access policies defined by RBAC.

[0041] A request processing service 208 manages received data storage and data retrieval requests (e.g., jobs to be executed on database data). For example, the request processing service 208 can determine the data required to process a received request (e.g., a data storage request or a data retrieval request). The data can be stored in a cache within the execution platform 110 or in a data storage device in the data storage system 106.

[0042] An administrative console service 210 supports access to various systems and processes by administrators and other system managers. Furthermore, the administrative console service 210 can receive requests to execute a job and monitor the workload on the system.

[0043] The Compute Service Manager 104 also includes a Job Compiler 212, a Job Optimizer 214, and a Job Executor 216. The Job Compiler 212 breaks a job down into several discrete tasks and generates the execution code for each of these tasks. The Job Optimizer 214 determines the best method for executing each task based on the data to be processed. The Job Optimizer 214 also performs various data cleansing operations and other data optimization techniques to improve the speed and efficiency of job execution. The Job Executor 216 executes the execution code for jobs that come in from a queue or are determined by the Compute Service Manager 104.

[0044] A job scheduler and coordinator 218 sends received jobs to the appropriate services or systems for compilation, optimization, and forwarding to the execution platform 110. The jobs can be prioritized and processed in that order. In some examples, the job scheduler and coordinator 218 prioritizes internal jobs scheduled by the Compute Service Manager 104 along with other "external" jobs, such as user queries scheduled by other systems in the database, but which may utilize the same processing resources in the execution platform 110. In other examples, the job scheduler and coordinator 218 identifies specific nodes in the execution platform 110 or assigns them the processing of specific tasks. A virtual warehouse manager 220 manages the operation of multiple virtual warehouses implemented in the execution platform 110.As explained below, each virtual warehouse comprises multiple execution nodes, each containing a cache and a processor.

[0045] Furthermore, the Compute Service Manager 104 includes a Configuration and Metadata Manager 222, which manages information relating to the data stored on remote data storage devices and in local caches (e.g., the caches in the Execution Platform 110). The Configuration and Metadata Manager 222 uses the metadata to determine which data micropartitions need to be accessed to retrieve data for processing a specific task or job. A Monitor and Workload Analyzer 224 monitors the processes performed by the Compute Service Manager 104 and manages the distribution of tasks (e.g., workload) across the virtual warehouses and execution nodes in the Execution Platform 110.The Monitor and Workload Analyzer 224 also redistributes tasks across the entire Data Platform 102 as needed, based on changing workloads, and can redistribute tasks based on a user query (e.g., an "external" query) that can also be processed by the Execution Platform 110. The Configuration and Metadata Manager 222 and the Monitor and Workload Analyzer 224 are coupled to a Data Storage Device 226. The Data Storage Device 226 is located in... Fig. 2 represents any data storage device within the data platform 102. For example, the data storage device 226 can represent caches in the execution platform 110, storage devices in the data storage system 106, or any other storage device.

[0046] The Compute Service Manager 104 validates communication from an execution platform (e.g., execution platform 110) to verify that the content and context of that communication are consistent with the task(s) assigned to the execution platform. For example, an instance of the execution platform executing query A should not be allowed to request access to data source D (e.g., data storage device 226), which is not relevant to query A; execution node 304a may communicate with another execution node (e.g., execution node 304b), but should not be allowed to communicate with a third execution node (e.g., execution node 316a); and such unauthorized communication may be recorded (e.g., in a log or elsewhere).Furthermore, the information stored on a particular execution node is limited to data relevant to the current query, and all other data is unusable and will be rendered unusable by destruction or encryption if the key is not available.

[0047] Fig. Figure 3 is a block diagram illustrating the components of the 110 execution platform, as shown in some examples. As in Fig. As shown in Figure 3, the Execution Platform 110 comprises several virtual warehouses, including Virtual Warehouse 302a, Virtual Warehouse 302b, and Virtual Warehouse 302c. Each virtual warehouse includes multiple execution nodes, each containing a data cache and a processor. The virtual warehouses can execute multiple tasks in parallel by utilizing multiple execution nodes. As mentioned earlier, the Execution Platform 110 can add new virtual warehouses and dissolve existing ones in real time based on the current processing demands of the systems and users. This flexibility allows the Execution Platform 110 to quickly provision large amounts of computing resources when needed, without being forced to continue paying for those resources when they are no longer required. Virtual warehouses can access data from any data storage device (e.g.,from each storage device in the data storage system 106).

[0048] Although each in Fig. While the virtual warehouse shown in Figure 3 comprises three execution nodes, a given virtual warehouse can contain any number of execution nodes. Furthermore, the number of execution nodes in a virtual warehouse is dynamic, so new execution nodes are created when additional capacity is needed, and existing execution nodes are deleted when they are no longer required.

[0049] Each virtual warehouse is capable of accessing any of the in Fig. The virtual warehouses are therefore not necessarily assigned to a specific data storage device 1 to N, but can access data from any of the data storage devices 1 to N within the data storage system 106. Similarly, each of the Fig. The three execution nodes shown can access data from each of the data storage devices 1 through N. In some examples, a particular virtual warehouse or execution node may be temporarily assigned to a specific data storage device, but the virtual warehouse or execution node can later access data from any other data storage device.

[0050] In the example of Fig. In section 3, virtual warehouse 302a comprises a variety of execution nodes, such as execution node 304a, execution node 304b, and execution node 304c. Execution node 304a includes a cache 306a and a processor 308a. Execution node 304b includes a cache 306b and a processor 308b. Execution node 304c includes a cache 306c and a processor 308c. Each execution node (1 through N) is associated with processing one or more data storage and / or data retrieval tasks. For example, a virtual warehouse can handle data storage and data retrieval tasks associated with an internal service, such as a clustering service, a materialized view update service, a file compaction service, a storage procedure service, or a file upgrade service.In other implementations, a specific virtual warehouse can handle data storage and data retrieval tasks in conjunction with a specific data storage system or data category.

[0051] Similar to the virtual warehouse 302a described above, virtual warehouse 302b comprises a variety of execution nodes, such as execution node 310a, execution node 310b, and execution node 310c. Execution node 304a includes cache 312a and processor 314a. Execution node 310b includes cache 312b and processor 314b. Execution node 310c includes cache 312c and processor 314c. Furthermore, virtual warehouse 302c comprises a variety of execution nodes, such as execution node 316a, execution node 316b, and execution node 316c. Execution node 316a includes cache 318a and processor 320a. Execution node 316b includes cache 318b and processor 320b. The execution node 316c includes the cache 318c and the processor 320c.

[0052] In some examples, the in Fig. The three execution nodes depicted are stateless with respect to the data they cache. For example, these execution nodes do not store or otherwise maintain any state information about themselves or the data cached by a particular execution node. Therefore, if an execution node fails, it can be transparently replaced by another node. Because no state information is associated with the failed execution node, the new (replacement) execution node can easily replace it without requiring any specific state to be restored.

[0053] Although the in Fig. While the three execution nodes shown each contain a data cache and a processor, alternative examples can include execution nodes with any number of processors and any number of caches. Furthermore, the cache size can vary between the different execution nodes. The examples in Fig. The three caches shown store data retrieved from one or more storage devices in the Data Storage System 106 on the local execution node. This reduces or eliminates the bottleneck problems encountered by platforms that constantly retrieve data from remote storage systems. Instead of repeatedly accessing data from the remote storage devices, the systems and procedures described here access data from the caches on the execution nodes, which is significantly faster and avoids the bottleneck problem described above. In some examples, the caches are implemented using high-speed storage devices, enabling fast access to the cached data. Each cache can store data from any storage device in the Data Storage System 106.

[0054] Furthermore, cache resources and compute resources can vary between different execution nodes. For example, one execution node might have extensive compute resources and minimal cache resources, making it suitable for tasks requiring substantial compute resources. Another execution node might have significant cache resources and minimal compute resources, making it suitable for tasks requiring the caching of large amounts of data. Yet another execution node might have cache resources that enable faster input / output operations, which is useful for tasks requiring the rapid scanning of large datasets.In some examples, the cache resources and compute resources allocated to a particular execution node are determined during the creation of the execution node based on the expected tasks that the execution node is intended to perform.

[0055] Furthermore, the cache and compute resources allocated to a particular execution node can change over time based on the tasks performed by that node. For example, an execution node might be allocated more processing resources if the tasks it performs become more processor-intensive. Similarly, an execution node might be allocated more cache resources if the tasks it performs require greater cache capacity.

[0056] Although virtual warehouses 1, 2, and N are connected to the same execution platform 110, they can be implemented using multiple computer systems at different geographic locations. For example, virtual warehouse 1 can be implemented by one computer system at a first geographic location, while virtual warehouses 2 and N can be implemented by another computer system at a second geographic location. In some examples, these different computer systems are cloud-based systems operated by one or more different entities.

[0057] Furthermore, each virtual warehouse, as in Fig. Figure 3 shows multiple execution nodes. The multiple execution nodes associated with each virtual warehouse can be implemented using multiple computer systems at multiple geographic locations. For example, an instance of virtual warehouse 302a implements execution node 304a and execution node 304b on a data processing platform at one geographic location and execution node 304c on a different data processing platform at a different geographic location. The selection of specific computer systems for implementing an execution node can depend on various factors, such as the amount of resources required for a particular execution node (e.g.,Requirements for processing resources and the cache), the resources available on specific computer systems, the communication capabilities of networks within a geographic location or between geographic locations, and the computer systems that already implement other execution nodes in the virtual warehouse.

[0058] A given execution platform 110 can contain any number of virtual warehouses. Furthermore, the number of virtual warehouses within a given execution platform is dynamic, so new virtual warehouses are created when additional processing and / or caching resources are needed. Likewise, existing virtual warehouses can be deleted when the resources associated with them are no longer required.

[0059] In some examples, the virtual warehouses can work with the same data in data storage system 106, but each virtual warehouse has its own execution nodes with independent processing and caching resources. This configuration allows requests to different virtual warehouses to be processed independently and without interference between requests. This independent processing, combined with the ability to dynamically add and remove virtual warehouses, supports adding new processing capacity for new users without impacting the performance observed by existing users.

[0060] Fig. This is a state diagram illustrating the phases of an optimized container image file system pipeline (400) for a self-optimizing container image file system, as shown in several examples. A data platform uses the optimization pipeline to dynamically optimize the container image file system based on application usage patterns.

[0061] In a commit phase 404, the data platform receives components of a container image file system (420). During the commit phase 404, a user uploads a container image to the data platform. In a build phase 406, the data platform executes a job to convert components of the container image into a non-optimized version of a container image file system, as shown in Fig. 5 and Fig. 6 is described in more detail. The container image file system is mounted (418) when an application of the container image is run from the data platform.

[0062] In a 402 execution phase, the data platform runs the application using a mounted version of the container image file system. During the initial application run, the mounted container image file system may not be optimized. During the 402 run phase, the data platform records directory and file access activity and generates 410 activity reports that capture file system usage patterns, as described in the documentation. Fig. 8 described in more detail.

[0063] In a learning phase 408, the data platform analyzes the activity reports 410 to create an optimization of the container image file system by analyzing the file access patterns during execution and the categorization of the content blobs into "hot" and "cold" based on usage, so that the container image file system can be dynamically rebuilt with an optimized blob organization, as referenced in Fig. 6 described in more detail.

[0064] Upon returning to build phase 406, the data platform dynamically rebuilds the container image file system, using the container image file system metadata 416 generated in learning phase 408 to optimize the structure of the container image file system and the organization of its content based on learned access patterns. This may involve grouping frequently accessed files together or reorganizing content blobs for more efficient access, as described in the following sections: Fig. 6 described in more detail.

[0065] When the container image is subsequently executed in operational phase 402, the container image's file system is mounted in its newly built, optimized version (see Fig. ), thereby making the optimized file system available to the application, utilizing the optimizations performed in the previous phases to improve startup and runtime performance.

[0066] This cyclical process allows the data platform to continuously learn from application behavior and dynamically optimize the container image file system accordingly, reducing boot times and improving overall performance for containerized applications. For example, the data platform improves performance in several ways. In some instances, it reduces disk space usage by selectively downloading content blobs based on actual application needs, thus avoiding unnecessary downloads of unused portions of the container image file system. In other instances, the data platform implements inline decompression for frequently accessed content when downloading content blobs and selectively applies compression based on expected usage patterns derived from directory and file access activity.In other examples, the data platform stores one or more content blobs in an object store per account, which deduplicates content blobs within an account, thus avoiding redundant storage of identical content. In further examples, the data platform preloads the directory and file contents of the one or more content blobs into a kernel page cache based on directory and file access activity, thereby speeding up access times for frequently used data. In some examples, the data platform categorizes content blobs based on usage patterns, enabling optimized loading strategies that minimize both storage space and computational overhead.In some examples, the data platform implements special compression and decompression strategies for frequently used content, including inline decompression during assembly, which reduces CPU usage by avoiding repeated compression / decompression cycles.

[0067] Fig. Figure 5 is a block diagram showing the components of an optimized container image file system computing environment (500) for managing container images and their file systems, illustrated with some examples. A data platform utilizes the system to manage container images and their file systems, as well as to optimize the container image file system.

[0068] A Compute Service Manager 502 coordinates operations with the application cluster 506 via system functions 522. Within the application cluster 506, an image registry component 512 stores and manages metadata for one or more container image file systems, such as optimized containers 516. In some examples, the application cluster 506 is a cluster hosted by the data platform that provides infrastructure services. The application cluster 506 might, for example, contain the image registry component 512, which stores and manages metadata for container image file systems. The image registry component 512 operates as a service within the application cluster 506 to process container image uploads from users and initiate asynchronous jobs to convert uploaded images into an optimized container image file system format.In some examples, the Application Cluster 506 provides a dedicated environment for these infrastructure services, maintaining separation from other data platform components and clusters while enabling integration through system functions and blob storage paths. In other examples, the Image Registry component 512 interfaces to a blob storage system to manage metadata and content blobs and coordinate with other components via defined communication channels, including blob write and read paths. This architecture allows the data platform to maintain centralized control over container image processing and optimization while delivering scalable services across the Application Cluster infrastructure. The operations of the Application Cluster 506 and the Image Registry component 512 are described in [reference to relevant documentation]. Fig. 6, Fig. 8 and Fig. 9 described in more detail.

[0069] System functions 522 provide functions for optimizing and using container image file systems and provide a transactional interface for creating, uploading, and transferring these file systems. The Binary Large Object (Blob) write path 524 represents a path for writing data blocks to the blob store 504, which the system functions use to upload blocks comprising a container image file system. A data storage system 510 provides storage resources for the overall system, while the blob store 504 specifically stores metadata blobs, which contain file system metadata of a container image file system, and content blobs, which store the actual contents of the container image file system.

[0070] Within a Container Service Node (508), several components interact to manage container execution and file system access. A Container Runtime Component (514) represents a customer's workload running in a container. The user-space file system (518) is a component that handles file system operations for the container images and serves as an interface between the container and the underlying container image file system. In some examples, the data platform implements a Container Service Node as part of a cluster infrastructure specific to each account. The Container Service Nodes operate within a cloud service provider environment and integrate with both application clusters and the Container Service infrastructure. In some examples, each Container Service Node contains core components for managing containerized workloads.This architecture allows the data platform to maintain isolated computing environments per account while simultaneously utilizing shared infrastructure services across the entire cloud service provider environment, as in . Fig. 6, Fig. 8 and Fig. 9 described in more detail.

[0071] The container runtime component 514 also interacts with a snapshotter component 520. The snapshotter component 520 manages the creation of container image file system snapshots, which are used to materialize a container image file system for the container runtime component 514 at runtime. The snapshotter component 520 creates objects that can be used by the container runtime component 514 to unpack a layer of a container image. The snapshotter component 520 creates file system snapshots for each layer of a container image. These snapshots represent the state of the container image's file system at different points in the container's layer hierarchy.During the process of creating file system snapshots of the container image, the Snapshotter component 520 also stores metadata in an object store of the container runtime component 514 to reflect the state of each snapshot. The Snapshotter component 520 communicates with the file system in the userspace component 518 via a gRPC / UDS channel 528, for example, a communication channel that uses gRPC over a Unix domain socket for efficient local communication.

[0072] The blob read path 526 facilitates the retrieval of data from the blob storage 504 to the file system in the userspace component 518. This allows the file system in the userspace component 518 to access the necessary data to provide file system services to the container runtime component 514.

[0073] Fig. Figure 6 shows an example of an optimized container image file system procedure 600 according to some examples. Although the example of an optimized container image file system procedure 600 represents a specific sequence of operations, the sequence can be modified without exceeding the scope of this disclosure. For example, some of the operations shown can be performed in parallel or in a different order that does not substantially affect the functionality of the optimized container image file system procedure 600. In other examples, different components of an example device or system implementing the optimized container image file system procedure 600 can perform functions essentially at the same time or in a specific order.

[0074] In Operation 602, a data platform uses an image registry component, such as the image registry component 512 from Fig. 5, to build a container image file system from a container image in Operation 602. Regarding Fig. For example, the data platform generates a metadata blob 702 and one or more content blobs 724, which comprise the metadata and content blobs of a container image file system 700 created from a container image. The metadata blob 702 contains a layer 704, which represents a specific layer of the container image. One or more content blob references 706 provide links to the associated content blob(s) 724 using one or more blob digests 708. A blob digest 708 is a hash value used to uniquely identify a corresponding content blob from one or more content blobs 724.

[0075] The metadata blob 702 contains one or more index nodes (inodes) 710. The inodes 710 contain metadata about a file or directory in a container image file system, except for the name and the actual data content. Each file and directory in the container image file system is assigned a unique inode, which contains one or more directories 712 and one or more files 718. A directory component contains information about the directory structure of the file system. A file component represents individual files within the file system. Each file can contain one or more data blocks 722 associated with it.

[0076] A content blob 724 contains the actual file data and can be structured to optimize access. It contains a blob header 726, which serves as an index for the content of the content blob 724. One or more segment pointers 728 provide a mechanism for locating specific data segments within the content blob. One or more data segments 730 contain the actual file content, which may be compressed to save space.

[0077] This structure enables efficient storage and querying of file system data. The metadata blob can be downloaded quickly to provide the overall file system structure, while the content blobs can be retrieved as needed. This approach allows for faster application startup times because the file system can be mounted before all the data has been downloaded.

[0078] Separating metadata from content also facilitates optimizations such as deduplication of large files across different container images and efficient rebuilding of file systems based on observed usage patterns. Unlike traditional container image layers, which require deduplication at the entire layer level, this method enables granular deduplication of individual files. The data platform leverages collected directory and file access activity to dynamically rebuild the container image file system and reorganize content blobs based on actual usage patterns, rather than being constrained by a layered organization.In this way, the data platform can group frequently accessed files and partition content blobs based on usage patterns, thereby exceeding the efficiency possible with layer-level deduplication.

[0079] In some examples, a container image file system contains a metadata blob (920) and one or more content blobs (724). The metadata blob (702) contains the directory hierarchy, file names and sizes, modification times, permissions, unique identifiers, extended attributes, link targets, and block indexes for the file contents of a container image file system. The data blocks themselves are stored in one or more content blobs. In some examples, the metadata blob is relatively small (e.g., a compressed, serialized metadata Protobuf message), typically 0.1–0.2% of the total file system size. In some examples, each content blob consists of a compressed, serialized BlobHeader Protobuf message followed by one or more file segments. A file segment is a byte range of a file (possibly the entire file, or possibly only a portion of it) and can be compressed or uncompressed.The BlobHeader acts as an index for all segments in the blob, including their offset and whether they are compressed. In some examples, content blobs have a variable size (e.g., between 4 MB and 64 MB), depending on how the contents of the container image file system are distributed across the one or more content blobs. In some examples, the metadata blob is compressed.

[0080] In some examples, a metadata blob is used to mount a container image filesystem, traverse the directory hierarchy of the container image filesystem, specify paths, read symbolic links, and open files. Once a non-empty regular file is read, its contents are loaded from one or more of the container image filesystem's content blobs. The loaded content blobs are determined by a read offset and length, as well as a blocklist in a corresponding file.

[0081] In some examples, when converting a container image to a container image file system, large files (defined as those that reach or exceed a configurable size parameter threshold) are packed into their own separate blobs. This enables deduplication of large files, thus reducing the overall amount of data that needs to be uploaded or downloaded. Smaller files (those that do not reach the configurable size parameter threshold) can be packed with other small files to reduce the number of trips to and from the blob data storage.

[0082] In some examples, a user uploads their container images to a file system image registry, such as the image registry component 512 of Fig. 5, which is hosted in an application cluster, e.g., in application cluster 506 of Fig. 5. The image registry not only stores these images in an image repository, but also invokes an asynchronous job to convert the container images into a container image file system format using an optimizer. During the optimization process, each layer of the container image is converted into its own container image file system.

[0083] In some examples, after all layers of the container image have been written, the layers are collapsed (flattened) into a single container image file system by resolving whiteouts and opaque directories. This is a metadata operation and requires no changes to the content blobs. For example, the metadata of each layer is analyzed to identify whiteouts and opaque directories. This information is then combined to create a unified view of the container image file system without altering the underlying content blobs. This allows for an efficient representation of the final state of the container image file system while preserving the original layer structure of the container image.

[0084] In some examples, layer information is stored in a metadata blob to perform this flattening operation. The layers are traversed sequentially, applying the changes represented by each layer's metadata to build the final filesystem structure of the container image. This might involve marking specific files or directories as deleted or hidden based on whiteouts, or replacing entire directory trees based on opaque directory markers.

[0085] In other examples, the flattening process can be optimized by using special data structures to efficiently track and merge changes to the container image file system across different levels. Techniques such as copy-on-write or reference counting can be used to minimize memory usage and improve performance with a large number of levels or complex file system hierarchies.

[0086] The flattened file system is more efficient to use because it avoids downloading large files that become inaccessible due to whiteouts or opaque directories. At the same time, the provenance of each file's level in the metadata blob is preserved, allowing files from common dependencies (i.e., base levels) to be served with inode numbers that maximize page cache utilization.

[0087] In some examples, a data persistence object is used to track the set of blobs in an object store that comprise a particular container image file system. The functionality for creating and using container image file systems is provided to an image registry via system functions. On the write path, the system functions provide a transaction interface through which the image registry executes the sequence of operations used to create a container image file system, request the upload of the container image file system blobs, manage the upload of the blobs, and transfer the container image file system. In some examples, a blob upload step returns a no-op if a blob with the same digest already exists in an account's object store.If the blob does not exist, a request to upload a blob returns the blob's location, the login credentials, and the encryption key for uploading the blob. In some examples, the blobs are encrypted before being uploaded.

[0088] In some examples, a layer-level container image file system in a data storage system is encrypted by a layer digest, while a collapsed image-level container image file system is encrypted by an image manifest digest.

[0089] In Operation 604, the data platform executes an application that accesses the container image file system, and in Operation 606, the data platform records the directory and file access activity on the container image file system by the application during application execution. For example, regarding Fig. 8, a sequence of operations of components of an optimized container image file system 800, includes an application 802 running on the data platform that accesses files and directories of a container image using services provided by the file system in user-space component 804. During the execution of the application, the user-space file system component 804 records file system access requests 810 made by the application 802. The user-space file system component 804 records detailed information about the directory and file accesses, including, but not limited to: which files are opened, read, and written; the number of times and frequency with which files and directories are accessed; the real-time of each access; the order of access; the completeness of the access; and the like.Component 804 for the user-space file system generates activity reports 818 containing the captured file access data and periodically sends these activity reports 812 to a snapshotter component 806. The snapshotter component 806 collects and consolidates the activity reports 820 received by the user-space file system 804. The consolidated activity reports 814 are then transmitted by the snapshotter component 806 to a store object 808, where the stored activity reports are associated with the specific application that generated the record directory and file access data.

[0090] In Operation 608, the data platform dynamically rebuilds the container image file system using directory and file access activities, resulting in a newly rebuilt container image file system. For example, the Snapshotter component 806 dynamically rebuilds the container image file system using the saved activity reports (see Fig. 8) In some examples, the Snapshotter component 806 analyzes the activity reports generated during previous application runs to determine which files and directories are frequently accessed during startup. The Snapshotter component 806 then uses this information to create a new metadata blob and one or more content blobs that reflect the optimized and rebuilt file system structure of the container image.

[0091] In some examples, the Snapshotter component 806 divides content blobs into two groups based on their expected usage during application startup. The first group contains content blobs that are expected to be fully used during startup. These "hot" blobs contain data that is likely to be fully accessed when the application begins running. The second group comprises content blobs that are not expected to be fully used during startup. These "cold" blobs contain data that is unlikely to be needed immediately. This division is based on the analysis of previous application runs and the resulting activity reports.The Snapshotter component 806 uses this information to determine which files and data segments are frequently accessed during startup and groups them into the first set of content blobs. The remaining content blobs are placed in the second set. By organizing the content blobs in this way, Snapshotter component 806 can prioritize downloading and processing the first set of content blobs, thereby improving startup performance by ensuring that the most urgently needed data is immediately available.

[0092] In some examples, the Snapshotter component 806 can perform more granular categorization based on specific usage patterns. For instance, Snapshotter component 806 can create categories for content blobs accessed during startup, for content blobs accessed during specific application operations, and for content blobs that are rarely or never accessed. This fine-grained categorization enables optimized loading and caching strategies.

[0093] In some examples, the Snapshotter component 806 compresses the contents of the first set of content blobs that are expected to be fully used when the application starts. During application startup, the system then decompresses this content using inline decompression when the container image file system is mounted. For example, the Snapshotter component 806 applies compression to the "hot" content blobs, which contain data that is likely to be fully accessed when the application starts. This compression reduces the amount of data that needs to be downloaded initially. In some examples, the Snapshotter component 806 uses a compression algorithm that enables streaming decompression for these "hot" content blobs, such as Zstandard or similar.When downloading the content blob, the Snapshotter component 806 decompresses the content blob on the fly before writing it to object storage. This approach enables efficient data storage and transfer while providing fast access at application startup. In further examples, the system can implement a Boolean field `BlobHeader.compressed` to indicate that all segment data in a blob is compressed. This allows for efficient compression of all segments into "hot" content blobs, rather than marking each compressed segment individually.

[0094] In some examples, the Snapshotter component 806 selectively applies decompression based on expected usage patterns derived from directory and file access activity. For instance, the system analyzes activity reports generated during previous application runs to determine which files and directories are likely to be accessed while the application is in use, not just at startup. The Snapshotter component 806 then uses this information to selectively decompress content blobs containing frequently accessed data while the application 802 is in use.

[0095] In some examples, the Snapshotter component 806 stores one or more content blobs in an object store per account, which deduplicates the content blobs within an account connected to the object store. For example, the Snapshotter component 806 writes metadata blobs and associated content blobs to an object store secured by the Cloud Service Provider's (CSP) storage. This object store is specific to each account and deduplicates blobs across the account using a checksum, such as a SHA256 checksum, of the blob content. In some examples, a transactional interface is used for blob storage operations. When a blob is uploaded, a check is performed to see if a blob with the same digest already exists in the account's object store. If such a blob exists, the upload operation is aborted, so the blob's content is deduplicated across the entire account.This approach reduces storage requirements, especially for large files such as libraries that can be shared by multiple container images within an account.

[0096] In some examples, the Snapshotter 806 component splits large files into separate content blobs of one or more content blobs and aligns them to historically accessed page regions. For example, Snapshotter 806 analyzes activity reports to determine which parts of large files are frequently accessed during application startup. It then splits these large files into separate content blobs, grouping the frequently accessed parts into "hot" blocks. In some examples, the system implements "file ripping" for certain file types, such as ELF (Executable and Linkable Format) files, shared objects, binaries, and so on, but is not limited to these. This process splits any page region of a file into individual content blobs without requiring a predefined piece size.This approach can significantly reduce the amount of data in "hot" content blobs, especially for large binaries and shared libraries that are only partially read at startup. In further examples, the Snapshotter component 806 can use ELF section headers or program headers to control the splitting process. By aligning the content blob boundaries with the natural subdivisions of the file structure, the system can optimize the loading of frequently used sections and defer the loading of less important sections, such as debug symbols.

[0097] In some examples, the system performs "reblobification" by grouping the accessed segments into a smaller set of content blobs. This reduces the number of content blobs that need to be downloaded immediately when the application starts, thereby reducing the total number of content blobs to load.

[0098] In other examples, the Snapshotter component 806 can perform resegmentation by splitting partially read files so that the page ranges being accessed are stored separately. This allows the system to move hot page ranges into hot blobs and keep cold page ranges separate, further optimizing access patterns. In some examples, the Snapshotter component 806 adjusts the compression strategies for frequently used segments and stores them uncompressed to avoid extraction and improve I / O efficiency.

[0099] In some examples, the Snapshotter component 806 stores a newly rebuilt container image file system with an updated version number while keeping an earlier version of the container image file system active. For instance, a container image file system data persistence object includes a version number attribute to support online optimization of the container image file system. This allows the data platform to use the most recent confirmed version of the container image file system when deploying it, while simultaneously creating a new, unconfirmed version. In some examples, when rebuilding the container image file system, the Snapshotter component 806 creates a new version with optimized content blob organization based on observed usage patterns.This new version of the container image file system can be created and saved without interrupting access to the currently active version. In further examples, the Snapshotter component 806 can implement a transactional interface for updating container image file system versions. This ensures that the transition between versions of a container image file system is atomic, enabling seamless updates without affecting ongoing operations that rely on the current active version.

[0100] In Operation 610, the data platform mounts the optimized and rebuilt container image file system during a subsequent application execution. Regarding Fig. For example, during a sequence of operations by components of an optimized container image file system, a container runtime component (900) communicates a file system request (912) to a snapshotter (904) to request a materialization or snapshot of a container image file system when it boots up to run an application. A snapshotter (904) receives the request (912) and, in response (914), requests metadata of the container image file system from an image registry (906).

[0101] If the requested container image file system was previously transferred to, optimized, and registered in image registry 906, image registry 906 submits a manifest 916 to snapshotter 904 in response to manifest request 914. The manifest may contain a metadata blob, including a directory hierarchy, file names and sizes, modification times, permissions, one or more unique identifiers, extended attributes, link targets, block indexes for file contents, and similar information. The manifest may also contain references to content blobs, including blob digests (SHA256), blob locations in object storage, encryption keys for accessing the blobs, CSP storage credentials to enable secure access to the contents, and similar information.The manifest can contain system information, including pre-signed URLs for blob access, mount point information, version numbers for file system optimization, configuration details required for file system operations, and similar information. The manifest can also contain activity data, including historical file and directory access patterns, information about preloaded objects, usage metrics and optimization data, segment compression status and formats to enable performance optimization, and similar information.

[0102] The snapshotter 904 receives the manifest 916 and generates a mount request 918 based on the manifest 916. The snapshotter 904 transmits the mount request 918 to a file system in the user-space component 908. The user-space file system component 908 receives the mount request 918 and then begins building the container image file system in a blob store 910 using a metadata blob 920 of the container image file system and one or more content blobs 924 associated with the metadata blob 920. The user-space file system component 908 assigns a mount location 922 for the container image file system to component 930 and transmits the mount location 922 to the snapshotter 904. The snapshotter 904 receives the mount location 922 and communicates the mount location 922 to the container runtime component 902.The container runtime component 902 receives the mount location 922 and uses the mount location 922 to access the container image file system optimized by the snapshotter 904.

[0103] In some examples, if the image registry 906 does not find that the requested container image file system is registered, the image registry 906 returns an "Image not found" 926 response to the manifest 916 from the snapshotter 904. The snapshotter 904 then sends an unoptimized file system mount request 928 to the user-space file system component 908. The user-space file system component 908 then mounts an unoptimized version of the requested container image file system.

[0104] This design allows the Snapshotter 904 and the user-space file system 908 to efficiently manage container image file systems, take advantage of optimizations when available, while maintaining compatibility with traditional image-pulling methods.

[0105] In some examples, the file system in user-space component 908 preloads directory and file contents from one or more content blobs into a kernel page cache based on directory and file access activity. For example, Snapshotter 904 analyzes activity reports generated during previous application executions to determine which directories and files are likely to be accessed during startup. During application startup, Snapshotter 904 instructs the file system in user-space component 908 to proactively load this data into the kernel page cache before the application executes. In some examples, Snapshotter 904 may implement a preload strategy that prioritizes loading the most frequently accessed directories and files to ensure that critical data is available in the kernel page cache as quickly as possible.In other examples, Snapshotter 904 can use the collected activity data to fine-tune the preloading process, adjusting the amount and type of preloaded data based on observed access patterns and available system resources. This adaptive approach enables optimized use of the kernel page cache, thereby improving application startup times and overall performance.

[0106] Fig. Figure 10 shows a schematic representation of a machine 1000 in the form of a computer system in which a set of instructions can be executed to cause the machine 1000 to perform one or more of the methods discussed herein according to the examples. In particular, it shows Fig. Figure 10 shows a schematic representation of the machine 1000 in the example form of a computer system in which instructions 1002 (e.g., software, a program, an application, an applet, or other executable code) can be executed to cause the machine 1000 to perform one or more of the methods discussed herein. For example, the instructions 1002 can cause the machine 1000 to perform one or more operations of one or more of the methods described herein. In this way, the instructions 1002 transform a general, unprogrammed machine into a specific machine 1000 (e.g., the Compute Service Manager 104, the execution platform 110, and the data storage devices 108-1 through 108-N of the data storage system 106) that is specifically configured to perform one of the functions described and illustrated in the manner described herein.

[0107] In alternative examples, Machine 1000 operates as a standalone device or can be coupled (e.g., networked) with other machines. In a networked deployment, Machine 1000 can operate as a server or client in a server-client network environment or as a peer in a peer-to-peer (or distributed) network environment. Machine 1000 can include a server computer, a client computer, a personal computer (PC), a tablet computer, a laptop computer, a netbook, a smartphone, a mobile device, a network router, a network switch, a network bridge, or any other machine capable of executing the instructions 1002, sequentially or otherwise, that specify the actions to be performed by Machine 1000.Even if only a single machine 1000 is depicted, the term "machine" is also intended to include a collection of machines that, individually or collectively, execute instructions 1002 to perform one or more of the methods described herein.

[0108] The machine 1000 comprises hardware processors 1004, memory 1006, and I / O components 1008, configured to communicate with each other, for example, via a bus 1010. In some examples, the hardware processors 1004 (e.g., a central processing unit (CPU), a RISC (Reduced Instruction Set Computing) processor, a CISC (Complex Instruction Set Computing) processor, a graphics processing unit (GPU), a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a radio frequency integrated circuit (RFIC), another hardware processor, or any suitable combination thereof) may comprise multiple processors, as illustrated by the processor 1012 and a processor 1014 that can execute the instructions 1002.The term "processor" is intended to include multi-core processors, which may comprise two or more independent processors (sometimes referred to as "cores") capable of executing instructions 1002 concurrently. Although... Fig. As shown in Figure 1004, which indicates multiple hardware processors, the machine 1000 can include a single single-core processor, a single multi-core processor (e.g., a multi-core processor), multiple single-core processors, multiple multi-core processors, or any combination thereof.

[0109] The memory 1006 can comprise main memory 1032, static memory 1016, and a memory unit 1018 with a machine memory medium 1034, which the hardware processors 1004 can access, for example, via the bus 1010. Main memory 1032, static memory 1016, and the memory unit 1018 store the instructions 1002, which embody one or more of the methods or functions described herein. During their execution by the machine 1000, the instructions 1002 can also reside wholly or partially in main memory 1032, in static memory 1016, in the memory unit 1018, in at least one of the hardware processors 1004 (e.g., in the processor's cache memory), or in a suitable combination thereof.

[0110] The input / output components 1008 include components for receiving inputs, providing outputs, generating outputs, transmitting information, exchanging information, acquiring measurements, etc. The specific I / O components 1008 included in a particular machine 1000 depend on the type of machine. For example, portable devices such as mobile phones will likely include a touch input device or other such input mechanisms, whereas a headless server machine will likely not include such a touch input device. It becomes clear that the I / O components 1008 can include many other components that are in Fig.10 are not shown. The I / O components 1008 are grouped according to their functionality only for the sake of simplifying the following discussion, and the grouping is in no way restrictive. In various examples, the I / O components 1008 may include output components 1020 and input components 1022. The output components 1020 may include visual components (e.g., a display such as a plasma display panel (PDP), a light-emitting diode (LED) display, a liquid crystal display (LCD), a projector, or a cathode ray tube (CRT)), acoustic components (e.g., loudspeakers), other signal generators, and so on. The input components 1022 may include alphanumeric input components (e.g., a keyboard, a touchscreen configured for alphanumeric input, a photoelectric keyboard, or other alphanumeric input components), dot-based input components (e.g., a keyboard, a touchscreen configured for alphanumeric input, a photoelectric keyboard, or other alphanumeric input components), and other types of input components (e.g., a keyboard, a keyboard, a keyboard, a keyboard, or other types of input components).B, a mouse, a touchpad, a trackball, a joystick, a motion sensor or other pointing device), tactile input components (e.g. a physical button, a touchscreen that provides the location and / or force of touches or touch gestures, or other tactile input components), audio input components (e.g. a microphone) and the like.

[0111] Communication can be implemented using a variety of technologies. The I / O components 1008 can include communication components 1024, which serve to connect the machine 1000 to a network 1036 or devices 1026 via a coupling 1030 or coupling 1028. The communication components 1024 can, for example, include a network interface component or another suitable device for connecting to the network 1036. In other examples, the communication components 1024 can include wired communication components, wireless communication components, cellular communication components, and other communication components to enable communication via other modalities. The devices 1026 can be another machine or any variety of peripheral devices (e.g., a peripheral device connected via a universal serial bus (USB)).As mentioned previously, the machine 1000 may correspond, for example, to the Compute Service Manager 104 or the Execution Platform 110, and the devices 1026 may include the Data Storage Device 226 or any other computer device described herein that communicates with the Data Platform 102 or the Data Storage System 106.

[0112] The various memories (e.g., 1006, 1016, 1032, and / or the memory of the processor(s) 1004 and / or the memory unit 1018) can store one or more sets of instructions 1002 and data structures (e.g., software) that embody or are used by one or more of the methods or functions described herein. When executed by the processor(s) 1004, these instructions 1002 perform various operations to implement the disclosed examples.

[0113] As used herein, the terms “machine storage medium,” “device storage medium,” and “computer storage medium” mean the same thing and may be used interchangeably in this disclosure. These terms refer to one or more storage devices and / or media (e.g., a centralized or distributed database and / or associated caches and servers) that store executable instructions and / or data. Accordingly, the terms include, but are not limited to, solid-state storage, optical, and magnetic media, including internal or external memory for processors. Specific examples of machine storage media, computer storage media, and / or device storage media include non-volatile memory, e.g., semiconductor memory devices, e.g.,Erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), field-programmable gate arrays (FPGAs) and flash memory devices, magnetic disks such as internal hard disks and removable disks, magneto-optical disks, and CD-ROM and DVD-ROM disks. The terms "machine storage media," "computer storage media," and "device storage media" expressly exclude carrier waves, modulated data signals, and other such media, at least some of which fall under the term "signal medium," which is explained below.

[0114] In various examples, one or more parts of network 1036 could be an ad-hoc network, an intranet, an extranet, a virtual private network (VPN), a local area network (LAN), a wireless LAN (WLAN), a wide area network (WAN), a wireless WAN (WWAN), a metropolitan area network (MAN), the internet, a part of the internet, a part of the public switched telephone network (PSTN), a simple legacy telephone network (POTS), a cellular telephone network, a wireless network, a Wi-Fi® network, another type of network, or a combination of two or more such networks. For example, network 1036, or a part of network 1036, could be a wireless or cellular network, and coupling 1030 could be a CDMA (Code Division Multiple Access) connection, a GSM (Global System for Mobile Communications) connection, or another type of cellular or wireless coupling.In this example, the 1030 coupling can implement any type of data transmission technology, such as Single Carrier Radio Transmission Technology (1xRTT), Evolution-Data Optimized (EVDO) technology, General Packet Radio Service (GPRS) technology, Enhanced Data rates for GSM Evolution (EDGE) technology, Third Generation Partnership Project (3GPP) including 3G, fourth generation (4G) mobile networks, fifth generation (5G) mobile networks, Universal Mobile Telecommunications System (UMTS), High-Speed ​​Packet Access (HSPA), Worldwide Interoperability for Microwave Access (WiMAX), Long Term Evolution (LTE), other standards defined by various standards organizations, other long-range protocols, or other data transmission technologies.

[0115] The instructions 1002 can be transmitted or received over the network 1036 using a transmission medium via a network interface device (e.g., a network interface component included in the communication components 1024) and using any of a number of known transmission protocols (e.g., Hypertext Transfer Protocol (HTTP)). Similarly, the instructions 1002 can be transmitted or received to the devices 1026 via a transmission medium through the coupling 1028 (e.g., a peer-to-peer coupling). The terms “transmission medium” and “signaling medium” are synonymous and may be used interchangeably in this disclosure.The terms "transmission medium" and "signal medium" are to be understood as encompassing any intangible medium capable of storing, encoding, or transmitting the instructions 1002 for execution by the machine 1000, and including digital or analog communication signals or other intangible media for facilitating communication of such software. The terms "transmission medium" and "signal medium" therefore include any form of modulated data signals, carrier waves, etc. The term "modulated data signal" refers to a signal in which one or more of its properties are set or modified in such a way as to encode information in the signal.

[0116] The various operations of the example procedures described here can be performed, at least partially, by one or more processors that are temporarily (e.g., by software) or permanently configured to perform the corresponding operations. Similarly, the procedures described here can be performed, at least partially, by a single processor. For example, at least some of the operations of the methods disclosed here can be performed by one or more processors. The execution of certain operations can be distributed among one or more processors, which may be located not only on a single machine but also distributed across a number of machines. In some examples, the processor or processors may be located at a single location (e.g.,in a home environment, an office environment, or a server farm), while in other examples the processors may be distributed across a number of locations.

[0117] In this document, the terms "a" or "an" are used, as is customary in patent documents, to include one or more than one, irrespective of other instances or uses of "at least one" or "one or more". In this document, the term "or" is used to refer to a non-exclusive "or", so that "A or B" includes "A but not B", "B but not A", and "A and B" unless otherwise specified. In the appended claims, the expressions "including" and "in which" are used as simple English equivalents of the respective terms "comprising" and "whereby". Likewise, in the following claims, the terms "including" and "comprising" are open, meaning that a system, device, article, or method that includes elements in addition to those listed after such a term in a claim still falls within the scope of that claim.

[0118] The described implementations of the subject can contain one or more features, alone or in combination, as exemplified below:

[0119] Example 1 is a machine-implemented procedure that includes: building a container image file system from a container image; running an application that accesses the container image file system; recording the directory and file access activity of the container image file system by the application during execution; dynamically restoring the container image file system using the directory and file access activity; and mounting the container image file system in its restored state during a subsequent execution of the application.

[0120] In Example 2, the subject of Example 1 includes the dynamic rebuild of the container image file system, which includes the creation of a new version of a metadata blob and one or more content blobs comprising the container image file system, wherein the one or more content blobs contain files that are grouped based on directory and file access activity.

[0121] In Example 3, the subject of one of Examples 1-2 involves partitioning two or more content blobs into a first set of content blobs that are expected to be fully used during the startup of the application, and a second set of content blobs that are not expected to be fully used during the startup of the application.

[0122] In Example 4, the subject of one of Examples 1-3 includes compressing the contents of the first set of content blobs and decompressing the contents of the first set of content blobs using inline decompression during the startup of the application.

[0123] In Example 5, the subject of one of Examples 1-4 involves performing inline decompression of compressed content blobs during assembly.

[0124] In Example 6, the subject encompasses one of Examples 1-5, with decompression being applied selectively based on expected usage patterns derived from directory and file access activity.

[0125] In Example 7, the subject of one of Examples 1-6 involves splitting large files into separate content blobs of one or more content blobs that are targeted to historically accessible page areas.

[0126] In Example 8, the subject of one of Examples 1-7 includes storing the one or more content blobs in an object store per account, which deduplicates the content blobs of the one or more content blobs within an account connected to the object store.

[0127] In Example 9, the subject comprises one of Examples 1-8, wherein the metadata blob is compressed and contains at least one of the following elements: directory hierarchy, file names, sizes, modification times, permissions, one or more unique identifiers, extended attributes, and link targets.

[0128] In Example 10, the subject comprises one of Examples 1-9, wherein the one or more content blobs have a variable size and include a header followed by one or more file segments, the header providing an index to all segments in a respective content blob of the one or more content blobs.

[0129] In Example 11, the subject of one of Examples 1-10 involves the partitioning of one or more content blobs into categories based on usage patterns.

[0130] In Example 12, the subject of one of Examples 1-11 includes preloading directory and file contents of one or more content blobs into a kernel page cache based on directory and file access activity.

[0131] In Example 13, the subject of one of Examples 1-12 involves saving a newly built container image file system with an updated version number, while a previous version of the container image file system remains active.

[0132] Example 14 is at least one machine-readable medium containing instructions which, when executed by a processing circuit, cause the processing circuit to perform operations to implement any of Examples 1-13.

[0133] Example 15 is a device comprising means for carrying out one of Examples 1-13.

[0134] Example 16 is a system for carrying out one of Examples 1-13.

[0135] Such examples of the subject matter of the invention may be referred to individually and / or collectively by the term "example" here, solely for the sake of simplicity and without any intention of voluntarily limiting the scope of this application to a single invention or concept if, in fact, more than one is disclosed. Although specific examples have been illustrated and described here, any arrangement that serves the same purpose may be used in place of the specific examples shown. This disclosure is intended to cover all adaptations or variations of the various examples. Combinations of the above examples and other examples not specifically described here will be obvious to the person skilled in the art upon review of the above description. QUOTES INCLUDED IN THE DESCRIPTION

[0000] This list of documents cited by the applicant was automatically generated and is included solely for the reader's convenience. The list is not part of the German patent or utility model application. The DPMA accepts no liability for any errors or omissions. Cited patent literature

[0000] US 63 / 718,403

[0001]

Claims

[1] Machine-implemented method, comprising: Building a container image file system from a container image; Running an application that accesses the container image file system; Recording directory and file access activities of the container image file system by the application during execution; Dynamic restoration of the container image file system using directory and file access activity, where the restoration of the container image file system results in a restored container image file system; and Mounting the newly built container image file system during a subsequent execution of the application. [2] Machine-implemented method according to claim 1, wherein the dynamic restoration of the container image file system comprises creating a new version of a Binary Large Object (Blob) for metadata and one or more content blobs comprising the container image file system, wherein the one or more content blobs contain files that are grouped based on directory and file access activity. [3] The machine-implemented method according to claim 2, which further comprises: Partitioning two or more content blobs into a first set of content blobs that are expected to be used in their entirety during application startup, and a second set of content blobs that are not expected to be used in their entirety during application startup. [4] The machine-implemented method according to claim 2 further comprises storing the one or more content blobs in an account-specific object memory, which deduplicates content blobs of the one or more content blobs within an account associated with the object memory. [5] Machine-implemented method according to claim 2, wherein the metadata blob is compressed and contains at least one of the following elements: directory hierarchy, file names, sizes, modification times, permissions, one or more unique identifiers, extended attributes and link targets. [6] Machine-implemented method according to claim 2, wherein the one or more content blobs have a variable size and comprise a header followed by one or more file segments, the header providing an index to all segments in a respective content blob of the one or more content blobs. [7] The machine-implemented method according to claim 2 further comprises partitioning the one or more content blobs into categories based on usage patterns. [8] The machine-implemented method according to claim 2 further comprises preloading directory and file contents of one or more content blobs into a kernel page cache based on directory and file access activity. [9] The machine-implemented method according to claim 1, further comprising storing a newly built container image file system with an updated version number while a previous version of the container image file system remains active. [10] A system that includes the following: at least one processor; and at least one memory that stores instructions which, when executed by the at least one processor, cause the system to perform operations that include the following Building a container image file system from a container image; Running an application that accesses the container image file system; Recording directory and file access activity on the container image file system by the application during execution; Dynamic restoration of the container image file system using directory and file access activity, where the restoration of the container image file system results in a restored container image file system; and Mounting the newly built container image file system during a subsequent execution of the application. [11] System according to claim 10, wherein the dynamic restoration of the container image file system comprises creating a new version of a metadata blob and one or more content blobs comprising the container image file system, wherein the one or more content blobs contain files that are grouped based on directory and file access activity. [12] System according to claim 11, wherein the operations further comprise: Splitting two or more content blobs into a first set of content blobs that are expected to be fully used during application startup, and a second set of content blobs that are not expected to be fully used during application startup. [13] System according to claim 11, wherein the operations further comprise splitting large files into separate content blobs of one or more content blobs aligned with historically accessed page regions. [14] System according to claim 11, wherein the operations further comprise storing the one or more content blobs in an account-specific object store, which deduplicates content blobs of the one or more content blobs within an account connected to the object store. [15] System according to claim 11, wherein the metadata blob is compressed and contains at least one of the following elements: directory hierarchy, file names, sizes, modification times, permissions, one or more unique identifiers, extended attributes and link targets. [16] System according to claim 11, wherein the one or more content blobs have a variable size and comprise a header followed by one or more file segments, the header providing an index to all segments in a respective content blob of the one or more content blobs. [17] System according to claim 11, wherein the operations further comprise preloading directory and file contents of one or more content blobs into a kernel page cache based on directory and file access activity. [18] System according to claim 11, wherein the operations further comprise saving a newly built container image file system with an updated version number, while a previous version of the container image file system remains active. [19] Machine storage medium that stores instructions which, when executed by one or more processors of a system, cause the system to perform operations which include: Building a container image file system from a container image; Running an application that accesses the container image file system; Recording directory and file access activities of the container image file system by the application during execution; Dynamic restoration of the container image file system using directory and file access activity, where the restoration of the container image file system results in a restored container image file system; and Mounting the newly built container image file system during a subsequent execution of the application. [20] Machine storage medium according to claim 19, wherein the dynamic restoration of the container image file system comprises creating a new version of a metadata blob and one or more content blobs comprising the container image file system, wherein the one or more content blobs contain files that are grouped based on directory and file access activity.