Method, system, and program for building or modifying container images

Abstract

A method is provided for building or modifying image files into a new image for deployment as a container. An intelligent layer control method is provided for building image files into a new container image while avoiding storing redundant files between read-write and read-only layers. A pre-commit processing module (PPM) filters updated files. The hash value of the updated file is compared to the hash value of the original file. If the hash values ​​are identical, the PPM updates the inode metadata of the original file with the metadata of the updated file and generates an OMD file removing the updated file. If the hash values ​​are not identical and the differ value is less than a threshold, the PPM generates an OMD file with updates to the inode metadata, content change flags, and / or delta content information reflecting the updated file and removes the updated file from the container file system. The OMD file is passed to the storage driver and merged with the original file content, overriding any changes to the metadata and / or content, to generate a single unified user view of the container file system.

Description

[Technical field]

[0001] The present disclosure relates generally to the field of building and modifying container images, and more specifically to controlling the presence of redundant files between read-write and read-only layers of images used to build or modify containers. [Background technology]

[0002] Container platforms operating within cloud computing environments have been widely adopted to build, share, and run applications. A container is a standard unit of software that packages software code and all of the software dependencies in a self-contained package, allowing applications to run quickly and reliably across multiple computing environments. Multiple containers can run on the same infrastructure and share the same operating system (OS) kernel with other containers, with each container running as an isolated process in user space. Containers are built from images that become containers at runtime. A container image is a lightweight, standalone executable package of software used to run applications and can include code, runtime, system tools, libraries, and configuration. Containers isolate software from the surrounding environment, ensuring that the software runs uniformly across environments, regardless of the infrastructure that runs the container.

[0003] An image file used to run a container is built as a series of layers. Each layer represents instructions in the image's template or a text file used to build the image itself. For example, when building an image using the Docker platform, a Dockerfile is used as the template to build the image. Each layer in an image file contains only one set of deltas from the previous layer. New layers result from adding and / or removing files from the image. Layers are stacked on top of each other, and every time a new container is created from an image, a new writable layer (called a container layer) is added on top of the read-only layer below. All changes made to a running container, such as writing new files, modifying existing files, and deleting files, are written to the thin writable container layer.

[0004] The storage driver manages the contents of image layers and writable container layers. The storage driver attempts to maximize storage efficiency by sharing and duplicating files or directories from lower layers in the image to other layers (including the writable layer) using the copy-on-write (CoW) technique. Due to limitations in storage technology and lack of image building skills, using CoW operations can become problematic over time because the images built can easily grow to unpredictable sizes and consume a lot of storage resources. Every time an image is modified during the build process, the image size can increase even if no files are added to the image. CoW triggers a copy-up operation every time a file is modified. For example, opening a file with WRITE permissions, changing file attributes, renaming a file, creating a hard link for a file, editing file contents, etc. The same file can exist multiple times in multiple layers within a single image file, because the same file is copied up to each new layer with the copy-up operation of the CoW technique.

[0005] Embodiments of the present disclosure recognize the limitations of using CoW to build container images and that images can grow in file size very quickly, even when there are minimal changes to the actual files present in the image layers. Thus, there is a need for an image building and storage solution that analyzes files between image layers, stores only minimal changes to files in an image, and provides a storage-friendly image layout that can align a user's current view of the image and container with the files that are intended to be accessible and viewable by the user. Summary of the Invention

[0005] Embodiments of the present disclosure relate to a computer-implemented method, associated computer system, and computer program product for building or modifying an image file for a container. The computer-implemented method includes inspecting, by a daemon, a file or directory of a container file system of the image file for changes to the file or directory, filtering, by the daemon, an updated file that was copied up to a read-write layer of the image file in response to inspecting that the container file system includes changes to the file or directory, and comparing, by the daemon, a hash value of an original file of the container file system from a lower read-only layer of the image file with a hash value of the updated file. and in response to comparing the hash value of the original file with the hash value of the updated file and finding that the hash value of the original file and the hash value of the updated file are identical or a difference between the hash value of the original file and the hash value of the updated file exceeds a default differ value, the daemon generates an override merge data (OMD) file including an override block and a merge block, the daemon updates metadata of the override block from inode metadata of the original file stored in a lower layer of the image file to metadata of the updated file, and the daemon removes the updated file from the container file system.

[0006] Embodiments of the computer-implemented method and related computer system and computer program products for building or modifying an image file into a new image for deployment as a container may further include comparing a difference between the hash value of the original file and the hash value of the updated file to determine whether the difference exceeds a default differ value. In response, the computer-implemented method further includes updating, by the daemon, a merge block of the OMD file with delta content information indicating that content has changed between the original file and the updated file and a pointer to a storage area containing the delta content.

[0007] In some embodiments, the storage driver of the daemon can scan the image file for each file present in the read-write layer. Upon detecting an OMD file in the read-write layer, the storage driver parses the OMD file for a content changed flag in a merge block, and upon identifying a value of the content changed flag in the OMD file that is true, the storage driver merges the content of the original file with the delta content of the OMD file and makes the content and delta content available to the user.

[0008] Further, upon parsing the OMD file and identifying a value for the content changed flag in the OMD file as false, the storage driver may make the content of the original file visible to the user, override the inode metadata of the original file with metadata in the override block of the OMD file, and make the metadata in the override block visible to the user. In some embodiments, the storage driver may not only override the inode metadata of the original file with metadata in the override block of the OMD file, but may also merge the delta content of the merge block of the OMD file with the content of the original file from the lower layer and make both the content of the original file and the delta content of the OMD file visible to the user.

[0009] The drawings contained herein are incorporated in and form a part of this specification. These drawings illustrate embodiments of the disclosure and, together with the description, serve to explain the principles of the disclosure. The drawings are merely illustrative of certain embodiments and are not intended to limit the disclosure. [Brief description of the drawings]

[0010] [Figure 1] FIG. 1 is a block diagram illustrating internal and external components of one embodiment of a computing system in which embodiments described herein may be implemented, in accordance with the present disclosure. [Diagram 2] FIG. 2 is a block diagram illustrating an extension of the computing system environment of FIG. 1, in which a computing system in accordance with the present disclosure is configured to operate in a network environment and perform the methods described herein. [Diagram 3] FIG. 1 is a block diagram illustrating a cloud computing environment according to the present disclosure. [Figure 4] FIG. 1 is a diagram of an embodiment of an abstract model layer of a cloud computing environment in accordance with the present disclosure. [Diagram 5] FIG. 1 is a functional block diagram illustrating one embodiment of a computing environment for building or modifying image files for deployment by containers in accordance with the present disclosure. [Figure 6] FIG. 1 is a high-level flowchart illustrating one embodiment of an architecture for building or modifying image files for deployment by containers in accordance with the present disclosure, and the workflow between components of the architecture. [Figure 7A] FIG. 13 is a functional block diagram illustrating one embodiment of a storage driver OM plug-in that performs override & merge operations using an original file and an OMD file located in a lower layer of an image file when no content changes have occurred between the original file and the updated file in accordance with the present disclosure. [Figure 7B]FIG. 13 is a functional block diagram illustrating one embodiment of a storage driver OM plug-in that performs override & merge operations to create a final file from an original file located in a lower layer of an image file and an OMD file located in an upper layer of the image when content changes occur between the original file and an updated file in accordance with the present disclosure. [Figure 8] FIG. 1A is a functional block diagram illustrating one embodiment of a storage driver OM plug-in that uses copy-on-write to control how an image's layers are stored and managed by a host node to present a single, unified view, and FIG. 1B is a functional block diagram illustrating one embodiment of a storage driver OM plug-in that uses copy-on-write to control how an image's layers are stored and managed by a host node, including overriding a current layer's OMD file with a file from a lower layer and merging it into a final file, within a single, unified user view. [Figure 9A] FIG. 1 is a flow diagram illustrating one embodiment of a method for building or modifying an image file for deployment by a container in accordance with the present disclosure. [Figure 9B] 9B is a continuation of the flow diagram of FIG. 9A illustrating an embodiment for building or modifying an image file for deployment by a container according to the present disclosure. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0011] The terms used herein are for the purpose of describing particular embodiments only and are not intended to limit the disclosure. As used herein, the singular forms "a", "an" and "the" are intended to include the plural forms unless the context clearly indicates otherwise. It will be further understood that the terms "comprises" and / or "comprising" as used herein indicate the presence of stated features, steps, operations, elements or components, or combinations thereof, but do not exclude the presence or addition of one or more other features, steps, operations, elements, components, or groups thereof, or combinations thereof.

[0012] The corresponding structures, materials, acts, and equivalents of all means-plus-function or step-plus-function elements within the scope of the appended claims are intended to include any structures, materials, or acts for performing a function in combination with other claim elements that are specifically claimed. The description of the present disclosure has been presented for purposes of illustration and description, but is not intended to be exhaustive or to limit the present disclosure to the disclosed form. Many changes and modifications will become apparent to those skilled in the art without departing from the scope of the present disclosure. The embodiments selected and described are intended to best explain the principles and practical applications of the present disclosure and to enable those skilled in the art to understand the present disclosure in various embodiments with various modifications as suited to the particular use contemplated.

[0013] overview

[0014] An embodiment of the present disclosure may be a computer-implemented method, system, or computer program product, or combination thereof, for building and / or modifying image files into new container images for deployment as containers. An embodiment of the present disclosure utilizes intelligent layer control while building or modifying an image that may avoid storage of redundant files between read-write and read-only layers of an image being built or modified using another base image. A daemon of a container operating system may receive a build request or a request to modify an existing image file, and upon receiving such a request, the daemon may pull and deploy the base image being used to create the new image.

[0015] A base image deployed as a container may trigger a daemon's pre-commit processing module (PPM) before performing a commit operation to commit the base image changes to a new image if the container file system of the image file contains changes to files in the file system. When triggered during the pre-commit phase, the PPM may perform a filtering operation to filter updated files that may have been copied up to the image's read-write layer and that differ from the original files located in the image's lower read-only layer as a result of changes to the files. The updated files may be filtered based on file type, file size, or other properties, parameters, or attributes, or a combination thereof. Once filtered, an embodiment of the PPM may compare the hash value of the original file from the image's lower read-only layer with the hash value of the updated file that was copied up to the read-write layer. If the hash values ​​are identical, this may indicate that the metadata of the updated file has changed from the original file. The PPM may generate an override merge data structure (OMD) file. As part of the OMD file generation, the PPM may update the inode metadata of the original file with the metadata of the updated file and store the metadata of the updated file in the override block of the OMD file. The PPM may remove the updated file from the container file system using the rm command or other removal method. The newly generated OMD file is provided to the storage driver OM plugin, allowing the storage driver to place the OMD file in the read-write layer (i.e., the container layer) for a commit operation, so that the metadata of the OMD file can override the inode metadata provided by the original file located in the lower layer of the existing image file.As a result of the commit operation, the read-write layer is added as a new top layer to the new image.

[0016] In some cases, when comparing the hash value of the original file with the hash value of the updated file, the difference in the hash values ​​can be measured against a default differ value (threshold) that can be manually set by a user or administrator. In situations where the difference in the hash values ​​exceeds the default differ value, the file changes of the updated file may be large and / or extensive compared to the original file in the read-only layer, and standard CoW and commit operations can be performed to commit the file changes of the image file to the new image. Similarly, if the comparison of the hash value of the original file with the hash value of the updated file does not exceed the threshold set by the default differ value, the PPM can generate an OMD file containing the updated metadata of the updated file, overriding the inode metadata of the original file, as described above. Additionally, the OMD file can also include a content change flag indicating whether the content of the updated file differs from the original file, and a delta pointer indicating the location of the content changes (delta content) that differ from the original file. Upon generation of the OMD file, the PPM can remove the updated files from the image file container file system and provide the OMD file to the storage driver OM plugin for placement in the container's read-write layer.

[0017] In embodiments of the storage driver OM plugin, the OMD file and image layering may be used to provide a single, unified view of the image to the user. For example, a container's Union Filesystem may ensure that the user can see what contents of the container. The file system may generate the user view from the container layer and the lower read-only layers. For example, by mapping files from the lower layers of the image file to the container layer and / or by placing one or more whiteout files in the container layer to hide the presence of files in the deleted read-only layer. In embodiments of the storage driver, the OMD file and files from the read-only layer may be used to perform override and merge operations to minimize the use of disk space consumed by the new image being created. In embodiments of the storage driver, the container layer may be scanned for OMD file names (i.e., .om.filename) placed by PPM. Found OMD files may be parsed by the storage driver for a content change (CC) flag whose value is true or false. If the CC flag is not present in the merge block of the OMD file (i.e., its value is "false"), the storage driver can copy-up the file contents from the original file in the layer below the image layer to the container layer and override the metadata information of the original file with the metadata in the override block of the OMD file.

[0018] Similarly, if the parsed OMD file contains a CC flag with a value of "true", the delta content referenced in the merge block of the OMD file can be merged with the original file's content from the lower layer of the image file, and both the content and the delta content are visible to the user. Additionally, the OMD file's metadata can override the original file's inode metadata, in which case the OMD file's metadata, rather than the inode metadata, is visible to the user. Once the container layer is finalized, the daemon can commit the container layer to the new image using the commit operation to generate a new image, and the container layer is written as a new top layer (read-only layer) of the new image.

[0019] Computing Systems

[0020] The present invention may be a system, method, or computer program product, or a combination thereof, in any possible level of technical detail integration. The computer program product may include a computer-readable storage medium having computer-readable program instructions for causing a processor to perform aspects of the present invention.

[0021] A computer readable storage medium may be any tangible device capable of holding and storing instructions for use by an instruction execution device. A computer readable storage medium may be, for example, but not limited to, an electronic storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination thereof. A non-exhaustive list of more specific examples of computer readable storage media includes: portable computer diskettes, hard disks, random access memory (RAM), read only memory (ROM), erasable programmable read only memory (EPROM or flash memory), static random access memory (SRAM), portable compact disk read only memory (CD-ROM), digital versatile disk (DVD), memory sticks, floppy disks, mechanically encoded devices such as punch cards or grooved structures on which instructions are recorded, and any suitable combination of the foregoing. As used herein, a computer-readable storage medium should not be construed as a transitory signal itself, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through a waveguide or other transmission medium (e.g., light pulses passing through a fiber optic cable), or electrical signals transmitted over electrical wires.

[0022] The computer readable program instructions described herein may be downloaded from a computer readable storage medium to an individual computing / processing device or to an external computer or storage device over a network, such as the Internet, a local area network, a wide area network, or a wireless network, or a combination thereof. The network may include copper transmission cables, optical transmission fiber, wireless transmission, routers, firewalls, switches, gateway computers, and / or edge servers. A network adapter card or network interface of each computing / processing device receives the computer readable program instructions from the network and forwards the computer readable program instructions for storage in a computer readable storage medium within the individual computing / processing device.

[0023] The computer readable program instructions for carrying out the operations of the present invention may be either source code or object code written in any combination of one or more programming languages, including assembler instructions, instruction set architecture (ISA) instructions, machine instructions, machine dependent instructions, microcode, firmware instructions, state configuration data, configuration data for integrated circuits, or object oriented programming languages ​​such as Smalltalk®, C++, and procedural programming languages ​​such as the “C” programming language or similar programming languages. The computer readable program instructions may be executed entirely on the user's computer, partly on the user's computer as a standalone software package, partly on the user's computer and partly on a remote computer, or entirely on a remote computer or server. In the latter scenario, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or wide area network (WAN), or the connection may be made to an external computer (e.g., via the Internet using an Internet Service Provider). In some embodiments, electronic circuitry including, for example, a programmable logic circuit, a field programmable gate array (FPGA), or a programmable logic array (PLA) can execute computer readable program instructions to individualize the electronic circuitry by utilizing state information of the computer readable program instructions to perform aspects of the invention.

[0024] Aspects of the present invention are described herein with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems) and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, are implemented by computer readable program instructions.

[0025] These computer readable program instructions may be provided to a processor of a computer or other programmable data processing apparatus to produce a machine, such that the instructions, which execute on the processor of the computer or other programmable data processing apparatus, create means for implementing the functions / operations specified in one or more blocks of the flowcharts and / or block diagrams. These computer readable program instructions may also be stored on a computer readable storage medium having instructions stored thereon, directing a computer, programmable data processing apparatus, or other device, or combination thereof, to function in a particular manner, such that the computer readable storage medium having the instructions stored thereon comprises an article of manufacture including instructions for implementing aspects of the functions / operations specified in one or more blocks of the flowcharts and / or block diagrams.

[0026] The computer readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device to cause the computer, other programmable apparatus, or other device to perform a series of operable steps to create a computer-implemented process such that the instructions, which execute on the computer, other programmable apparatus, or other device, implement the functions / acts specified in one or more blocks of the flowcharts and / or block diagrams.

[0027] The flowcharts and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present invention. In this regard, each block in the flowcharts or block diagrams may represent a module, segment, or portion of instructions, including one or more executable instructions for implementing the specified logical function. In some alternative implementations, the functions shown in the blocks may occur out of the order shown in the figures. For example, two blocks shown in succession may actually be performed as one step, or may be executed simultaneously, substantially simultaneously, partially or fully in a time-overlapping manner, or the blocks may sometimes be executed in reverse order depending on the functionality involved. It should also be noted that each block in the block diagrams and / or flowchart illustrations, and combinations of blocks in the block diagrams and / or flowchart illustrations, are implemented by a special-purpose hardware-based system that performs the specified functions or actions or executes a combination of special-purpose hardware and computer instructions.

[0028] Although the description of various embodiments of the present invention has been presented for illustrative purposes, the description is not intended to be exhaustive or limited to the disclosed embodiments. Many changes and modifications will be apparent to those skilled in the art without departing from the scope of the present invention. The terms used in this specification are selected to best explain the principles of the embodiments, practical applications, or technical improvements beyond the technology found in the market, or to enable those skilled in the art to understand the embodiments disclosed herein.

[0029] FIG. 1 is a block diagram illustrating an embodiment of a computing system 100, which may be a simplified example of a computing device (i.e., a physical bare metal system and / or a virtual system) capable of performing the computing operations described herein. The computing system 100 may be representative of one or more computing systems or devices implemented as described further below in accordance with embodiments of the present disclosure. It should be appreciated that FIG. 1 merely provides an illustration of one implementation of the computing system 100 and does not imply any limitation with respect to the environment in which various implementations may be implemented. In general, the components illustrated in FIG. 1 may represent any electronic device, whether physical or virtualized, capable of executing machine-readable program instructions.

[0030] 1 illustrates an example of a computing system 100, the computing system 100 can take many different forms, such as bare metal and virtualized computing systems, including systems having container-oriented and microservices-oriented architectures. For example, the computing system 100 can take the form of a desktop computing system, a laptop, a notebook, a tablet, a server, a client device, a network device, a network terminal, a thin client, a thick client, a kiosk, a mobile communication device (e.g., a smartphone), a multiprocessor system, a microprocessor-based system, a minicomputer system, a mainframe computing system, a smart device, or an Internet of Things (IoT) device, or combinations thereof. The computing system 100 can operate in a local computing environment, a networked computing environment, a containerized computing environment including one or more clusters, pods, or containers, or a distributed cloud computing environment, or combinations thereof, and can include any of the systems or devices described herein or additional computing devices or systems known or used by those of skill in the art, or combinations thereof.

[0031] Computing system 100 may include a communications fabric 112 that may provide electronic communications between one or more processors 103, memory 105, persistent storage 106, cache 107, communications units 111, and one or more input / output (I / O) interfaces 115. Communications fabric 112 may be implemented with any architecture designed to pass data and / or control information between processors 103 (such as microprocessors, CPUs, and network processors), memory 105, external devices 117, and any other hardware components within computing system 100. For example, communications fabric 112 may be implemented as one or more buses, such as an address bus and a data bus.

[0032] Memory 105 and persistent storage 106 may be computer readable storage media. An embodiment of memory 105 may include random access memory (RAM) or cache 107 memory, or both. In general, memory 105 may include any suitable volatile or non-volatile computer readable storage medium and may include firmware or other software programmed into memory 105. Programs 114, applications, processes, services, and installed components thereof, as described herein, may be stored in memory 105 and / or persistent storage 106 for execution and / or access by one or more of the respective processors 103 of computing system 100.

[0033] Persistent storage 106 may include multiple magnetic hard disk drives, solid state hard drives, semiconductor storage devices, read only memory (ROM), erasable programmable read only memory (EPROM), flash memory, or any other computer readable storage medium capable of storing program instructions or digital information. Embodiments of the media used by persistent storage 106 may also be removable. For example, a removable hard drive may be used for persistent storage 106. Other examples include optical and magnetic disks, thumb drives, and smart cards that are inserted into a drive to transfer to another computer readable storage medium that is also part of persistent storage 106.

[0034] The communication unit 111 is provided to facilitate electronic communication between the computing system 100. For example, it facilitates electronic communication between one or more computer systems or devices over a communication network. In an exemplary embodiment, the communication unit 111 may include a network adapter or network interface, such as a TCP / IP adapter card, a wireless interface card, or other wired or wireless communication link. The communication network may include, for example, copper wire, optical fiber, wireless transmission, routers, load balancers, firewalls, switches, gateway computers, edge servers, or other network hardware or combinations thereof, which may be part of or connected to nodes of a communication network, including devices, host systems, terminals, or other network computer systems. Software and data used to practice embodiments of the present disclosure may be downloaded to the computing system 100 operating in a network environment (e.g., via the Internet, a local area network, or other wide area network) via the communication unit 111. The software and data of the programs 114 or applications may be loaded from the communication unit 111 into the persistent storage 106.

[0035] One or more I / O interfaces 115 allow for input and output of data to and from other devices connected to the computing system 100. For example, the I / O interface 115 can provide a connection to one or more external devices 117, such as one or more smart devices, IoT devices, recording systems such as camera systems or sensor devices, input devices such as keyboards, computer mice, touch screens, virtual keyboards, touch pads, pointing devices, or other human interface devices. The external devices 117 can also include portable computer-readable storage media, such as thumb drives, portable optical or magnetic disks, and memory cards. The I / O interface 115 can be connected to a human-readable display 118. The human-readable display 118 provides a mechanism for displaying data to a user and can be, for example, a computer monitor or computer screen, such as by displaying data as part of a graphical user interface (GUI). The human-readable display 118 can also be an embedded display and can function as a touch screen, such as the built-in display of a tablet computer.

[0036] FIG. 2 illustrates an expansion of the computing system 100 environment shown in FIG. 1 to illustrate that the methods described herein may be implemented in a variety of computing systems operating within a networked environment. Types of computing systems 100 range from small handheld devices, such as handheld computer / cell phone 110, to large mainframe systems, such as mainframe computer 170. Examples of handheld computers 110 include personal entertainment devices such as personal digital assistants (PDAs), Moving Picture Experts Group Layer-3 Audio (MP3) players, portable televisions, and compact disc players. Other examples of information handling systems include pen or tablet computers 120, laptop or notebook computers 130, workstations 140, personal computer systems 150, and servers 160. Other types of information handling systems not shown separately in FIG. 2 are represented by information handling system 180.

[0037] Many computing systems may include a non-volatile data store, such as a hard drive and / or non-volatile memory. The embodiment of the information handling system shown in FIG. 2 includes separate non-volatile data stores (more specifically, server 160 utilizes non-volatile data store 165, mainframe computer 170 utilizes non-volatile data store 175, and information handling system 180 utilizes non-volatile data store 185). The non-volatile data stores may be components external to the various computing systems or may be internal to one of the computing systems. In addition, removable non-volatile storage device 145 may be shared between two or more computing systems using various techniques, such as connecting removable non-volatile storage device 145 to a USB port or other connector of a computing system. In some embodiments, the network of computing systems 100 may utilize clustered computing and components that function as a single pool of seamless resources when accessed over network 250 by one or more computing systems. For example, such embodiments may be used in data centers, cloud computing networks, storage area networks (SANs), and network attached storage (NAS) applications.

[0038] As shown, the various computing systems 100 may be networked using a computer network 250 (referred to herein as “network 250”). Types of networks 250 that may be used to interconnect the various information handling systems may include a local area network (LAN), a wireless local area network (WLAN), a home area network (HAN), a wide area network (WAN), a backbone network (BBN), a peer-to-peer network (P2P), a campus network, an enterprise network, the Internet, a single tenant or multi-tenant cloud computing network, a public switched telephone network (PSTN), and any other network or network topology known by one of ordinary skill in the art for interconnecting computing systems 100.

[0039] Cloud computing is a service delivery model for enabling convenient, on-demand network access to a shared pool of configurable computational resources (e.g., networks, network bandwidth, servers, processing, memory, storage, applications, virtual machines, and services) and for rapidly provisioning and releasing these resources with minimal administrative effort or interaction with the service provider. The cloud model can include at least five characteristics, at least three service models, and at least four deployment models.

[0040] The features are as follows:

[0041] On-Demand Self-Service: Cloud consumers can unilaterally provision computing capabilities, such as server time and network storage, automatically as needed, without the need for human interaction with the service provider.

[0042] Broad network access: Cloud capabilities are available over the network and can be accessed using standard mechanisms, facilitating usage by heterogeneous thin- or thick-client platforms (e.g., mobile phones, laptops, and PDAs).

[0043] Resource Pool: The provider's computing resources are pooled and offered to multiple consumers using a multi-tenant model, with different physical and virtual resources dynamically allocated and reallocated according to demand. Consumers typically have a sense of location independence in that they have no control or knowledge regarding the exact location of the resources offered to them, although at a higher level of abstraction they may be able to specify a location (e.g., country, state, or data center).

[0044] Rapid Elasticity: Capacity is provisioned quickly and elastically, in some cases automatically, so it can be quickly scaled out and quickly released to quickly scale in. Capacity available for provisioning often appears to the consumer as unlimited, available for purchase in any quantity at any time.

[0045] Metered Services: Cloud systems leverage metering to automatically control and optimize resource usage at a level of abstraction appropriate to the type of service (e.g., storage, processing, bandwidth, and active user accounts). Resource usage can be monitored, controlled, and reported, providing transparency to both providers and consumers of the services utilized.

[0046] The service model is as follows:

[0047] SaaS (Software as a Service): The capability provided to the consumer is the use of the provider's applications running on a cloud infrastructure. Those applications can be accessed from a variety of client devices through thin-client interfaces such as web browsers (e.g., web-based email). The consumer does not manage or control the underlying cloud infrastructure, including the network, servers, operating systems, storage, or individual application functionality, except for the possibility of limited user-specific application configuration settings.

[0048] PaaS (Platform as a Service): The capability provided to a consumer is to deploy applications that the consumer creates or acquires, written using programming languages ​​and tools supported by the provider, onto a cloud infrastructure. The consumer does not manage or control the underlying cloud infrastructure, including networks, servers, operating systems, or storage, but does have control over the deployed applications and, in some cases, the configuration of the application hosting environment.

[0049] Infrastructure as a Service (IaaS): The capability provided to a consumer is the provisioning of processing, storage, network, and other basic computing resources onto which the consumer can deploy and run any software, which may include operating systems and applications. The consumer does not manage or control the underlying cloud infrastructure, but has control over the operating systems, storage, deployed applications, and in some cases, limited control over selected network components (e.g., host firewalls).

[0050] The deployment model is as follows:

[0051] Private Cloud: This cloud infrastructure is operated solely for an organization. It is managed by that organization or a third party and can exist on-premise or off-premise.

[0052] Community Cloud: This cloud infrastructure is shared by multiple organizations to support a specific community with shared concerns (e.g., mission, security requirements, policy, and compliance considerations). It may be managed by the organization or a third party and may reside on-premise or off-premise.

[0053] Public Cloud: The cloud infrastructure is made available to the general public or large industry organizations and is owned by an organization that sells cloud services.

[0054] Hybrid Cloud: The cloud infrastructure is a composite of two or more clouds (private, community, or public) that remain a unique entity but are joined by standardized or proprietary technologies that enable data and application portability (e.g., cloud bursting for load balancing between clouds).

[0055] A cloud computing environment is a service-oriented environment with an emphasis on statelessness, loose coupling, modularity, and semantic interoperability. At the heart of cloud computing is an infrastructure that includes a network of interconnected nodes.

[0056] Referring now to the drawings, FIG. 3 is an illustration of a cloud computing environment 300. As shown, the cloud computing environment 300 includes a cloud network 350 including one or more cloud computing nodes 310 that are used by cloud subscribers for user devices 305a-305n (generally referred to herein as end user devices 305) to access one or more software products, services, applications, and / or workloads provided by a cloud service provider or tenant of the cloud network 350. Examples of user devices 305 are depicted and include devices such as desktop computers, laptop computers 305a, smartphones 305b or mobile phones, tablet computers 305c, and smart devices such as smart watches 305n and smart glasses. The nodes 310 can communicate with each other and can be grouped together physically or virtually in one or more networks, such as private, community, public, or hybrid clouds, or combinations thereof, as described herein above (not shown). This allows the cloud computing environment 300 to provide an infrastructure, platform, and / or SaaS that does not require cloud consumers to maintain resources on local computing devices. It will be understood that the types of end user devices shown in Figure 3 are intended to be merely exemplary, and that the computing nodes 310 of the cloud computing environment 300 can communicate with any type of computerized device over any type of network and / or network addressable connections (e.g., using a web browser).

[0057] Referring now to Figure 4, there is shown a set of functional abstraction layers provided by cloud computing environment 300. It should be understood in advance that the components, layers, and functions shown in Figure 4 are intended to be illustrative only and that embodiments of the present invention are not limited thereto. As shown, the following layers and corresponding functions are provided:

[0058] Hardware and software layer 460 includes hardware and software components. Examples of hardware components include mainframes 461, RISC (reduced instruction set computer) architecture based servers 462, servers 463, blade servers 464, storage devices 465, and networks and networking components 466. In some embodiments, software components include network application server software 467 and database software 468.

[0059] The virtualization layer 470 provides an abstraction layer at which the following examples of virtual entities are provided: virtual servers 471 , virtual storage 472 , virtual networks including virtual private networks 473 , virtual applications and operating systems 474 , and virtual clients 475 .

[0060] The management layer 480 may provide the following functions: Resource provisioning 481 provides dynamic procurement of computing and other resources utilized to execute tasks within the cloud computing environment 300. Metering and pricing 482 provides cost tracking and billing or invoices for the utilization of resources within the cloud computing environment 300. In one example, the resources may include application software licenses. Security provides identity verification of cloud users and tasks, as well as protection of data and other resources. User portal 483 provides access to the cloud computing environment 300 for users and system administrators. Service level management 484 provides allocation and management of cloud computing resources to meet required service levels. Service level agreement (SLA) planning and execution 485 provides pre-provisioning and procurement of cloud computing resources in anticipation of upcoming demands and in accordance with SLAs.

[0061] The Workload layer 490 provides examples of functionality that can leverage cloud computing environments to deploy containerized workloads to users. Examples of workloads and functionality that come from this layer include: software development and lifecycle management 491, data analytics processing 492, multi-cloud management 493, transaction processing 494, database management 495, and video conferencing 496.

[0062] A system for constructing and modifying image files with intelligent layer control

[0063] It will be readily understood that the example components as generally described and illustrated in the figures herein could be arranged and designed in a wide variety of configurations. Thus, the following detailed description of at least one embodiment of a method, apparatus, non-transitory computer-readable medium, and system, as represented in the accompanying figures, is not intended to limit the scope of the claimed application, but is merely representative of selected embodiments.

[0064] The example features, structures, or characteristics described throughout this specification may be combined or removed in any suitable manner in one or more embodiments. For example, use of the phrase "exemplary embodiment," "some embodiments," or other similar phrases throughout this specification refers to a particular feature, structure, or characteristic associated with an embodiment being included in at least one embodiment. Thus, the appearance of the phrase "exemplary embodiment," "some embodiments," "in other embodiments," or other similar phrases throughout this specification do not necessarily all refer to the same group of embodiments, and the described features, structures, or characteristics may be combined or removed in any suitable manner in one or more embodiments. Furthermore, in the figures, even if the connections depicted are one-way or two-way arrows, any connection between elements may enable one-way or two-way communication, or both. Also, any device depicted in the figures may be another device. For example, where a mobile device is shown transmitting information, a wired device may also be used to transmit information.

[0065] Referring to the drawings, FIG. 5 illustrates an embodiment of a computing environment 500 in which images 519, 520 can be built and modified using intelligent layer control for deployment as containerized workloads 525 executing within a container runtime environment 521 of a host node 501. As depicted in the example embodiment of FIG. 5, the computing environment 500 of a container system can include multiple components arranged to communicate with each other. For example, in the example embodiment, the computing environment 500 includes components such as one or more clients 503, a host node 501, a registry 515, and / or a user device 305. The clients 503 can be the primary way for administrators and users who run and manage the container system to interact and communicate with each daemon 509. The clients 503 can send commands to the daemons 509 via a command line interface (CLI) 505 and / or a remote API 507 (e.g., REST API), over a network interface or over a UNIX® socket, instructing the daemons 509 to execute the commands. Examples of commands submitted by client 503 may include, but are not limited to, "build," "pull," "push," and "run" commands. Although FIG. 5 depicts client 503 communicating with a single daemon 509, embodiments of client 503 may communicate with multiple daemon 509 processes that may run on a single host node 501 or across multiple host nodes 501. Moreover, client 503 may reside on the same host node 501 as daemon 509 or may reside on a remote host that is positioned to communicate with one or more daemons 509.

[0066] The host node 501 may be described as a computing system 100 (physical or virtual) that hosts a daemon 509. Examples of the host node 501 may be any computing system 100 described above, such as a laptop, desktop, server, virtual machine thereof, or computational resources provided by the cloud network 350, or a combination thereof. The daemon 509 may listen for incoming commands and API requests from the client 503 and manage the objects of the container system, including images, containers 523, networks, and storage volumes (such as local storage 511). The daemon 509 may be responsible for building and running containers 523 from images in the container runtime environment 521. The images may reference read-only templates that contain instructions for creating containers 523. Images may consist of layers of images that are cached at the host node 501 and shared across multiple containers 523. An embodiment of a container 523 may be described as an executable instance of an image. Containers 523 can be created, started, stopped, moved, or deleted by a client 503 using a CLI 505 or a remote API 507. Containers 523 can be connected to one or more networks, persistent storage 106 (such as local storage 511 or other non-volatile data stores 165, 175, 185, or a combination thereof). Containers 523 can also be created in a new image 520 based on the current state of the container 523. A container 523 can be defined by its image and by any configuration options provided to the container when the container 523 is created or started. When a container 523 is removed, any changes to the state of the container 523 that are not stored in persistent storage 106 may be lost.

[0067] Image files used to create containers 523 may be built and modified by users of the container system or other users who may share image files publicly or privately. Image files may be published to a registry 515 where the images may be stored in one or more image repositories 517a, 517b, ..., 517n (generally referred to herein as "image repositories 517"), such as image libraries. Registry 515 may be part of host node 501 or may be a location external to host node 501 and may include a pointer to the storage location of the image file in one of image repositories 517. An embodiment of registry 515 may be a public registry or a private registry. Upon execution of a build, pull, or run command, daemon 509 may retrieve the requested image from registry 515 and store the pulled image 519 locally in local storage 511 of host node 501. In contrast, when a push command is executed by a client 503, a previously pulled image 519 or a new image 520 is sent from local storage 511 to the registry 515, where it may be stored by one or more image repositories 517.

[0068] An embodiment of the daemon 509 may be instructed to build a new image from an existing base image or to run an instance of an existing image. Upon execution of a command by the client 503, the daemon 509 may pull the image needed to fulfill the client 503 request from the registry 515 and deploy an instance of the image as a container 523. If the image is already present in the local storage 511, the previously pulled image 519 may be run without needing to pull the image again from the registry 515. Upon running the pulled image 519, the daemon 509 may create one or more containers 523 in the container runtime environment 521. The container 523 may include all applications needed for the workload 525 and associated libraries 527 needed to run the workload 525 of the container 523. The user device 305 may access and use the running workload 525 or the updated workload 526 of the container runtime environment 521 via an application user interface (UI), such as an application UI 531, as shown in FIG. 5.

[0069] An embodiment of a system configured to build or modify an image file, such as the system depicted in computing environment 500, may include a pre-commit module 510 and a storage driver OM plugin 512. The term "module" may refer to a hardware module, a software module, or a module may be a combination of hardware or software resources or both. An embodiment of a hardware-based module may include a self-contained component, such as a chipset, special purpose circuitry, one or more memory 105 devices and / or persistent storage 106 devices. A software-based module may be part of or linked to a program 114 that includes specifically programmed instructions loaded into a memory 105 device or persistent storage 106 device of one or more computing systems 100 operating as part of computing environment 500, program code. An embodiment of the pre-commit module 510 and storage driver OM plugin 512 may be integrated into a daemon 509, in some embodiments, as part of a daemon process or as a plugin, or both. In alternative embodiments, the pre-commit processing module 510 and / or the storage driver OM plugin 512 may be a process running separately within the memory 105 of the host node 501, may be in remote communication with the host node 501, or both.

[0070] An embodiment of a pre-commit processing module 510 (also referred to herein as “PPM 510”) may be triggered prior to performing a commit operation and may be responsible for performing pre-commit functions. The pre-commit functions may be directed to filtering updated files based on file size and / or file type, performing a hash comparison between the original and updated files located in the container layer 601 of the pulled image 519, and updating the inode metadata, CC flags, and / or delta content information to generate an OMD file 620. An embodiment of a storage OM driver plug-in 512 (also referred to herein as “storage driver 512”) may receive the OMD file 620 generated by the PPM 510, place the OMD file 620 in the container layer 601, and undergo a commit process, thereby committing the OMD file 620 to a new image file as part of a new image layer. The storage driver 512 may parse the OMD file 620 for file content 723 changes between the original file and the update file of the image, merge the delta file content 623 of the update file with the delta file content 623 of the original file, override the file inode metadata 721 of the original file to display the latest file metadata 621 of the update file in the user view 711 of the container 523, and apply a whiteout file 603 to hide files that are not intended to be seen in the user view 711 of the container 523. An embodiment of the storage driver 512 may further apply the OMD file 620 to the process of deploying the new image 520 to a new container 524.

[0071] FIG. 6 provides a detailed diagram illustrating the architecture and process by which the daemon 509 creates a new image 520 for deployment as a new container 524 from a pulled image 519 currently deployed as a container 523. However, in other embodiments, the depicted process may also be applied to building a new image using a template file (i.e., a Docker file or other text file containing build instructions, etc.). The embodiment illustrates that the daemon 509 operates by intelligently controlling the files of the image files present in layers 0, 1, 2 using the PPM 510 and storage driver 512 as shown to reduce redundant files between the container layer 601 (a read-write layer) and the read-only layers 0, 1, 2 of the pulled image 519. During the pre-commit phase of building or modifying an image file, an embodiment of the PPM 510 may inspect the file system of the container 523 for file or directory changes. For example, in Docker, the docker diff command may be used, specifying the container ID of the container to be inspected. Examination of a container 523 may enumerate changes to the files and directories of the container's file system since the container 523 was created. An example of an examination of a container's 523 file system is shown in Figure 6, where files depicted with "A" represent the addition of a file or directory at a particular layer, files depicted with "D" represent files or directories that have been deleted, and files depicted with "C" represent files or directories that have been modified from files in a previous layer.

[0072] An embodiment of the PPM 510 may apply a filter operation 607 to files or directories of an image file that are specified during inspection to include changes to files or directories in the read-only layers (layers 0, 1, and 2) of the image file that are copied up to the container layer 601. As shown in FIG. 6, during inspection of a pulled image 519 that is being used as a base image to create a new image 520, layer 2 is the top layer 650 of the pulled image 519 and includes three files or directories that have been copied up to the container layer 601 that are marked as having changes compared to the files in the read-only lower layers (i.e., layers 1 or 0), referred to as the original files. The filter operation 607 may be applied to the updated files that are copied up to the container layer 601 depending on the type of file and the size of the file. As shown in FIG. 6, the filtered group 604 includes updated files in the upper layer 650 that have been copied up to the container layer 601, and thus is selected for filtering and additional pre-commit processing by the PPM 510 before committing the file changes of the container 523 to the new image 520. Meanwhile, newly added files present in the upper layer 650 can be copied up to the container layer 601 and committed by the daemon 509 using the conventional commit process 605 without being pre-processed by the PPM 510. Moreover, some files or directories in the upper layer 650 are not copied up to the container layer 601. For example, files in the group 602 that are being deleted in layer 2 have a whiteout file 603 applied to prevent the deleted files from being copied up and becoming visible in the container layer 601. Once the whiteout file 603 is applied, the daemon can proceed with the commit for the changes to which the whiteout file 603 has been applied by submitting a delete action to the conventional commit process 605.

[0073] The filtered group 604 of files being processed during the pre-commit phase by the PPM 510 may be subjected to a hash value comparison step 609. During the hash value comparison step 609 of processing during the pre-commit phase, the PPM 510 may compare the hash value of the updated file being copied up to the container layer 601 with the original file from the lower layers 709 of the image for each file or directory identified in the read-only layer as having been modified from the original file (located in the lower layers 709 of the image file). In step 611, the PPM module 510 may determine whether the hash value of the updated file and the hash value of the original file are identical based on the comparison of the hash values ​​in step 609. In a situation where the updated file and the original file have identical hash values, the PPM 510 proceeds to step 617, where the PPM 510 generates an OMD file 620 and updates the file inode metadata 721 of the original file in the lower layers 709 of the pulled image 519 to the updated file metadata 621 of the updated file. After generating the OMD file 620 and applying the updates to the file inode metadata 721, the PPM 510 may remove the updated file from the container's file system.

[0074] Alternatively, during the comparison of hash values ​​in step 609, if PPM 510 detects that the hash values ​​of the original file and the updated file are not identical in step 611, PPM 510 may further compare the difference in the hash values ​​(i.e., the differ value) to a threshold value set by default and / or set by a user or administrator of the container system (default differ value), as depicted by step 613. In a situation where the differ value between the hash of the original file and the hash of the updated file exceeds the threshold value, the conventional commit process 605 may be used to commit the changes or settings in the updated file to the new image 520. However, if the difference between the hash value of the original file and the hash value of the updated file does not exceed the threshold value, PPM 510 may proceed to step 615 and generate an OMD file 620. In the OMD file 620, the PPM 510 may update the file inode metadata 721 to the file metadata 621 of the updated file and may also update the OMD file 620 to have a value of the CC flag set to true and a delta pointer to point to the location of the delta content that is different from the file content 723 of the original file in the lower layer 709. Once the OMD file 620 is created and updated, the updated file processed by the PPM 510 may be removed by the PPM 510. For example, a rm command may be used to remove or unlink a file or directory from the updated file. The OMD file 620 created by the PPM 510 may be provided to the storage driver 512 placement to the commit process 605 and / or applied to the process of deploying the new image 520 as a new container 524.

[0075] An example of the structure of an OMD file 620 is shown in Figures 6-7B. An embodiment of an OMD file 620 may include an override block and a merge block. The override block may include file metadata 621 for the updated file used to generate the OMD file 620. Examples of metadata tracked and stored by the file metadata 621 of the override block may include, but are not limited to, changes in access mode, changes in user ID (UID) and / or group ID (GID), changes in content size, resaving of the file resulting in an updated access date, etc. An embodiment of a merge block may include delta file contents 623, which may include a content change flag indicating whether the file contents have changed between the updated file and the original file, and a delta pointer, which may point to a storage area containing the delta file contents 623 that differ between the original file contents and the updated file contents.

[0076] An embodiment of storage driver 512 may receive an OMD file 620 from PPM 510 and place the OMD file 620 in the container layer 601 of the image file, replacing any update files removed by PPM 510. An embodiment of OMD file 620 may be distinguished from the original file and the updated file by the filename structure. An embodiment of OMD file 620 may use the prefix ".om." before the filename. For example, the file may be named .om.filename, where "filename" may be any name for the file selected by the user, and the "filename" portion of the filename may be the same as the filename of the original file in the lower layer, or the updated file replaced by OMD file 620, or both. For example, a file named "infer.log" generated as OMD file 620 may be named ".om.infer.log".

[0077] An embodiment of the storage driver 512 may parse and apply the OMD file 620 to make the entire file available for the user to view and use once the new image 520 is created. An embodiment of the storage driver 512 may incorporate the merge and override blocks of the OMD file 620 into the user view generated by the file system of the container by performing an “override&merge” function. An embodiment of the override&merge function may be implemented in two scenarios. The first scenario is when there is no change in the file contents 723 and only differences exist between the file inode metadata 721 of the original file and the file metadata 621 of the updated file, resulting in the file inode metadata 721 being overridden with the file metadata 621 stored by the OMD file 620. A second scenario in which the override & merge functionality may be applied is when minor changes are made to the content of the copy-up file (i.e., the updated file) in the read-only layer and the file content 723 of the original file, and the size of the resulting updated file incorporating the content changes is large (e.g., 100MB, 500MB, or even 1GB or more).

[0078] The override operation of the override&merge feature can be applied to the override block of the OMD file 620 and the metadata of the original file. The override operation ensures that the file metadata 621 of the OMD file 620 is in the container layer 601 that is committed to the new layer of the new image 524, and that when the user view 711 is generated by the container's file system, the user sees the file metadata 621 instead of the existing file inode metadata 721 of the original file. On the other hand, the merge operation can combine the file contents 723 of the original file with the delta file contents 623 of the OMD file 620 when changes to the file contents 723 occur as a result of a difference between the original and updated file contents that led to the generation of the OMD file 620. If the file contents 723 change, the merge block of the OMD file 620 will contain a content change flag indicating that a change (i.e., a delta) to the file contents 723 has occurred. The presence of the CC flag may be indicated by a value of "true" and the delta pointer may identify in the merge block where the delta contents are stored. During the merge operation, the file contents 723 of the original file along with the delta file contents 623 of the OMD file 620 may be mapped into the container layer 601 and made viewable to the user by the container file system through the user view 711. In the absence of the CC flag, there are no changes to the file contents 723 and therefore only the file contents 723 are available to be viewed by the container file system in the user view 711.

[0079] An embodiment of the storage driver 512 can selectively apply override & merge features to files of an image by identifying an OMD file 620 in the container's read-write layer. The storage driver can scan files in the container layer 601 for files that contain a naming structure indicative of an OMD file 620, for example by scanning for the filename prefix .om. (e.g., .om.filename). Files mapped up to the container layer 601 with a filename name that does not indicate that the file itself is an OMD file 620 can be committed directly to the new image 520 by the daemon 509 using the commit process 605. Files identified by the storage driver as OMD files 620 can be parsed for the CC flag. If the CC flag is not present in the OMD file 620, the file contents 723 from the lower layers 709 are made available to the user in the user view 711 by the container file system, and the file metadata 621 of the OMD file 620 is also made available for viewing by the user in the user view 711.

[0080] FIG. 7A illustrates an example of the override&merge function in a scenario where the storage driver 512 prepares the container layer 601 for the commit process 605 to a new image 520 and the file system generates a user view 711 from the container layer 601 and the lower layer 709 of the image file. The new image 520 resulting from the commit process 605 may include a combination of a portion of the OMD file 620 and the original file located in the lower layer 709, where the file contents 723 are unchanged between the OMD file 620 and the original file. In this particular example, only the file inode metadata 721 has changed. As shown, only the mode of the original file located in the lower layer 709 has changed from 444 to 644 in the file metadata 621 of the OMD file 620. The delta file contents 623 of the OMD file 620 show that the value of the CC flag is “false” and the delta pointer is NULL, indicating that there is no change in the file contents 723 between the original file and the OMD file 620. When the storage driver 512 performs the override & merge function, a user browsing the container file system through the generated user view 711 will see the file metadata 621 of the OMD file 620 and the file contents 723 of the original file.

[0081] In contrast to FIG. 7A, FIG. 7B depicts storage driver 512 performing an override&merge function where there are changes to both the OMD file 620 and the file contents 723 of the original file that differ from the file inode metadata 721. First, looking at the differences between the file inode metadata 721 and the file metadata 621 of the OMD file, it is clear that in this example, both the mode and size of the file are different. As shown, the file metadata 621 shows a mode of value 755 and a size of 3423 megabytes (mb), which is quite different from the file inode metadata 721 of the original file which shows a mode of value 555 and a size of 3400mb. Thus, during the execution of the override&merge function, the file inode metadata 721 is overridden by the file metadata 621 of the OMD file 620, resulting in the file metadata 621 being mapped to the user view 711 and both the file contents 723 and the delta file contents 623 being mapped to the user view 711, as shown.

[0082] FIG. 8A provides an example of storage driver 512 applying copy-on-write operations to the container file system of image files to create a single unified view in user view 711 that can be committed to new image 520. As shown in the example of FIG. 8A, the base layer (i.e., layer 0) 801 of the file system includes file3 and infer.log, which were part of the base image. When building the container image, infer.log can be a 1 gigabyte (GB) or larger file that resides in a read-only layer. During the modification of layer 1 addition, file2 was added. The most recent modification to the file system depicted in layer 2 added file1, removed file3, and added a new or different copy of infer.log. As shown, although file1 and file2 are in different layers, both can be mapped from their respective layers to user view 711 and made visible to the user. File3 was removed from the container file system. To make file3 invisible to the user view 711, the storage driver 512 may place a whiteout file 603 in layer 2 to prevent the original file3 in the base layer 801 from becoming visible or being mapped to the user view 711. With respect to the infer.log, the infer.log in layer 2 is a different copy than the infer.log located in the base layer 801. In this particular example, the updated infer.log in layer 2 is shown to be mapped directly into the user view 711, as opposed to creating an OMD file 620 and performing an override&merge operation.For example, the difference between the hash value of the updated infer.log in layer 2 and the hash value of the original infer.log in base layer 801 may exceed the default differ value, and therefore the updated infer.log from layer 2 is directly mapped in its entirety to the user view 711 rather than creating an OMD file 620 and running the override&merge function.

[0083] FIG. 8B illustrates an alternative example to the example of FIG. 8A. Similar to FIG. 8A, in FIG. 8B, File1 and File2 are mapped to the user view 711. File3 is deleted in the file system, so a whiteout file (.wh.File3) 603 is placed in layer 2 to prevent File3 from appearing as not being mapped to the user view 711 and to prevent the user from seeing File3 in the container's file system. The infer.log in the example of FIG. 8B is different because an OMD file 620 has been inserted by the storage driver 512 into the read-write layer (i.e., layer 2 in this example) instead of the updated infer.log shown in FIG. 8A. The presence of the OMD file 620 is acknowledged by the filename ".om.infer.log". Instead of directly mapping the .om.infer.log to the user view 711 as described above, the override&merge function 813 is performed using both the original infer.log located in the base layer 801 and the .om.infer.log located in layer 2. As a result of the override&merge function 813, the "infer.log" made visible in the user view 711 by the container file system includes at least the file metadata 621 of the .om.infer.log along with the file contents 723 of the original infer.log located in the base layer 801, and if the CC flag is present in the .om.infer.log, the delta file contents 623 can be made viewable as part of the infer.log visible to the user in the user view 711.

[0084] A method for constructing and modifying image files with intelligent layer control

[0085] The drawings in Figures 9A-9B depict an embodiment of a method for constructing or modifying an image file for deployment by a container 523 using intelligent layer control to manage redundant content between layers as described above in Figures 3-8A-B using one or more computing systems generally defined by the computing system 100 of Figures 1-2. More specifically, an embodiment of the method described herein is depicted by an embodiment of the container system and environment described in Figures 5-8A-8. One skilled in the art should recognize that the steps of the method described in Figures 9A-9B may be performed in a different order than presented and may not require the execution of all steps described herein.

[0086] An embodiment of a method 900 illustrated by FIGS. 9A-9B may begin at step 901. During step 901, a daemon 509 may receive a request from a client 503 to build a container image using an existing image or to modify an existing container image. In response to the request, at step 903, the daemon 509 may pull a base image or an existing image specified by the client's request from a registry 515 or its image repository 517 to the local storage 511 of the host node 501. If the base image or the image to be modified was previously pulled to the local storage 511 of the host node 501, the daemon may not need to pull the pulled image 519 from the registry. At step 905, the daemon 509 may deploy the pulled image 519 as a container 523 in the container runtime environment 521 to modify or build a new image 520 within the container to be deployed.

[0087] At step 907 of the method 900, prior to creating the new image 520 by committing the changes written to the container layer 601 to the new image 520, the daemon 509 may inspect the files or directories of the file system of the container 523 for changes to the file system. At step 909, output from the inspection of the container file system may be analyzed to determine whether the container file system includes changes to one or more files or directories in the read-only layer of the file system. At step 909, if the container file system does not include changes to files or directories in the read-only layer of the file system, the method may proceed to step 911, whereby the daemon 509 may commit the changes to the pulled image 519 using a conventional commit process and output the new image 520. Conversely, if the container file system includes changes to one or more files or directories in the read-only layer of the pulled image 519, the method may proceed to step 913 and trigger the pre-commit processing module 510.

[0088] During step 913, the PPM 510 filters files or directories identified by the daemon check as having changed between the lower read-only layer of the pulled image 519 and the updated file of the pulled image 519. The PPM 510 may filter files or directories based on file type, file size, or other parameters and attributes, or a combination thereof. In step 915, the PPM 510 may compare a hash value of the original file from the lower layer of the pulled image 519 with a hash value of the updated file from the upper layer of the pulled image 519. In step 917, a determination is made based on the comparison of the hash values ​​by the PPM 510 as to whether the hash values ​​are identical. If the hash values ​​are identical, the method 900 proceeds to step 919, where the PPM 510 updates the file inode metadata 721 to generate an OMD file 620 and includes the file metadata 621 of the updated file in an override block of the OMD file 602. After generation, the PPM 510 can remove the updated files from the container's file system, for example by running the rm command to unlink the files or directories from the updated files.

[0089] During step 917, if the hash value of the original file of the lower layer is not identical to the hash value of the updated file, method 900 may proceed to step 921. During step 921, PPM 510 may further check whether the difference in the hash values ​​exceeds a default differ value or a threshold value set by a user or administrator of the container system. If the difference in the hash values ​​exceeds the default differ value or threshold value, method 900 may proceed to step 911 and commit the changes to pulled image 519 using a conventional commit process to create new image 520. Alternatively, at step 921, if the difference in the hash values ​​does not exceed the default differ value or threshold value, method 900 may proceed to step 923. During step 923, the PPM 510 may update the file inode metadata 721, the content changed flag, and / or the delta file contents 623 to generate an OMD file 620 that includes the file metadata 621 for the updated file and an indication of whether the value of the content changed flag is true or false, and if true, places a delta pointer in a merge block of the OMD file 620 that indicates where the delta contents of the updated file that differ from the original file are stored. Once the OMD file 620 is generated by the PPM 510, the PPM 510 may remove the updated file from the container's file system. In step 925 of the method 900, the OMD file 620 generated by the PPM 510 may be provided to the storage driver 512. In an embodiment of the storage driver 512, the OMD file 620 may be placed in the container layer 601.

[0090] In step 927, the storage driver 512 may scan the container file system for the OMD file 620. In step 929, the storage driver 512 determines whether an OMD file is found in the container file system based on the filename of each file located in the container layer 601. If an OMD file 620 is not found in the container layer 601, the method 900 proceeds to step 911, where the changes in the container layer 601 are committed using a conventional commit process. Conversely, if an OMD file 620 is found in step 929, the method 900 proceeds to step 931, where the storage driver 512 parses the OMD file 620. As a result of parsing the OMD file 620, in step 933, the storage driver 512 determines whether the OMD file 620 includes a CC flag indicating that the content has changed from the original file located in a lower layer. If no CC flag is found during step 933, the method 900 may proceed to step 935, in which the storage driver 512 copies-up the file contents 723 of the original file in the lower layer 709 to the user view 711, and proceeds to step 939, described below. Alternatively, if a CC flag is identified as being present in the OMD file 620, the method proceeds to step 937. During step 937, the storage driver 512 merges the changes in the content of the OMD file 620 (i.e., the delta file contents 623) with the delta file contents 623 of the original file located in the lower layer 709, and maps both the delta file contents 623 and the file contents 723 to the user view 711. Such mapping allows the user to view the contents of the container file system including the merged contents of both the original file contents and the delta contents that appear in the updated file.

[0091] In step 939, the storage driver 512 overrides the file inode metadata 721 from the original file in the lower layer 709 and maps the file metadata 621 of the OMD file 620 to the user view 711. In step 941, the read-write layer (container layer 601) is finalized based on the OMD file 620 and the original file that form the entire content available to the user as seen by the user via the user view 711. In step 943, the daemon 509 performs a commit operation that creates a new image 520 from the changes to the container 523 as represented by the read-write layer that is added as a new layer on top of the existing layers in the new image 520. In step 945, the daemon 509 deploys a new container 524 that runs the new image 520.

Claims

1. 1. A computer-implemented method for building or modifying an image file into a new image for deployment as a container, the computer-implemented method comprising: checking, by a daemon, a file or directory of a container file system of said image file for changes to said file or directory; filtering updated files copied up from the image file to a read-write layer by the daemon in response to checking that the container file system contains changes to the file or directory; comparing, by the daemon, a hash value of an original file of the container file system from a lower read-only layer of the image file with a hash value of the updated file; comparing the hash value of the original file with the hash value of the updated file, and generating, by the daemon, an override merge data (OMD) file including an override block and a merge block in response to the hash value of the original file and the hash value of the updated file being identical or a difference between the hash value of the original file and the hash value of the updated file exceeding a default differ value; updating, by the daemon, metadata of the override block from inode metadata of the original file stored in the lower layer of the image file to metadata of the updated file; removing, by the daemon, the updated file from the container file system; 4. A computer-implemented method comprising:

2. the difference between the hash value of the original file and the hash value of the updated file exceeds a default diff value, and the computer-implemented method further comprises: updating, by the daemon, the merge block of the OMD file with delta content information indicating that content has changed between the original file and the updated file, and a pointer to a storage area containing the delta content. The computer-implemented method of claim 1 , further comprising:

3. placing the OMD file in the read-write layer by a storage driver of the daemon; scanning, by the storage driver, the read-write layer for each file present in the read-write layer; upon detecting the OMD file in the read-write layer, parsing, by the storage driver, the OMD file for a content change flag in the merge block; parsing the OMD file and identifying a value of the content change flag in the OMD file as true, merging, by the storage driver, the content of the original file with the delta content of the OMD file; The computer-implemented method of claim 1 , further comprising:

4. 4. The computer-implemented method of claim 3, further comprising, upon parsing the OMD file and identifying a value of the content change flag in the OMD file as false, overriding, by the storage driver, the inode metadata of the original file with metadata in the override block of the OMD file.

5. 4. The computer-implemented method of claim 3, further comprising overriding, by the storage driver, the inode metadata of the original file with metadata in the override block of the OMD file.

6. 4. The computer-implemented method of claim 3, further comprising: executing, by the daemon, a commit operation that creates the new image including a new read-only layer that includes the file in a read-write layer placed on top of all previous read-only layers from the image file.

7. 2. The computer-implemented method of claim 1, wherein generating the OMD file including the override block and the merge block occurs when there is no content change between the original file and the updated file, when there is only a change in one or more properties of the inode metadata of the original file stored in the lower layer, or when there is a change between the original file and the updated file and a file size change is below a threshold level.

8. A computing program product for building or modifying an image file into a new image for deployment as a container, comprising: One or more computer readable storage media having computer readable program instructions stored thereon, the program instructions comprising: checking, by a daemon, a file or directory of a container file system of said image file for changes to said file or directory; filtering updated files copied up from the image file to a read-write layer by the daemon in response to checking that the container file system contains changes to the file or directory; comparing, by the daemon, a hash value of an original file of the container file system from a lower read-only layer of the image file with a hash value of the updated file; comparing the hash value of the original file with the hash value of the updated file, and generating, by the daemon, an override merge data (OMD) file including an override block and a merge block in response to the hash value of the original file and the hash value of the updated file being identical or a difference between the hash value of the original file and the hash value of the updated file exceeding a default differ value; updating, by the daemon, metadata of the override block from inode metadata of the original file stored in the lower layer of the image file to metadata of the updated file; removing, by the daemon, the updated file from the container file system; the one or more computer-readable storage media performing a computer-implemented method comprising:

4. A computing program product, comprising:

9. the difference between the hash value of the original file and the hash value of the updated file exceeds a default differ value, and the computer-implemented method further comprises: updating, by the daemon, the merge block of the OMD file with delta content information indicating that content has changed between the original file and the updated file, and a pointer to a storage area containing the delta content.

9. The computing program product of claim 8, further comprising:

10. placing the OMD file in the read-write layer by a storage driver of the daemon; scanning, by the storage driver, the read-write layer for each file present in the read-write layer; upon detecting the OMD file in the read-write layer, parsing, by the storage driver, the OMD file for a content change flag in the merge block; parsing the OMD file to identify a value of the content change flag in the OMD file as true, merging the content of the original file with the delta content of the OMD file by the storage driver and placing the content and delta content into a final file in a user-viewable layer of the image file.

9. The computing program product of claim 8, further comprising:

11. 11. The computing program product of claim 10, further comprising, upon parsing the OMD file to identify a value of the content change flag in the OMD file as false, overriding, by the storage driver, the inode metadata of the original file with metadata in the override block of the OMD file.

12. 11. The computing program product of claim 10, further comprising overriding, by the storage driver, the inode metadata of the original file with metadata in the override block of the OMD file.

13. 11. The computing program product of claim 10, further comprising: executing, by the daemon, a commit operation that creates the new image including a new read-only layer that includes the file in a read-write layer placed on top of all previous read-only layers from the image file.

14. 9. The computing program product of claim 8, wherein generating the OMD file including the override block and the merge block occurs when there are no content changes between the original file and the update file, when there are only changes to one or more properties of the inode metadata of the original file stored in the lower layer, or when there are changes to the content between the original file and the update file and the change in file size is below a threshold level.

15. 1. A computer system for building or modifying an image file into a new image for deployment as a container, comprising: A processor; a computer-readable storage medium coupled to the processor, the computer-readable storage medium being capable of, via the processor: checking, by a daemon, a file or directory of a container file system of said image file for changes to said file or directory; filtering updated files copied up from the image file to a read-write layer by the daemon in response to checking that the container file system contains changes to the file or directory; comparing, by the daemon, a hash value of an original file of the container file system from a lower read-only layer of the image file with a hash value of the updated file; comparing the hash value of the original file with the hash value of the updated file, and generating, by the daemon, an override merge data (OMD) file including an override block and a merge block in response to the hash value of the original file and the hash value of the updated file being identical or a difference between the hash value of the original file and the hash value of the updated file exceeding a default differ value; updating, by the daemon, metadata of the override block from inode metadata of the original file stored in the lower layer of the image file to metadata of the updated file; removing, by the daemon, the updated file from the container file system; said computer-readable storage medium including program instructions for performing a computer-implemented method comprising: A computer system comprising:

16. the difference between the hash value of the original file and the hash value of the updated file exceeds a default diff value, and the computer-implemented method further comprises: updating, by the daemon, the merge block of the OMD file with delta content information indicating that content has changed between the original file and the updated file, and a pointer to a storage area containing the delta content.

16. The computer system of claim 15, further comprising:

17. placing the OMD file in the read-write layer by a storage driver of the daemon; scanning, by the storage driver, the read-write layer for each file present in the read-write layer; upon detecting the OMD file in the read-write layer, parsing, by the storage driver, the OMD file for a content change flag in the merge block; parsing the OMD file and identifying a value of the content change flag in the OMD file as true, merging, by the storage driver, the content of the original file with the delta content of the OMD file; 16. The computer system of claim 15, further comprising:

18. 20. The computer system of claim 17, further comprising, upon parsing the OMD file and identifying a value of the content change flag in the OMD file as false, overriding, by the storage driver, the inode metadata of the original file with metadata in the override block of the OMD file.

19. 20. The computer system of claim 17, further comprising overriding, by the storage driver, the inode metadata of the original file with metadata in the override block of the OMD file.

20. 16. The computer system of claim 15, wherein generating the OMD file including the override block and the merge block occurs when there is no content change between the original file and the updated file, when there is only a change in one or more properties of the inode metadata of the original file stored in the lower layer, or when there is a change between the original file and the updated file and the change in file size is below a threshold level.

Images