Measurement of containers

Abstract

A procedure that includes: in a computer system, acquiring a first integrity measurement corresponding to a software container, wherein acquiring the first integrity measurement comprises a hardware processor of the computer system measuring a given layer of a plurality of layers of a layered file system structure corresponding to the software container, the given layer comprising a plurality of files, and the first integrity measurement comprises an integrity measurement of the plurality of files; Storing the first integrity measurement in a secure memory of the computer system, Measuring metadata associated with the container to perform a second integrity measurement, wherein the metadata is associated with an instantiation of the container; Storing the second integrity measurement in the secure memory such that a content of the secure memory includes the first integrity measurement and the second integrity measurement; wherein the measuring of the metadata is performed during a runtime of the container corresponding to the instantiation of the container; where the measurement of the given layer is performed before the instantiation of the container; and where the contents of the secure storage are used to verify the integrity of the software container.

Description

BACKGROUNDA computing platform (e.g., a server) may be subject to a security attack in which an external device attempts to access information stored on the computing platform or damage components of the computing platform. To prevent or at least contain such security attacks, the computing platform may have various access restriction mechanisms, such as firewalls, keywords, keys, etc. Moreover, the measurements of the computing platform may be analyzed to detect and mitigate security attacks. To this end, the computing platform may measure its software components regularly and log the measurements. The protocols may be sent to a remote verifier or other site for analysis, along with a testable proof of authenticity of the protocols.US 2019 / 0 156 023 A1 describes a computer implemented method for providing security to a software container. US 2018 / 0 349 610 A1 describes systems, devices and methods for constructing a hardware-based trust chain in a computer system. It is an object of the invention to propose a method, a non-transitory machine-readable storage medium and a system for checking the integrity of a software container. This object is achieved by a method according to the invention as claimed in claim 1, a non-transitory machine-readable storage medium according to the invention as claimed in claim 9 and a system according to the invention as claimed in claim 12.BRIEF DESCRIPTION OF THE DRAWINGSFIG. 1 is a schematic diagram of a computer network including an agent for measuring container images and container instantiations, according to an example implementation. FIG. 2 shows an illustration of the agent measuring one or more layers of a container image according to an example implementation. FIG. 3 illustrates how the agent measures metadata representing information about the container image, according to an example implementation. FIG. 4 illustrates how the agent measures one or more layers of the overlay file system of a container, according to an example implementation. FIG. 5 illustrates how the agent measures metadata representing information about an instantiated container, according to an example implementation. FIG. 6 shows a timeline of container-related measurements and their corresponding triggers over an example life cycle of a container, according to an example implementation. FIG. 7 is a flow diagram illustrating a process for detecting and storing a measurement corresponding to a software container, according to an example implementation. FIG. 8 is an illustration of a non-transitory machine-readable storage medium storing instructions to cause a machine to measure layers of a container image, measure an overlay file system associated with a container, and store the measurements in secure storage, according to an example implementation. FIG. 9 is a schematic diagram of a system including a processor to measure layers of an upper directory and a lower directory of an overlay file system corresponding to a container and store the measurements in secure storage, according to an example implementation.DETAILED DESCRIPTIONA computing platform may perform from time to time integrity measurements (also referred to herein as "measurements") of its software components (e.g., drivers, an operating system, a boot loader, virtual machines, containers, etc.). In this context, an "integrity measurement" (or "measurement") of a software component refers to data representing evidence that can be analyzed to assess the integrity of the software component, i.e., determine whether or not the software component has been compromised based on the data. In general, logged integrity measurements may be analyzed and compared to base references (e.g., signatures, hashes, values, etc.) to evaluate whether the computing platform is deemed trusted, determine whether the computing platform has been exposed to a security attack, determine security deficiencies, identify software components that have been compromised, etc.Integrity measurement of a software component may take many different forms. An integrity measurement may be, for example, a plain text metadata value representing a feature of a software component; a tuple of metadata values representing various features of a software component; a hash value representing a fingerprint of a file accessed by the software component; a hash value representing a fingerprint of an executable file of the software component and inputs to the executable file; a hash value representing a fingerprint of a particular sequence of files accessed by the software component; and so forth.A trusted computing platform computing base (TCB) may make and log the platform integrity measurements. As used herein, a "trusted computing base" refers to a number of hardware, firmware, and / or software components of the computing platform that form the core of the security of the platform. In other words, the TCB may be inherently trusted software, hardware, or a combination thereof.Remote verification is a process that allows a computing platform to prove to a remote test center (e.g., a server) that the computing platform is trusted. More specifically, a remote test center may request the computing platform with an approval request, and in response to the approval request, the computing platform may send integrity measurement protocols to the remote test center, along with evidence (e.g., a signed summary of measurement hashes) that the integrity measurement protocols are authentic. After the authenticity of the integrity measurement protocols has been verified, the remote verification site or other site may analyze the integrity measurement protocols to assess potential security concerns for the computing platform and, if the computing platform is affected, recommend and / or provide appropriate remedial action to the computing platform.The computing platform TCB may make computing platform integrity measurements in response to a number of different events, such as booting or booting the computing platform, a request for verification, launching a software component, accessing a file, and so forth. According to example implementations, the computing platform may store one or more protocols of integrity measurements in a non-volatile memory of the computing platform; and for purposes of proving authenticity of the protocols, the computing platform may store hashes derived from the measurement protocols in a secure trusted memory of the computing platform, e.g., in the memory of the platform configuration register (PCR) in a trusted platform module (TPM). In response to an acknowledgment request from a remote test center, the computing platform may provide the integrity measurement protocols and provide an authenticated summary of the hashes stored in PCR memory to prove that the integrity measurement protocols are authentic.According to example implementations described herein, the computing platform may perform integrity measurements on a software container of the computing platform in response to certain triggering events. In this context, a "container" (also referred to herein as an "instantiated container," "container instance," or "software container") generally refers to a virtual runtime environment for one or more applications and / or application modules, and this virtual runtime environment is configured to interface with an operating system kernel. For example, a container for a particular application may contain executable code for the application and its dependencies, such as system tools, libraries, configuration files, executable files, and binary files for the application. In accordance with example implementations, the container includes an interface for incorporating the kernel but not the kernel. For example, a particular computing platform may include multiple containers sharing an operating system kernel via corresponding operating system kernel mount interfaces. Docker containers and rkt containers are examples of software containers.The container is a runtime unit that has a corresponding overlay file system. As described below, the overlay file system has a layered file system structure that includes lower layers corresponding to a container image and an upper layer that serves as a workspace for the file system. An instantiated container is created from the container image at the load time, and multiple instantiated containers may be created from the same container image.The container image forms invariable layers of the container's overlay file system. The "invariable" nature of the container image here means that the container image and the corresponding lower layers of the overlay file system formed from the container image should be write-protected (i.e. should not be altered). The container image includes directories, files, executables, libraries, etc. A base layer of the container image includes a root file system (i.e., the root directory and files) and a basic configuration for the root file system (e.g., the entry point of the root file system, environmental variables, etc.). The container image may also contain one or more layers (so-called "intermediate layers") above the base layer. Together, the layers of the container image form a multi-layer file system (referred to herein as a "container image file system").According to example implementations, the container overlay file system is a union file system that allows files and directories of different file systems to be overlaid into a single coherent file system. More specifically, the container's overlay file system contains a lower directory formed from the container image file system. The lower directory is read-only, which is consistent with the unalterability of the container image. The container's overlay file system also includes an upper directory that contains a container layer in which files can be read and written.Due to the Union file system, files in the layers having the same paths are typically merged, with the top layer file versions controlling the views of the files. For example, in a copy-on-write operation, a file may be read from the lower directory, changed, and then written to the upper directory in the changed form; the changed file is displayed by the container hang, even though the file in the lower directory has not been changed. As another example, a file that does not exist in the lower directory may be created and stored in the upper directory; and as another example, a file of the upper directory may be deleted, although the file may be present in the lower directory.Instantiated containers may share container image layers, which may be advantageous from the standpoint of re-using frequently used image layers as well as saving storage resources. For example, a container image base layer may correspond to a particular database management application, e.g., a particular Structured Query Language (SQL) database management application. For example, a user may create a container that has the basic functions of the SQL database application, but is further adapted with certain statistical tools. For example, the user may use a container build engine or a container engine to create a container build file in which the base level of the SQL database management application is specified as the parent level of the container image and in which another level corresponding to a particular statistics package is specified for the container image. For example, the container containing the SQL database application with the statistics package may be bound to another container containing the same base layer for read-only access, so that reuse of the base layer does not consume additional memory resources.Container measurements can be made at both the load time and the run time. One way to measure the integrity of a container is to measure the entire container image at the load time by calculating a hash value representing the entire container image. Due to the invariable nature of the container image, the calculated hash value should not deviate from an expected hash value for the container image.In this context, a "hash" (also referred to herein as a "hash value") is generated by applying a cryptographic hash function to a value (e.g., an input such as an image). A "cryptographic hash function" may be a function provided by the execution of machine readable instructions by a processor (e.g., one or more central processing units (CPUs), one or more CPU processing cores, etc.). The cryptographic hash function may receive an input and the cryptographic hash function may then generate a hexadecimal string corresponding to the input. The input may include, for example, a string of data (e.g., the data structure in memory that is characterized by a beginning memory address and an ending memory address). In such an example, the cryptographic hash function outputs a hexadecimal string based on the string of data. In addition, any minute change in input may change the output hexadecimal string. In another example, the cryptographic hash function may be a secure hash function (SHA), a Federal Information Processing Standards (FIPS) approved hash function, a National Institute of Standards and Technology (NIST) approved hash function, or another cryptographic hash function. In some examples, instead of a hexadecimal format, another format may be used for the string.Measuring a container by computing a hash over the entire container image may be inefficient from both the standpoint of processing resources and storage resources. Thus, multiple containers may share a base layer and possibly intermediate layers, such that the calculation of hashes over the entirety of the container images may entail unnecessary hash processings.According to the example implementations described herein, an agent of a computing platform performs layer-oriented integrity measurements of a container. In this context, the agent that makes one or more "layer oriented container integrity measurements" refers to determining one or more integrity measurements for a particular layer of a container image (at load time) or a particular layer of the overlay file system of a container (at run time).As a more specific example of the layer-oriented container integrity measurements, according to some implementations, the agent may calculate a hash for each individual file of a layer, and the resulting hashes correspond to a set of measurements for the layer. As another example, in accordance with some implementations, the agent may calculate hashes for groups of files (e.g., hashes for different subdirectories), and these hashes correspond to a series of measurements for the group of files (e.g., particular directories). Layered container integrity measurements for individual files or file groups may be particularly advantageous for detecting file manipulations. According to further implementations, if the goal is not to identify file manipulations, the agent may compute a single hash value that serves as a measurement for the entire layer.According to example implementations, the agent may measure multiple layers of the container image. For example, the agent may calculate one or more hashes (depending on whether file manipulations are to be detected) for the base layer and for each intermediate layer of the container image. Since the layers of the container image are invariable, the measurements should correspond to the expected values. Similarly, the agent may also measure one or more layers of the instantiated container overlay file system. The intermediate and base layers of the instantiated container's file system should match the expected values, as these layers are invariable. The agent may measure the upper container layer of the overlay file system, in accordance with example implementations. Although the container layer is dynamic in nature, the agent may generate individual file and / or group measurements for the container layer when measuring the container layer, which may analyze or ignore certain files and / or directories when analyzing the measurements of the container layer.The agent may make integrity measurements other than layer-oriented integrity measurements, in accordance with example implementations. For example, in some implementations, the agent may read metadata from container images to make corresponding measurements, read metadata of the instantiated container file system to make corresponding measurements, measure files accessed during container runtime, and so forth.As further described herein, in accordance with example implementations, the agent may perform the integrity measurements in response to certain container-related events, such as container launch requests, file access requests, container capture requests, approval requests, and so forth. As also further described herein, in accordance with example implementations, the agent logs or records the integrity measurements to create one or more worklog integrity protocols; and the agent stores hashes representing the logged integrity measurements in a secure memory of a security module of the computing platform such that a remote verifier can use an authenticated summary of the hashes (provided by the security module) to verify that the workload integrity protocol(s) are trusted.Referring to FIG. 1, as a more specific example, a computer network 100 may include one or more computer platforms 110, in accordance with some implementations; and as shown in FIG. 1, a particular computer platform 110 may execute one or more container instantiations or "containers 120.". According to example implementations, the computing platform 110 may be any processor-based device, such as a server, a client, a thin client, a desktop, a portable computer, a laptop, a tablet computer, a notebook, a smartphone, a portable computer, and so forth.Container 120 may be connected to a "node" that refers to a unit of processing resources sufficient to execute container 120. The node may be an actual or physical node, e.g., a node formed from the computing platform 110 or a partition thereof; or, in other example implementations, a particular node may be a virtual node, e.g., a virtual machine, located on the physical computing platform 1 10.The computing platform 110 may have any number of different architectures. In the example architecture illustrated in FIG. 1, computing platform 110 includes one or more processors 124 (e.g., one or more central processing units (CPUs), one or more CPU processing cores, etc.). Moreover, the computing platform 110 may have other associated hardware, such as a memory 128, a network interface 146, a security module such as a trusted platform module (TPM) 150, mass storage devices 142, input / output devices, and so forth.The memory 128 is generally non-transitory storage media of the computing platform 110. The memory 128 may be comprised of semiconductor memory devices, memristor-based memory devices, magnetic memory devices, phase-change memory devices, a combination of devices of one or more of these memory technologies, etc. Moreover, the memory 128 may represent a collection of memories from volatile and nonvolatile memory devices.In accordance with example implementations, the memory 128 stores data representing one or more workload integrity protocols 137. According to some implementations, a workload integrity protocol 137 is associated with a particular container 120 and includes entries; and each entry of the workload integrity protocol 137 corresponds to a different integrity measurement for the container 120 and, in addition to storing data representing the integrity measurement, may include data representing one or more of: an identifier for an event associated with the integrity measurement; a timestamp; an identifier for a type or category of the integrity measurement; and so forth. In accordance with further example implementations, a workload integrity protocol 137 may include integrity measurements for multiple containers 120. Moreover, a particular workload integrity protocol 137 entry may store multiple measurements of a container 120, in accordance with further example implementations. Regardless of the organization of the logged integrity measurements, generally, load time and run time integrity measurements for the containers may be logged and stored in memory 128.As shown in FIG. 1, memory 128 may also store other data 136 and machine executable instructions 132 in addition to workload integrity protocol(s) 137. One or more processors 124 may execute machine-executable instructions 132 to form one or more software components discussed herein, such as a container engine 160, a container 120, an operating system 125, and a container measurement agent 164 (hereinafter, "agent 164").In the example implementation illustrated in FIG. 1, the computer network 100 includes one or more remote verifiers 195 that can communicate with the computer platforms 110 via the network fabric 180 to issue verification requests to the computer platforms 110. As part of the verification, the computing platform 110, when requested by a remote test center 195, may present the workload integrity protocol 137 along with an authenticated summary of hashes 153 as evidence of authenticity of the integrity measurements in the workload integrity protocol 137, in accordance with example implementations.In general, network fabric 180 may include components and utilize protocols associated with one or more types of communication networks, such as (as examples) fibre channel networks, iSC networks, ATA over Ethernet (AoE) networks, HyperSC networks, Gen-Z fabrics, dedicated management networks, local area networks (LANs), wide area networks (WANs), global networks (e.g., the Internet), wireless networks, or any combination thereof.In accordance with example implementations, agent 164 is designed to make container load and run time integrity measurements, log the integrity measurements in one or more workload integrity logs 137, and store hashes 153 representing the integrity measurements in a secure memory of computing platform 110, e.g., a platform configuration register (PCR) memory 151. As mentioned above, a signed summary of the hashes 153 may be used by a remote test center 195 to prove that the logged integrity measurements are authentic (i.e., to prove that the logged integrity measurements have not been manipulated or otherwise compromised).In the example implementation of FIG. 1, PCR memory 151 is trusted secure memory that is part of TPM 150. It should be noted that the TPM 150 is an example of a trusted security module. The computing platform 110 may use any number of different trusted security modules, according to the many possible implementations. Depending on the particular implementation, the security module may be a hardware-based security module, such as a TPM; or, in accordance with other implementations, the security module may be a virtual security module, such as a virtual TPM (vTP), implemented via software. Regardless of whether the security module is implemented in hardware, firmware or software, the security module has a secure memory for storing hashes representing measurement protocols.In accordance with examples where the security module is implemented in hardware, the module may be an integrated circuit (or "chip") located on the motherboard of the computing platform 110 and may be separate from the processors 124 of the computing platform 110. The security module is typically tamper proof and has been developed according to industry standards to provide security features while resisting tampering and malicious software. Moreover, according to example implementations, the security module may include a cryptographic processor (e.g., a dedicated microprocessor or microprocessor core to perform cryptographic operations) developed according to industry standards to provide security functions while resisting manipulations and malicious software.As shown in FIG. 1, in some implementations, the security module may be a TPM commercially available from suppliers such as Infineon Technologies, Nuvoton, and STMicroelectronics. In other examples, a security module may be a firmware-based security coprocessor, such as a TPM implemented in ARM trustzone available from ARM Ltd. of Cambridge, UK, or Intel SGX available from Intel of Santa Clara, CA.The PCR memory 151 represents the storage area of one or more platform configuration registers (PCRs) of the TPM 150. For example, a particular PCR may correspond to a workload integrity log 137 and store a hash that corresponds to 153desired a current state of the workload integrity log 137 and also corresponds to the order in which the logged integrity measurements were added to the workload integrity log 137. In accordance with example implementations, agent 164 logs one or more integrity measurements in workload integrity log 137 and then extends PCR memory 151. In this context, the term "extension of the PCR memory" generally refers to the change of the content of the PCR memory 151 such that the changed content represents both the last measurement(s) and a master book of the measurements leading to the last measurement(s).In accordance with example implementations, agent 164 may expand PCR memory 151 for a new integrity measurement by using a "PCR expansion command.". Execution of the PCR extension command causes a hash value to be calculated based on the current content of the PCR and the new integrity measurement(s), and this calculated hash value is then stored in the PCR memory 151 instead of the current content. As such, the hash value may be used by a remote check center 195 to authenticate the workload integrity protocol 137 because the hash value is unique to the current version of the workload integrity protocol 137 and the order in which the workload integrity protocol 137 was changed to arrive at the current version. According to example implementations, in response to a request for verification from a remote test site 195, the TPM 150 may sign the PCR memory content to form an authenticated test value that is sent to the test site 195 along with the data representing the current version of the workload integrity protocol 137.The container 120 of FIG. 1 illustrates an instantiated container that corresponds to a container image that can be specified by a build file. The build file may designate one or more layers of the image that are shared by other containers. In accordance with some implementations, at load time, container engine 160 may launch or instantiate a particular container 120 based on its corresponding build file in response to a message from container orchestrator 190. In this manner, container orchestrator 190 may launch one or more containers 120 on one or more nodes to form a corresponding group or cluster of nodes for the purpose of performing an activity established by the user via container orchestrator 190.As described herein, in accordance with example implementations, agent 164 makes load time and runtime integrity measurements 174 (also referred to herein as "measurements 174") for a given container 120; stores integrity measurements 174 in a workload integrity protocol 137; and in accordance with example implementations, agent 164 communicates with TPM 150 (as represented by reference number 176) for each integrity measurement 174 to expand PCR memory 151 accordingly. The integrity measurements at "load time" relate to measurements of the container image before the container 120 is instantiated, and the integrity measurements at "run time" relate to measurements of the overlay file system of the instantiated container.FIG. 1 shows an example of container-related data or information that may be measured by agent 164 to provide corresponding integrity measurements 174, in accordance with example implementations: container image layers 166; container layers 168 of the instantiated container overlay file system; container metadata 170 (i.e., metadata describing features of instantiated container 120); and container image metadata 172 (i.e., metadata describing features of the container image).Agent 164 may take a measurement of load time integrity in response to an event corresponding to the start or instantiation of container 120. For example, in accordance with example implementations, agent 164 may "latch" into commissioning a container 120 to perform one or more load time integrity measurements before container 120 is instantiated and to expand PCR memory 151 based on integrity measurement(s) 174. In this context, "engaging" in startup refers to agent 164 engaging in startup of container 120 to stop startup, which allows agent 164 to perform a series of measurement-related actions before agent 164 cancels the stop and allows startup of container 120. The incorporation into the commissioning of the container 120 can take place in various ways, according to the many possible implementations. For example, agent 164 may be a registered plug-in in container engine 160 or container orchestrator 190.In general, each instantiated container 120 is constructed from a corresponding fixed container image that includes container image layers 166. In this way, the container image includes a base image layer and possibly one or more upper intermediate layers. Moreover, each instantiated container 120 has an overlay file system that includes multiple container layers 168: lower invariable layers that correspond to the container image and form a lower directory of the overlay file system, and an upper container layer that forms an upper directory of the overlay file system and corresponds to the read / write work area for the container 120.To benefit from the sharing efficiency of container image planes, in accordance with example implementations, agent 164 may measure one container image plane 166 in response to instantiation of a particular container 120 and then reuse the measurement for another container image (corresponding to another instantiation of container 120) that shares plane 166. In accordance with example implementations, agent 164 may determine multiple measurements for a particular container image layer 166. For example, agent 164 may determine hashes for respective individual files or respective groups of files of container image layer 166.In accordance with further example implementations, agent 164 may determine a single measurement (e.g., a single hash) for the entire container image plane 166. For example, in accordance with some implementations, agent 164 may determine hashes for the individual files and then determine a single hash (i.e., measurement) for container image plane 166 via the concatenation of the file hashes. An advantage of a single hash value for the level measurement is that memory space in the workload integrity protocol 137 is saved. An advantage of having multiple hash values as measurements of the container image layer 166 is that a finer granularity is provided that allows later analysis of the measurement to identify a particular file or file group that has been compromised. Depending on the particular implementation, agent 164 may determine measurements for a single container image plane 166 (e.g., the base plane) or measurements for each of multiple image planes 166 (e.g., measurements for each of the base plane and each of particular intermediate planes of the container image or measurements for each single plane 166 of the container image).Agent 164 may also make measurements of charging time other than layer-oriented measurements. For example, the load time measurements may include a measurement of the container image metadata 172. Generally, the container image metadata 172 represents features or attributes of the container image. Agent 164 may read container image metadata 172 from one or more sources, e.g., container engine 160, container orchestrator 190, operating system 125, or another device. In accordance with some implementations, a measurement of the container image metadata 172 may be that the agent 164 reads the metadata 172 from one or more sources and then creates a tuple of entries (i.e., one entry per metadata category) that forms the corresponding measurement. For example, a remote test site 195 may match metadata values to reference values or apply policies to make relatively complex decisions about the metadata values included in the measurement.The runtime measurements may include measuring one or more layers 168 of the instantiated container overlay file system. The container's top directory (i.e., the container layer) is dynamic because it is the application's working directory that may undergo many reads and writes during the container's 120 lifecycle. In accordance with some implementations, the agent 164 may determine hash values for individual files of the container layer due to the dynamic nature of the container layer, and these hash values may form corresponding measurements for the container layer. The measurement of individual files allows remote review site 195 to flexibly apply policies such as ignoring files or directories (e.g., log or cache directories), review particular files and / or directories (e.g., particular executable files), etc.In accordance with some implementations, the runtime measurements may include measurements of one or more layers of the instantiated container overlay file system lower directory (i.e., the invariable container image-derived layers below the container layer). Depending on the particular implementation, agent 164 may determine measurements for individual files or a measurement for the entire layer for a particular layer of the lower directory. Moreover, depending on the implementation, agent 164 may determine baseline lower level measurements, baseline selected lower level measurements, or baseline values for each of multiple levels of the lower level.Agent 164 may also make measurements other than layer-oriented measurements of instantiated container 120. For example, in accordance with example implementations, the runtime integrity measurements may include integrity measurements of metadata representing features or attributes of the instantiated container 120. Such metadata measurements may be later examined by a remote test center 195 (using value comparisons and policies), for example, to determine whether the container 120 has the expected number and order of layers, whether the container 120 behaves as expected, and so forth.FIG. 2 is an illustration 200 showing how agent 164 makes one or more charge time measurements 220 of a container image 202, in accordance with some implementations. In this example, the container image 202 includes a base layer 166 and one or more top layers 166. The layers 166 are invariable and as such, hashes of the layers 166 as well as hashes generated by combining hashes derived from individual layers 166 should correspond to reference hashes. As mentioned above, in accordance with some implementations, the measurement 220 may be a hash value for a particular layer 166, a hash value for a particular group of layers 166, a hash value for all layers 166, etc. Moreover, a hash value for a particular level 166 may be derived based on the binary value of the entire level 166, based on hash values for individual files of level 166, based on hash values for groups of files of level, etc.FIG. 3 is an illustration 300 of the agent 164 making a measurement 312 of the container image metadata 172, according to an example implementation. The measurement 312 may be a tuple of entries corresponding to one or more entries of the container image metadata 172. FIG. 3 shows three examples of container image metadata 172: image animation metadata 312, trust metadata 308, and entry point metadata 304. The measurement 312 may be, for example, a tuple from all three entries 304, 308, and 312; a tuple from other or additional entries of the container image metadata 172 than the entries 304, 308, 312; and so forth. For example, in the example container image metadata 172 shown in FIG. 3, the entry point metadata 304 may represent a default command that is executed when a container instantiation 120 begins or starts. For example, trust metadata 308 may represent a signature and / or certificate associated with the container image using a particular mechanism such as Docker Content Trust or Docker Notary. In general, the trusted metadata 304 enables verification of both the integrity and the publisher of all data received from a registration over any channel.The image-animation metadata 312 may include, for example, information about the container image itself and the registry from which the container image was downloaded. In general, a remote verifier 195 may examine the metadata of the image animation 312 to verify whether the container image has been downloaded from a particular registry. The measurement 312 may include metadata from other and / or other metadata categories, in accordance with other implementations.FIG. 4 is an illustration 400 of the agent 164 performing one or more runtime measurements 428 of the overlay file system 404 of an instantiated container. Overlay file system 404 in this example includes lower layers 168 corresponding to container image layers 166 and forming a lower directory 470 and an upper layer 168 corresponding to an upper container layer 412 forming an upper directory 480. The container layer 412 is dynamic in that the files of the layer 412 can be written, deleted, created, etc. In Figure 400, four files 430, 434, 436 and 438 are visible from a summary directory 408 corresponding to a hang-in point of the operating system. Although files 430 and 436 are not part of container layer 412, they are read-only files from lower directory 470. Although the file 434 in the lower directory 470 cannot be changed, the file 434 is part of the container layer 412 in this example, and as such, the version of the file 434 in the container layer 412 may be different from the version in the lower directory 470. As also shown in Figure 4, file 438 exists in upper directory 480 and does not have a counterpart in lower directory 470.In accordance with some implementations, the agent 164 may make one or more runtime measurements 428 of the container's overlay file system 404. For example, agent 164 may determine a hash or set of hashes for a particular layer 168 of lower directory 470; and the hash(s) may / may correspond to a measurement / measurements of layer 166. Agent 164 may make similar measurements from multiple or all layers 168 of lower table 470 according to example implementations. Agent 164 may also measure container layer 412 of upper directory 480, in accordance with example implementations. For example, agent 164 may calculate hashes for individual files and / or directories of container layer 412, and these hashes may correspond to measurements 428. Thus, the agent 164 may (as examples) make one or more of the following measurements 428 of the overlay file system 404: measurement(s) 428 of a particular layer 168 of the lower directory 470; measurement(s) 428 of multiple layers 168 of the lower directory 470; measurement(s) 428 of all layers 168 of the lower directory 470; measurement(s) 428 of container layer 412 files of the upper directory 480; and / or measurement(s) 428 of directories or groups of container layer 412 files of the upper directory 480.FIG. 5 is an illustration 500 of the agent 164 making a measurement 550 of the metadata 170 of the instantiated container 120, according to an example implementation. The measurement 550 may be, for example, a tuple of metadata entries corresponding to image identification (ID) metadata 504, level list metadata 508, open port metadata 512, associated volume metadata 516, environmental variable metadata 520, command metadata 524, argument metadata 528, security options metadata 532, network settings metadata 536, namespace metadata 540, and privilege metadata 544.The image ID metadata 504 represents an identifier for the image used as a basis for the container 120. The identifier may be used to associate the image with a known measured image. The metadata of the level list 508 represents a list of the levels that make up the container's overlay file system lower directory. The open port metadata 512 represents the number of open ports for the container 120 and may be used to verify that the container 120 does not open more ports than are expected for the specific workload. The metadata to the associated volumes 516 represents the associated volumes for the container 120 and may be used to verify the container's access to the host file system.The environmental variable metadata 520 represents the environmental variables used by the container 120 and may be used to verify the container's access to host information. The command metadata 524 represents the commands used by the container 120 and may be used to verify whether the container 120 is executing with the expected application.Argument metadata 258 represents arguments used by container 120, and may be used to check whether container 120 is executing with the expected arguments. The security option metadata 532 represents security options (e.g., security module settings such as ApparmorProfil settings or security feature settings of an operating system kernel such as SECCmp settings) used by the container 120, and may be used, for example, to verify whether or not the container 120 bypasses native security mechanisms.The network settings metadata 536 represents network settings such as a dynamic name server (DNS), network type, hosts, etc., and may be used to check whether or not the container 120 is communicating with the expected entities. The namespace metadata 540 may be used to check whether or not a container 120 is running in an isolated namespace. Based on the metadata about the privilege 544, it can be checked whether or not the container 120 extends privilege.In accordance with further example implementations, agent 164 may make metadata-related measurements of the instantiated container from other and / or other metadata categories.FIG. 6 shows an example timeline 600 of triggers or events 602, which may be connected to a container 120 over its life cycle, and the corresponding actions 603, which may be performed by the agent 164 in response to the events 602. As described further below, actions 603 include performing and logging measurements by agent 164 as well as extending PCR memory 151 by agent 164.As shown in FIG. 6, the first event occurs at time 604, at which the container engine 160 requests the start of the container 120. This event causes agent 164 to measure the image of the container from time 606 to time 612, as shown by reference numeral 608, according to embodiments. During this period, agent 164 ceases to place container 120 in service. As described herein, as part of container image measurement, agent 164 may measure one or more layers of the container image and measure container image metadata. At time 612, the measurement of the container image is complete and agent 164 may then log the start-up event measurement(s) and expand the PCR memory as shown in reference number 614. After the PCR memory is expanded, agent 164 may de-lock so that container 120 may operate (see time 620).Continuing with example timeline 600, further events may trigger measurements by agent 164. For example, at time 624, a file access event occurs (e.g., a file is read from or written to the container file system) causing agent 164 to measure the file from time 626 to time 629, as shown at 628; and then agent 164 logs the measurement of the file access event and expands the PCR memory, as shown at reference number 630.Next, in the example timeline 600, at time 640, an event occurs to perform container detection. Note that the container detection execution event may occur according to a schedule (e.g., a periodic schedule). In other words, in accordance with example implementations, a container detection event occurs at certain predetermined times to cause agent 164 to measure container 120, as represented by reference number 646, and therefore make one or more runtime container measurements. The container detection event may be due to other causes, in accordance with other implementations. For example, a capture container event may arise from the start of another instance of the same container 120. The container measurement 646 may include the agent 164 detecting container file system runtime measurements, which may include container layer file measurements, one or more lower directory layer measurements, one or more lower directory layer measurements, one or more lower directory layer measurements, and so forth as described herein. Agent 164 may then log the container detection event measurement(s) and expand the PCR memory, as shown at reference number 649.FIG. 6 also shows an example of a verification request at time 660. A verification request can be made, for example, as part of a remote verification. In response to the verification request, agent 164 may measure the container image (e.g., measurements of the container image layer and the container image metadata) and the container file system (e.g., measurements of the container layer files and the lower directory level), from time 664 to time 668 (as shown by reference number 666), according to example implementations. Agent 164 may then log the measurements of the verification request and expand the PCR memory, as shown at reference number 669.At time 670, container 120 ends for example timeline 600. According to example implementations, in response to the end of container event, multiple actions may be taken by agent 164, such as recording the event (i.e., recording the completion of the container) in workload integrity protocol 137 and / or in a system protocol; resetting the PCR memory after signing the final PCR values; archiving the protocol; etc. It is noted that actions 603 of agent 164 and triggering events 602 of FIG. 6 are merely examples because a particular container 120 may have other, fewer, or more triggering events. In general, agent 164 may respond to a number of different events, such as operating system and system calls, to measure container 120 runtime activities, such as file access (as described above), mount, network access, and so forth. Generally, these events trigger container runtime measurements by agent 164 based on configured policies. Moreover, as also exemplarily described above, the actions 603 may be triggered by explicit requests to perform runtime measurements, e.g., by a request of a remote verifier, a user, a request to detect container events, or generally any authorized entity. This allows periodic and / or demand-controlled measurements to be performed flexibly in a relatively complex system, where measurement of all aspects of a container may not be practicable.Referring to FIG. 7, in accordance with example implementations, a process 700 in a computer system includes detecting (block 704) a first measurement corresponding to a software container. Detecting the measurement includes a hardware processor of the computer system measuring a given layer of a plurality of layers of a layered file system structure corresponding to the software container. The given layer includes a plurality of files, and the first measurement includes a measurement of the plurality of files. The process 700 includes storing (block 708) the first measurement in a secure memory of the computer system. Contents of the secure memory are used to verify the integrity of the software container.Referring to FIG. 8, a machine readable storage medium 800 stores instructions 804 in accordance with embodiments. The instructions 804, when executed by a machine, cause the machine to, in conjunction with a container load time, measure each layer of a plurality of layers of a container image to provide a plurality of first measurements, and store the plurality of first measurements in a secure memory. The contents of the secure memory are used to verify the integrity of the container. The instructions 804, when executed by the machine, further cause the machine to, in conjunction with a runtime of the container, measure an overlay file system to provide a second measurement; and store the second measurement in the secure storage such that the content includes the plurality of first measurements and the second measurement.As shown in FIG. 9, according to example implementations, a system 900 includes a hardware security module 904, a processor 912, and a memory 916. The hardware security module 904 includes a secure memory 908 for storing content used to verify the integrity of a container. The memory 916 stores instructions 918 that, when executed by the processor 912, cause the processor 912 to measure each layer of a plurality of layers of a lower directory of an overlay file system corresponding to the container to provide a plurality of first measurements; and measure a container layer of an upper directory of the overlay file system to provide a second measurement. The instructions 918, when executed by the processor 912, further cause the processor 912 to store the first measurements and the second measurement in the secure memory 908.According to embodiments, the first measurement corresponds to an image of the container and the indicated layer corresponds to a lower layer of the plurality of layers. A particular advantage is that the measurement of a base layer used for several container images can be reused.According to example implementations, metadata associated with a container is measured to obtain a second measurement, and the second measurement is stored in the secure memory such that the content includes the first measurement and the second measurement. A particular advantage is that metadata can be used to determine whether the structure of the container image has changed or whether the instantiated container behaves as expected.According to example implementations, the metadata may be associated with an image of a container. A particular advantage is that the metadata can be used to confirm a particular layer structure of the container image, to check whether a layer has been omitted, a layer has been added, a layer arrangement has been changed, etc.According to example implementations, the metadata may include data representing at least one of: an entry point command corresponding to a request to start instantiation of the container, a signature corresponding to the image, a certificate corresponding to the image, or a manifest of the image. A particular advantage is that other data can also be used to validate the container image in addition to a measurement of a specific layer of the container.According to example implementations, the metadata may be associated with an instantiation of the container. A particular advantage is that the metadata can be used to check whether the container behaves as expected.According to example implementations, the measurement of the metadata may be performed in response to a request to start another instantiation of the container. A particular advantage is that both dynamic and static aspects of the container can be checked between container instantiations.Although the present disclosure has been described with respect to a limited number of implementations, those skilled in the art having the advantages of this disclosure will recognize numerous modifications and variations thereof. It is intended that the appended claims cover all such modifications and variations.

Claims

A method comprising: in a computer system, detecting a first integrity measurement corresponding to a software container, wherein detecting the first integrity measurement comprises a hardware processor of the computer system that measures a given layer of a plurality of layers of a layered file system structure corresponding to the software container, wherein the given layer comprises a plurality of files, and the first integrity measurement comprises an integrity measurement of the plurality of files; storing the first integrity measurement in a secure memory of the computer system, measuring metadata associated with the container to perform a second integrity measurement, wherein the metadata is associated with an instantiation of the container; storing the second integrity measurement in the secure memory such that a content of the secure memory includes the first integrity measurement and the second integrity measurement; wherein the measuring of the metadata is performed during a runtime of the container corresponding to the instantiation of the container; wherein the measuring of the given layer is performed prior to the instantiation of the container; and wherein the content of the secure memory is used to verify an integrity of the software container.The method of claim 1, wherein: the first integrity measurement corresponds to an image of the container; and the given layer corresponds to a base layer of the plurality of layers.The method of claim 1, further comprising: initiating the sensing of the first integrity measurement of the given layer in response to a request to start instantiation of the software container.The method of claim 1, wherein: the metadata is associated with an image of the container.The method of claim 4, wherein the metadata comprises data representing at least one of: an entry point command corresponding to a request to start instantiation of the container, a signature corresponding to the image, a certificate corresponding to the image, or a manifest of the image.The method of claim 1, further comprising: performing the measuring of the metadata and the measuring of the given layer in response to a verification request.The method of claim 1, wherein the metadata comprises data representing at least one of: an image identifier for a base layer of the plurality of layers, a list of the plurality of layers, an open port number, an identifier of an associated volume, an environmental variable, a command, an argument, a security option, a network setting, a namespace, or a privilege.The method of claim 1, further comprising: performing the measuring of the metadata in response to a request to start further instantiation of the container.A non-transitory machine-readable storage medium storing instructions that, when executed by a machine, cause the machine to: in conjunction with a container load time, measure each layer of a plurality of layers of a container image to obtain a plurality of first integrity measurements; store the plurality of first integrity measurements in a secure memory, wherein a content of the secure memory is used to verify an integrity of the container; in conjunction with a container run time, measure metadata representing an overlay file system to provide a second integrity measurement; store the second measurement in the secure memory such that the content of the secure memory includes the plurality of first integrity measurements and the second integrity measurement; wherein the measuring of the metadata is performed during a runtime of the container corresponding to the instantiation of the container; and wherein the measuring of the layers is performed prior to the instantiation of the container.The storage medium of claim 9, wherein the instructions, when executed by the machine, further cause the machine to measure a plurality of layers of the overlay file system corresponding to the image and measure a container layer of the overlay file system.The storage medium of claim 9, wherein the metadata comprises data representing at least one of: an image identifier for a base layer of the plurality of layers, a list of the plurality of layers, an open port number, an identifier of an associated volume, an environmental variable, a command, an argument, a security option, a network setting, a namespace, or a privilege.A system comprising: a hardware security module having a secure memory for storing content used to verify integrity of a container; a processor; and a memory for storing instructions that, when executed by the processor, cause the processor to: measure each layer of a plurality of layers of a lower directory of an overlay file system corresponding to the container to obtain a plurality of first integrity measurements; measure a container layer of an upper directory of the overlay file system to obtain a second integrity measurement; store the plurality of first integrity measurements and the second integrity measurement in the secure memory; measure first metadata representing information about the container to make a third integrity measurement; second metadata representing information about an instantiation of the container to make a fourth integrity measurement; store the third integrity measurement and the second integrity measurement in secure storage; measure the first metadata in response to a command to start instantiation of the container; measure the second metadata in response to a predetermined event, wherein the predetermined event comprises a periodically triggered event or an event to start another instantiation of the container.The system of claim 12, wherein the instructions, when executed by the processor, further cause the processor to measure a file of an instantiation of the container in response to a file access request to provide a third integrity measurement, and store the third integrity measurement in the secure memory.The method of claim 1, wherein detecting the first integrity measurement comprises applying a hash function to the content of the given layer to provide the first integrity measurement.The storage medium of claim 9, wherein the instructions, when executed by the machine, further cause the machine to apply a hash function to the content of each layer to provide the plurality of first integrity measurements.The system of claim 12, wherein the plurality of first integrity measurements comprise hashes representing evidence that can be analyzed to assess the integrity of the software container.

Images