Verification of containers based on comparative measurements

Abstract

Embodiments include methods for a software integrity tool of a host computing system configured with a runtime environment arranged to execute containers. Such methods include obtaining reference data associated with a first plurality of files, which are from respective file systems of at least one first container. For each file, the reference data includes information needed for a comparative measurement according to a security level assigned to the file. Such methods include obtaining a second container having a file system with a second plurality of files and performing a comparative measurement of at least a portion of the second plurality of files based on the reference data associated with the first plurality. Such methods include, when the comparative measurement indicates success, using at least one previous measurement performed on the respective at least one first container as a measurement of the least a portion of the file system of the second container.

Description

[0001] VERIFICATION OF CONTAINERS BASED ON COMPARATIVE MEASUREMENTS

[0002] TECHNICAL FIELD

[0003] The present disclosure relates generally to the field of communication networks, and more specifically to techniques for virtualization of network functions (NFs) using container-based solutions that execute in a host computing system (e.g., cloud, data center, etc.).

[0004] INTRODUCTION

[0005] Conventionally, telecommunication equipment was provided as integrated software and hardware. More recently, virtualization technologies decouple software and hardware such that network functions (NFs) can be executed on commercial off-the-shelf (COTS) hardware. For example, mobile networks can include virtualized network functions (VNFs) and non-virtualized network elements (NEs) that perform or instantiate a NF using dedicated hardware. In the context of an exemplary fifth-generation (5G) network architecture, various radio access network (RAN) nodes (e.g., centralized units) and various NFs in the 5G core network (5GC) can be implemented as combinations of VNFs and NEs.

[0006] In some cases, NFs can be obtained from a vendor as packaged in “containers,” which are software packages that can run on commercial off-the-shelf (COTS) hardware. A computing infrastructure provider (e.g., hyperscale provider, communication service provider, etc.) typically provides resources to vendors for executing their containers. These resources include computing hardware as well as a software environment that hosts or executes the containers, which is often referred to as a “runtime environment” or more simply as “runtime”.

[0007] For example, Docker is a popular container runtime that runs on various Linux and Windows operating systems (OS). Docker creates simple tooling and a universal packaging approach that bundles all application dependencies inside a container to be run in a Docker Engine, which enables containerized applications to run consistently on any infrastructure.

[0008] SUMMARY

[0009] Conventionally, when a runtime such as Docker instantiates a container holding a software image (e.g., of a NF), there was no way to verify that the runtime actually instantiates what was intended. One previous solution involved verification of cryptographic hashes for each container instance, but this involves time consuming calculations for a relatively large number of files. Thus, there is a need for a less complex approach to detect flaws in or attacks on the file system of a container. Embodiments of the present disclosure address these and other problems, issues, and / or difficulties, thereby facilitating more efficient use of runtimes that host containerized software, such as virtual NFs of a communication network.

[0010] Some embodiments include exemplary methods (e.g., procedures) for a software integrity tool of a host computing system configured with a runtime environment arranged to execute containers that include applications.

[0011] These exemplary methods include obtaining reference data associated with a first plurality of files, which are from respective file systems of at least one first container. For each of the first plurality of files, the reference data includes information needed for a comparative measurement according to a security level assigned to the file. These exemplary methods include obtaining a second container having a file system with a second plurality of files and performing a comparative measurement of at least a portion of the second plurality of files based on the reference data associated with the first plurality of files. These exemplary methods include, when the comparative measurement indicates success, using at least one previous measurement performed on the respective at least one first container as a measurement of the least a portion of the file system of the second container.

[0012] In some embodiments, the first and second pluralities of files include one or more common file groups. In other embodiments, the first plurality of files are from a file system of a single first container, with the first container and the second container being based on a common container image.

[0013] In some embodiments, performing the comparative measurement based on the reference data comprises, for each file of the at least a portion of the second plurality of files, performing the comparative measurement of the file based on the following: a security level assigned to the file, and reference data for a corresponding file of the first plurality. In some of these embodiments, performing the comparative measurement of the file based on the assigned security level includes determining or calculating a representation of the file based on the assigned security level, and comparing the representation of the file to the reference data for the corresponding file of the first plurality.

[0014] In some variants of these embodiments, the comparative measurement of the at least a portion of the second plurality of files indicates success when the comparative measurement of each file indicates a match between the representation of the file and the reference data for the corresponding file of the first plurality. In some variants of these embodiments, the representation of the file based on the assigned security level is one of the following:

[0015] • identifying information for a physical file in the host computing system, when a first security level is assigned; • the file, when a second security level is assigned; or

[0016] • a digest for the file, when a third security level is assigned.

[0017] Similarly, in some variants of these embodiments, the reference data for each file of the first plurality includes one of the following:

[0018] • identifying information for a physical file in the host computing system, when a first security level is assigned;

[0019] • a copy of the file or a reference to a corresponding file, when a second security level is assigned; or

[0020] • a digest for the file, when a third security level is assigned.

[0021] In some further variants, a match between the representation of the file and the reference data for the corresponding file of the first plurality is based on one of the following:

[0022] • when the first security level is assigned, the file and the corresponding file are the same physical file in the host computing system;

[0023] • when the second security level is assigned, a byte-by-byte match between contents of the file and contents of the corresponding file; or

[0024] • when the third security level is assigned, a byte-by-byte match between a digest for the file and the digest for the corresponding file.

[0025] In some embodiments, using the at least one previous measurement includes sending a completed measurement of the second container to an attestation verification system (AVS) for verification of the file system of the second container. The completed measurement includes the at least one previous measurement.

[0026] In some of these embodiments, the first plurality of files are from a file system of a single first container, with the first container and the second container being based on a common container image. In such case, the at least one previous measurement is a single previous measurement, which is performed on all files of the first plurality that are included based on a container digest policy associated with the host computing system. Also, when the comparative measurement indicates success, the previous measurement is used as a measurement of all files, of the second plurality, that are included based on the container digest policy.

[0027] In other of these embodiments, the first and second pluralities of files include one or more common file groups, and the at least one previous measurement is performed on the files of the one or more common file groups in the respective file systems of the at least one first container. In such case, when the comparative measurement indicates success, the at least one previous measurement is used as a measurement of the one or more common file groups in the file system of the second container. In some variants of these embodiments, the second container and the at least one first container are based on different container images. Other embodiments include host computing systems configured to perform the operations corresponding to any of the exemplary methods described herein. Other embodiments also include non-transitory, computer-readable media storing computerexecutable instructions that, when executed by processing circuitry of a host computing system, configure the host computing system to perform operations corresponding to any of the exemplary methods described herein.

[0028] These and other disclosed embodiments may significantly reduce the resources needed for measurement and verification of multiple containers instantiated on a host computing system, thereby increasing the speed of startup when container verification is performed. Embodiments may be utilized together with or independent of existing solutions that are based on cryptographic measurements of files associated with a container. Moreover, embodiments may provide such performance improvements when used with multiple containers that are based on the same image or on different images but with some parts of the respective file systems in common. Moreover, embodiments operating at the host level may facilitate better security than conventional verification performed within the container, since they prevents a container from false self-attestation.

[0029] These and other objects, features, and advantages of the present disclosure will become apparent upon reading the following Detailed Description in view of the Drawings briefly described below.

[0030] BRIEF DESCRIPTION OF THE DRAWINGS

[0031] Figure 1 shows an exemplary Network Function Virtualisation Management and Orchestration (NFV-MANO) architectural framework for a 3 GPP-specified network.

[0032] Figure 2 shows an exemplary high-level architecture for a Docker Engine.

[0033] Figure 3 shows an example computing configuration that uses the Docker Engine shown in Figure 2.

[0034] Figure 4 shows a high-level flow diagram of a verification procedure for a container executed by a host computing system, according to some embodiments of the present disclosure.

[0035] Figure 5 shows an exemplary verification procedure based on a full measurement for a container executed by a host computing system, according to some embodiments of the present disclosure.

[0036] Figure 6 shows the full measurement and comparative measurement operations of Figure 4 in more detail.

[0037] Figure 7 shows a host computing system in which a first container has been instantiated and is running, according to some embodiments of the present disclosure. Figures 8A-B illustrate certain operations of Figure 6 in more detail.

[0038] Figures 9-12 show block diagrams of host computing systems according to various embodiments of the present disclosure.

[0039] Figures 13-14 show two example arrangements of reference data stored by a software integrity tool, according to various embodiments of the present disclosure.

[0040] Figures 15 A- B show an exemplary method (e.g., procedure) for a software integrity tool configured to execute in a host computing system, according to various embodiments of the present disclosure.

[0041] Figure 16 is a block diagram illustrating an exemplary container-based host computing system suitable for implementation of various embodiments described herein.

[0042] DETAILED DESCRIPTION

[0043] Embodiments briefly summarized above will now be described more fully with reference to the accompanying drawings. These descriptions are provided by way of example to explain the subject matter to those skilled in the art and should not be construed as limiting the scope of the subject matter to only the embodiments described herein. More specifically, examples are provided below that illustrate the operation of various embodiments according to the advantages discussed above.

[0044] In general, all terms used herein are to be interpreted according to their ordinary meaning to a person of ordinary skill in the relevant technical field, unless a different meaning is expressly defined and / or implied from the context of use. All references to a / an / the element, apparatus, component, means, step, etc. are to be interpreted openly as referring to at least one instance of the element, apparatus, component, means, step, etc., unless explicitly stated otherwise or clearly implied from the context of use. The operations of any methods and / or procedures disclosed herein do not have to be performed in the exact order disclosed, unless an operation is explicitly described as following or preceding another operation and / or where it is implicit that an operation must follow or precede another operation. Any feature of any embodiment disclosed herein can apply to any other disclosed embodiment, as appropriate. Likewise, any advantage of any embodiment described herein can apply to any other disclosed embodiment, as appropriate.

[0045] Note that the description given herein focuses on a 3 GPP cellular communications system and, as such, 3GPP terminology or terminology similar to 3GPP terminology is oftentimes used. However, the concepts disclosed herein are not limited to a 3GPP system and can be applied to any communication system that may benefit from them. In addition, functions and / or operations described herein as being performed by a telecommunications device or a network node may be distributed over a plurality of telecommunications devices and / or network nodes.

[0046] As briefly discussed above, virtualization technologies decouple software and hardware such that network functions (NFs) can be executed on commercial off-the-shelf (COTS) hardware. ETSI GR NFV 001 (vl .3.1) published by the European Telecommunications Standards Institute (ETSI) describes various high-level objectives and use cases for network function virtualization (NFV). The high-level objectives include the following:

[0047] • Rapid service innovation through software-based deployment and operationalization of network functions and end-to-end services.

[0048] • Improved operational efficiencies resulting from common automation and operating procedures.

[0049] • Reduced power usage achieved by migrating workloads and powering down unused hardware.

[0050] • Standardized and open interfaces between network functions and their management entities so that such decoupled network elements can be provided by different entities.

[0051] • Greater flexibility in assigning VNFs to hardware.

[0052] • Improved capital efficiencies compared with dedicated hardware implementations.

[0053] Similarly, the various NFV use cases described in ETSI GR NFV 001 can be divided roughly into the following groups or categories:

[0054] • Virtualization of telecommunication networks.

[0055] • Virtualization of services in telecommunication networks, e.g., Internet-of-Things (loT) virtualization, enhanced security, network slicing, etc.

[0056] • Improved operation of virtualized networks, e.g., rapid service deployment, continuous integration / continuous deployment (CI / CD), testing and verification, etc.

[0057] For example, mobile or cellular networks can include virtualized NFs (VNFs) and nonvirtualized network elements (NEs) that perform or instantiate a NF using dedicated hardware. In the context of an exemplary 5G network architecture, various RAN nodes (e.g., centralized units) and various NFs in the 5GC can be implemented as combinations of VNFs and NEs.

[0058] In general, a (non-virtual) NE can be considered as one example of a physical network function (PNF). From a high-level perspective, a VNF is equivalent to the same NF realized by an NE. However, the relation between NE and VNF instances depends on the relation between the corresponding NFs. A NE instance is 1 : 1 related to a VNF instance if the VNF contains the entire NF of the NE. Even so, multiple instances of a VNF may run on the same NF virtualization infrastructure (NFVI, e.g., cloud infrastructure, data center, etc.). Both VNFs and NEs need to be managed in a consistent manner. To facilitate this, 3GPP specifies a Network Function Virtualisation Management and Orchestration (NFV-MANO) architectural framework. Figure 1 shows an exemplary mobile network management architecture mapping relationship between NFV-MANO architectural framework and other parts of a 3GPP- specified network. The arrangement shown in Figure 1 is described in detail in 3GPP TS 28.500 (vl7.0.0) section 6.1, with certain portions of this description provided below for context.

[0059] The architecture shown in Figure 1 includes the following entities, some of which are further defined in 3GPP TS 32.101 (vl7.0.0):

[0060] • Network Management (NM), which plays one of the roles of operation support system (OSS) or business support system (BSS) and is the consumer of reference point Os-Ma- nfvo;

[0061] • Device Management (DM) / Element Management (EM), if the EM includes the extended functionality, it can manage both PNFs and VNFs;

[0062] • NFV Orchestrator (NFVO);

[0063] • VNF Manager (VNFM);

[0064] • Virtualized infrastructure manager (VIM);

[0065] • Itf-N, interface between NM and DM / EM;

[0066] • Os-Ma-nfvo, reference point between OSS / BSS and NFVO;

[0067] • Ve-Vnfm-em, reference point between EM and VNFM;

[0068] • Ve-Vnfm-vnf, reference point between VNF and VNFM; and

[0069] • NFVI, the hardware and software components that together provide the infrastructure resources where VNFs are deployed.

[0070] EM / DM is responsible for FCAPS (fault, configuration, accounting, performance, security) management functionality for a VNF on an application level and NE on a domain and element level. This includes:

[0071] • Fault management for VNF and physical NE.

[0072] • Configuration management for VNF and physical NE.

[0073] • Accounting management for VNF and physical NE.

[0074] • Performance measurement and collection for VNF and physical NE.

[0075] • Security management for VNF and physical NE.

[0076] • VNF lifecycle management (LCM), such as requesting LCM for a VNF by VNFM and exchanging information about a VNF and virtualized resources associated with a VNF.

[0077] In some cases, NFs can be obtained from a vendor as packaged in “containers,” which are software packages that can run on COTS hardware. More specifically, a container is a standard unit of software that packages application code and all its dependencies so the application runs quickly and reliably in different computing environments. A computing infrastructure provider (e.g., hyperscale provider, communication service provider, etc.) typically provides resources to vendors for executing their containers. These resources include computing hardware as well as a software environment that hosts or executes the containers, which is often referred to as a “runtime.”

[0078] When using containers in VNF implementations, it is common that multiple instances of the same container are started on the same host to increase capacity and improve resilience to faults. Moreover, the respective file systems of different containers started from different images often share many common files.

[0079] Docker is a popular container runtime that runs on various Linux and Windows operating systems (OS). Docker creates simple tooling and a universal packaging approach that bundles all application dependencies inside a container that is run on the Docker Engine. Specifically, Docker Engine enables containerized applications to run consistently on any infrastructure. A Docker container image is a lightweight, standalone, executable package of software with everything needed to run an application, including code, runtime, system tools, system libraries, and settings. Docker container images become containers at runtime, i.e., when the container images run on the Docker Engine. Multiple Docker containers can run on the same physical or virtual machine and share the OS kernel with other Docker containers, each running as isolated processes in user space. Some, or even all, of the Docker containers can be instances spawned from the same image.

[0080] Each container image is built up from layers, using the overlay2 storage driver for the overlay file system which is part of the Linux OS kernel. Overlay is a copy-on-write implementation, which means that a new resource isn’t automatically created on duplication of an object. Rather, a separate resource is allocated for a duplicated object only when (if ever) the duplicated object is modified. As such, the same physical file can be used in multiple containers based on the same image, so long as no modification has occurred.

[0081] It is common that multiple container images share one or more layers. In some cases, a file belonging to a higher layer can override a file belonging to a lower layer, such that the file belonging to the lower layer is not used in that container. Nevertheless, multiple container images running on the same machine often share several files. Consequently, due to copy-on-write operation, the same physical file can be used in multiple containers that are based on different images but have one or more container layers in common.

[0082] Figure 2 shows an exemplary high-level architecture for a Docker Engine, with various blocks shown in Figure 2 described below. Containerd implements a Kubemetes Container Runtime Interface (CRI) and is widely adopted across public clouds and enterprises. Kubemetes is a common platform used to provide cloud-based web services, such as cloud-native network function components. Kubemetes can coordinate a highly available cluster of connected computers (also referred to as “processing elements” or “hosts”) to work as a single unit. Kubemetes deploys applications packaged in containers (e.g., via its runtime) to decouple them from individual computing hosts.

[0083] In general, a Kubemetes cluster consists of two types of resources: a “master” that coordinates or manages the cluster and “nodes” or “workers” that run applications. Put differently, a node is a virtual machine (VM) or physical computer that serves as a worker machine. The master coordinates all activities in a cluster, such as scheduling applications, maintaining applications' desired state, scaling applications, and rolling out new updates. Each node has a Kubelet, which is an agent for managing the node and communicating with the Kubemetes master, as well as tools for handling container operations. The Kubemetes cluster master starts the application containers and schedules the containers to run on the cluster's nodes. The nodes communicate with the master using the Kubemetes API, which the master exposes. End users can also use the Kubemetes API directly to interact with the cluster.

[0084] A “pod” is a basic execution unit of a Kubemetes application, i.e., the smallest and simplest unit that can be created and deployed in the Kubemetes object model. A pod represents processes running on a cluster and encapsulates an application’s container(s), storage resources, a unique network IP address, and options that govern how the container(s) should run. Put differently, a Kubemetes pod represents a single instance of an application, which can consist of one or more containers that are tightly coupled and that share resources.

[0085] Returning to Figure 2, BuildKit is an open source tool that takes the instructions from a Dockerfile and builds (or creates) a Docker container image. This build process can take a long time so BuildKit provides several architectural enhancements that makes it much faster, more precise, and portable. The Docker Application Programming Interface (API) and the Docker Command Line Interface (CLI) facilitate interfacing with the Docker Engine. For example, the Docker CLI enables users to manage container instances through a clear set of commands.

[0086] As shown in Figure 2, the Docker Engine also provides functions such as distribution, orchestration, and networking. The Docker Engine also provides volumes functionality, which is a preferred mechanism for persisting data generated and / or used by Docker containers. Compared to bind mounts, in which a file or directory on the host machine is mounted into a container, volumes are independent of the directory structure and OS of the host machine and are completely managed by Docker. Figure 3 shows an example computing configuration that uses the Docker Engine. At the bottom are the computing infrastructure (310, also referred to as “host”) that runs the OS (320, e.g., Windows, Linux, etc.). The Docker Engine (330) runs on top of the OS and executes applications 1-N as Docker containers (340, also referred to as “containerized applications”).

[0087] Typically, the OS is the best place to implement observability, security, and networking functionality due to the OS kernel’s privileged ability to oversee and control the entire system. At the same time, an OS kernel is difficult to evolve due to its central role and strict requirements for stability and security. Thus, the rate of innovation at OS level has been slower compared to innovation outside of the OS, such as Kubemetes, Docker Engine, etc.

[0088] In Unix-based OS such as Linux, each file-system object (e.g., file or directory) is indexed by an “inode”, which uniquely identifies a physical position of the object in the file system on each storage device. Inodes stores metadata for every file in a table-like structure usually located near the beginning of a file system partition. Every file in a directory is an entry with the filename and corresponding inode number. All other information about the file is retrieved from the inode table by referencing the inode number.

[0089] Multiple identical files on the same storage device can be represented by the same physical file system object and will thus have the same inode, which can easily be determined by Linux commands such as “Is” or “stat”. Some implementations also provide functions that determine whether two given files are reflections of the same physical file system object. In general, these functions operate by checking whether the files are on the same device and have the same inode.

[0090] The Linux kernel’s integrity subsystem consists of two major components. The Integrity Measurement Architecture (IMA) collects hashes of files when opened before their contents are accessed for read or execute. IMA stores the file hashes in kernel memory where user applications cannot access or modify them, and allows local and remote parties to verify the measured values. The Extended Verification Module (EVM) detects offline tampering of the security extended attributes.

[0091] IMA maintains a runtime measurement list and, if anchored in a hardware Trusted Platform Module (TPM), an aggregate integrity value over this list. IMA’s measurements of files are based on their inodes. One solution for integrity verification solution of Docker containers based on IMA measurements is described in “Integrity Verification of Docker Containers for a Lightweight Cloud Environment” by DeBenedictis and Lioy, published in Future Generation Computer Systems, August 2019. In this solution, each inode is only measured once even if it is linked to more than one file.

[0092] International application PCT / EP2022 / 080206 describes techniques that identify whether a certain container has been instantiated, which is done autonomously and / or independently from the container runtime environment (e.g., Docker). The disclosed techniques also perform software attestation (e.g., calculating a digest) on a set of files present within the container. For example, the computing host can detect when a new container is instantiated and then measure selected parts of that container’s file system. The host then signs the measurement with a key only accessible to the host. The signed measurement can be verified and compared against a known-good value by a verification instance within the cluster (e.g., an Attestation Verification Service). For example, the known-good value may have been previously calculated by a vendor of the container during container image creation and before delivering the container image to the intended user.

[0093] The solution described in PCT / EP2022 / 080206 has been proposed for standardization in ETSI NFV-SEC023. Although this solution may be effective for container verification, it involves complex calculations of cryptographic hashes for a relatively large number of files for each started container. Calculation of cryptographic hash functions (e.g., SHA-256) is well- known for being time consuming. As such, measurements needed for verification of a container can take a relatively long time, which has a negative impact on overall startup performance.

[0094] Embodiments of the present disclosure address these and other problems, issues, and / or difficulties by techniques that make a computing host’s measurement of a container file system, or group of files in a container, faster and lighter. Embodiments utilize similarities between containers started on a machine, either containers started from the same image or from different images that have files in common. Embodiments utilize information created during measurement of a file system of a previously started, identical, or partly identical, container to achieve a faster and lighter measurement of a container’s file system.

[0095] In more detail, instead of measuring a file (or group of files) using cryptographic hash functions to create a digest, embodiments re-use an earlier performed measurement of an identical file (or identical group of files). For example, the earlier measurement may be based on a cryptographic hash function. The file(s) from this earlier measurement will then act as reference file(s) for the determination of whether the file(s) to be measured are identical. This determination of identicality can be based on one or more of the following techniques or criterial:

[0096] • The files are two reflections of the same physical file on the file system.

[0097] • A byte-by-byte comparison of the files (which may include file properties such as file type and access rights) indicates that the files are identical.

[0098] • A cryptographic hash calculation of the files (which may include the file properties) produces an identical result. This hash calculation may be the same as the measurement that is always performed for the first started container on the machine. Embodiments can also utilize a combination of these three techniques for determining identicality. For example, the first technique (reflections of same physical file) may be slightly less secure but has a significant reduction in verification time compared to the other two. The second technique (byte-by-byte comparison) may also improve verification time relative to the third technique (cryptographic hash comparison). Even if cryptographic hashes need to be calculated for some files, being able to use the first or second technique instead of the third for a large portion of the file system can result in significant performance improvements.

[0099] As an alternative to comparing a file to be measured against a reference file, the file can be compared against stored information related to a previously measured file. For example, the stored information can be information about the physical file (first technique above) or a cryptographic hash of the file (third technique above).

[0100] Embodiments described herein provide various benefits and / or advantages. For example, embodiments may significantly reduce the resources needed for measurement and verification of multiple containers instantiated on a host computing system, thereby increasing the speed of startup when container verification is performed. Embodiments may be utilized together with or independent of existing solutions that are based on cryptographic measurements of files associated with a container. Moreover, embodiments may provide such performance improvement when used with multiple containers that are based on the same image or on different images but with some parts of the respective file systems in common. Moreover, embodiments operating at the host level may provide better security than verification performed within the container, since they prevents a container from false self-attestation.

[0101] Embodiments of the present disclosure utilize a digest based on cryptographic hash calculations of files included in a container’s file system. These calculations are also referred to as a “measurement”. Figure 4 shows a high-level flow diagram of a verification procedure for a container executed by a host computing system, according to some embodiments of the present disclosure. In block 410, the host detects startup of a container. In block 420, the host identifies the container being started. In blocks 430-440, the host determines whether measurements need to be performed on the container and, if so, the file(s) in the container file system to be measured. In block 450, the host determines whether a previous measurement of the file(s) exists. If not, the host proceeds to block 460 where a measurement is performed. If a previous measurement of the file(s) exists, the host proceeds to block 470 where a comparative measurement is performed.

[0102] The first time the files in a container’s file system are measured by a host upon container startup, a full measurement is done by concatenating cryptographic hash calculations from each file to be measured in a determined order to create a digest. This digest can be signed together with a container locator tag and sent to an Attestation Verification System (AVS) for evaluation.

[0103] Figure 5 shows an exemplary signaling diagram for a verification procedure based on a full measurement for a container executed by a host computing system (“host”, 510), according to some embodiments of the present disclosure. Although the operations shown in Figure 5 are given numerical labels, this is done to facilitate explanation rather than to require or imply a sequential order, unless stated to the contrary below.

[0104] As shown in Figure 5, the host is arranged to execute a container (520) that includes an application (522) and an attest client (524). Additionally, the host is arranged to execute a software integrity tool (530) and a container orchestrator (540). In some cases, the software integrity tool may include or be associated with a Hardware-Mediated Execution Enclave (HMEE, 532), which provides hardware-enforced isolation of both code and data.

[0105] For example, an HMEE can act as a root of trust and can be used for attestation. In such a scenario, a remote verifier can request a quote from the HMEE, possibly via an attestation agent. The HMEE will provide signed measurement data (the “quote”) to the remote verifier. The remote verifier can then verify the signature and the quote with its own stored data. In this manner, HMEE-based attestation can provide the remote verifier with assurance of the right application executing on the right platform. HMEE is further specified in ETSI GR NFV-SEC 009.

[0106] As a prerequisite, the software integrity tool is running on the host, such as illustrated in Figure 5. In operation 1, the orchestrator decides to instantiate a container instance originating from a container image. In operation 2, based on identifying a pattern at the system level, software integrity tool understands that a container runtime has instantiated a container instance. The software integrity tool also identifies a process identifier (PID) associated with the container. For example, the PID may be a Docker PID, assigned by the Docker Engine.

[0107] The detection of a container start-up in operation 2 can be implemented in various ways. In some embodiments, eBPF can be used to detect the start of new processes and recognize a certain chain of started processes indicating the start of a new container. Such embodiments are independent of container runtime software, even if they may require adaptation to support different container runtime solutions. By using eBPF, these embodiments can efficiently detect start of a new container while being fail-safe and container independent. In other embodiments, functionality in the container runtime software can be used to detect the start of new containers and to achieve the PID of the container.

[0108] After the container has been instantiated, the attest client internal to the container generates a random container locator tag in operation 3. The length should be long enough to avoid collisions. The attest client stores the container locator tag in the container (e.g., at a predefined path) and, in operation 4, sends the container locator tag to an AVS (550). Alternately, the attest client can send data that enables identification of the container locator tag, such as a digest. Note that the AVS may be external to the host (as shown) or internal to the host.

[0109] After the software integrity tool has knowledge of the PID, it performs operations 5-8. In operation 5, the software integrity tool performs measurements on the newly started container’s file system. For example, the software integrity tool can compute a digest from SHA-255 hashes of the files of the container’s file system in a predefined order. The file system of the container measured in operation 5 can be fetched on different paths of the host. One place to fetch it from is / proc / [PID] / root. Another place to fetch it from is the driver of the container runtime.

[0110] In some embodiments, a container digest policy (or more simply “digest policy”) may specify which file system folders to include in the digest computation, e.g., some files may be excluded from the digest computation. The container digest policy may have an associated security level policy that assigns one of a plurality of security levels to each file being measured, as discussed in more detail below.

[0111] In operation 6, the software integrity tool locates and reads the container locator tag from within the container, e.g., from the predefined path. In operation 7, the software integrity tool digitally signs the digest obtained in operation 5 and the container locator tag obtained in operation 6. If the software integrity tool includes or is associated with an HMEE, that can be used to provide additional security for handling of key material used for signing. In such case, only the host has access to the key needed to verify the source of the measurement.

[0112] In operation 8, the software integrity tool sends to the AVS the measurement result signed together with the container locator tag. In operation 9, the AVS attempts to match the container locator tag received in operation 8 with a tag it has received previously, e.g., in operation 4. In case there is no match or the AVS understands the container locator tag has recently been received e.g., a replay attack, the procedure would typically stop or transition into error handling. Alternately, if operation 8 occurs before operation 4, the AVS may attempt to match the later- received tag from the attest client with an earlier-received tag from the software integrity tool.

[0113] In operations 10-11, the AVS compares the received measurement value with a list of known-good values and responds to the attest client with the result, i.e., attestation success or failure. The AVS can locate the correct attest client with the help of the container locator tag, which maps to the sender of the message in operation 4. In operations 12-13, the container receives the result from the attest client and either continues container setup if attestation was successful or starts error handling if attestation failed. Figure 6 shows the full measurement (460) and comparative measurement (470) operations of Figure 4 in more detail. The full measurement operations correspond generally to the procedure shown in Figure 5, except that they also include block 461 in which the measurement result (e.g., digest) for the measured file(s) are stored for subsequent comparison measurement operations. In this manner, the next time the file(s) in the container’s file system need to be measured (e.g., upon startup of a new container based on the same image), the previously-performed measurement can be used to speed up the measurement process.

[0114] Figure 7 shows a host computing system (700) in which a first container (“Container 1”, 721) has been instantiated and is running. A software integrity tool (710) performed a measurement based on cryptographic hash calculations on selected files 1-4 from the file system of Container 1 at container initialization time, according to procedure discussed above. Several files of Container 1 are excluded from (or not included in) the measurement according to the relevant digest policy. A second container (“Container 2”, 722) is started on the same host, and its files to be measured are expected to be identical with the measured files of Container 1, e.g., because the two containers are instantiated from the same image. If these files of Container 2 are determined to be identical based on the comparative measurement operations, the digest from the measurement of Container 1 can be reused as the digest for Container 2.

[0115] The comparative measurement operations performed by the software integrity tool in the right-side flow of Figure 6 include processing each file of the second container needing to be measured for verification (block 471). The operations of file processing are described in more detail below. If all files needing to be measured are processed successfully, the measurement result from the previous full measurement of the first container from the same image is retrieved in block 472 and used by the AVS for comparison.

[0116] If the files of a second container needing to be measured are identical to the corresponding files of a first container previously measured - which is likely if they are created from the same image - the digest of the files of the second container should be identical to the digest of the files of the first container. Different techniques can be used to determine whether such identicality exists. In some embodiments, each file to be measured can be assigned one of a plurality of different security levels, which can be provided together with the digest policy mentioned above. The security level can be explicitly indicated for a file, or it can be associated with file attributes and / or file location.

[0117] Figures 8A-B illustrate the operations of block 471 where each file is processed according to an associated security level. Each security level can correspond to a different file verification technique. For example, all files assigned minimum security level 1 require at least the streamlined but less secure first comparative verification technique summarized above (i.e., reflections of same physical file), while all files assigned intermediate security level 2 require at least the second comparative verification technique summarized above (i.e., byte-by-byte comparison). In contrast, all files assigned maximum security level 3 require the third comparative verification technique summarized above (i.e., cryptographic hash comparison).

[0118] If the two containers to be compared are created from the same image, the corresponding files in these two containers are reflections of the same physical files in the host file system, as mentioned above. This can be determined using the first comparative verification technique at security level 1 (i.e., for files assigned that security level). To conclude that two files are two reflections of the same physical file on a file system, a comparison is made between (inode, device number) tuples for the two files (if both tuples are available). If the two tuples are identical - implying that the two files are on the same device and have the same inode - they are necessarily the same physical file.

[0119] In some deployments, each container may be allocated its own virtual device and associated device number, which maps to a physical device number for the host. In this case, the virtual device numbers of the two containers will not match. Even so, a file of a container is still a reflection of the corresponding file of the container image so long as it remains untouched and unchanged from when it was instantiated based on the image. Thus, if the files of the first and second containers being compared are both untouched and unchanged from the image, they are both reflections of the corresponding file of the image. Accordingly, files of a second container are determined to be untouched and unchanged based on any of the following information being identical to corresponding information for files of a first container previously instantiated and currently running on the host:

[0120] • inode;

[0121] • dates / times for Access, Modify, Change and Birth;

[0122] • file size;

[0123] • file access rights; and

[0124] • file metadata.

[0125] Comparing file metadata may be performed more quickly than comparing other types of information, but it can also be may slightly less secure since there is no comparison of actual file information.

[0126] If the file is assigned security level 2 or if verification at security level 1 has failed, the second comparative verification technique at security level 2 in Figure 8A involves a byte-by-byte comparison of file(s) of the second container and corresponding file(s) of the first container previously instantiated and running. This comparison can also include certain file properties, such as file type and access rights. This technique is slower than the first comparative verification technique at security level 1, but reliably indicates whether the files are identical and can be much faster than performing comparisons involving a cryptographic hash calculation of file(s) of the second container.

[0127] If the file is assigned security level 3 or if verification at security level 2 has failed, the third comparative verification technique at security level 3 in Figure 8B involves retrieving a digest based on the previous full measurement, performing cryptographic hash calculations to obtain a digest for files of the second container, and comparing the two digests to see if they match. If the digests do not match, the comparative measurement verification of block 471 is determined to have failed and the operation of block 470 reverts to the full measurement operations of block 460.

[0128] The arrangement of the operations in Figures 6 and 8A-B is intended to be illustrative rather than exhaustive. The operations shown in these figures may be performed in different orders than shown, and there may be additional operations not explicitly shown. Moreover, certain operations may be omitted to reduce complexity and / or due to lack of needed inputs.

[0129] As an alternative to comparing the files in the second container (according to assigned security level) with files of a previously started first container that is currently running, the files in the second container can be compared with reference data stored by the software integrity tool. For example, the reference data can be associated with a reference container. For this alternative, the following reference data is needed for the comparison:

[0130] • Security level 1 : inode and dates / times for Access, Modify, Change, and / or Birth;

[0131] • Security level 2: copy of file; and

[0132] • Security level 3 : digest of file.

[0133] One advantage of this alternative is that the reference container does not need to be running at the time of the measurement so long as its image is in storage accessible to the software integrity tool. Another advantage of this alternative is that it is independent of post-measurement modifications of the first container.

[0134] Figure 9 shows a block diagram of the host computing system (900) that implements this alternative. In this arrangement, Container 1 no longer needs to be running on the host since the software integrity tool (910) previously stored reference data from a previous measurement of Container 1. For files 1 and 2 assigned security level 3, the stored reference data are file digests. For files 3 and 4 assigned security level 1, the stored reference data are respective references to the corresponding physical files, e.g., (inode, dates / times).

[0135] Container 2 (922) is started on the same host and its files to be measured are expected to be identical to the previously measured files of Container 1. If these files of Container 2 are determined to be identical based on the comparative measurement operations, the digest from the measurement of Container 1 can be reused as the digest for Container 2. This determination is made based on comparing the stored digests of files 1 and 2 with the calculated digests of corresponding files 1 and 2 in Container 2 (security level 3), and by verifying that files 3 and 4 of Container 2 are using the physical files identified by the stored references for those files (security level 1).

[0136] Since a container image is built of layers, which may be shared between images, physical files are often shared between as described above. In some embodiments, a container measurement can be split into multiple parts, such that each measurement part is for a different group of the files of the container file system. The groups can be selected based on expected sharing of files among containers created from different images. The measurement part for each group can be a digest calculated for files of that group. Thus, the digests calculated for corresponding groups of files in two containers (e.g., from different images) can be compared using the techniques described above.

[0137] Figure 10 shows a block diagram of a host computing system (1000) that implements comparative measurements according to these embodiments. The file system of Container 1 (1021) includes two file groups, A and B, with group A including files expected to be re-used in containers created from one or more other images that share one or more layers with the image from which Container 1 was created. Digests A and B for file groups A and B are calculated by the software integrity tool (1010) when Container 1 is started, using cryptographic hash techniques such as described above.

[0138] Container 2 (1022) is instantiated from a different image than used to instantiate Container 1, but some layers in the image used for Container 2 are identical to layers of the image used for Container 1. In particular, file group A in Container 2’s file system is expected to be identical to file group A in Container 1 ’ s file system. Thus, if a measurement of Container 1 exists, the security level-specific techniques shown in Figures 8A-B can be used when measuring files from file group A of Container 2. If the comparative measurement of file group A is successful according to the security level-specific technique employed, digest A for Container 1 can be used also as digest for the files in file group A of Container 2.

[0139] In some embodiments, a container to be measured can include file groups that are common with other file groups from multiple containers that were previously measured. Figure 11 shows a block diagram of a host computing system (1100) that implements comparative measurements based on reference data according to these embodiments.

[0140] The file system of Container 1 (1121) includes two file groups, A and B, containing files expected to be re-used in containers created from one or more other images that share one or more layers with the image from which Container 1 was created. Digests A and B for file groups A and B are calculated by the software integrity tool (1010) when Container 1 is started, using cryptographic hash techniques such as described above.

[0141] The file system of Container 2 (1122) includes three file groups, C-E, containing files expected to be re-used in containers created from one or more other images that share one or more layers with the image from which Container 2 was created. In this example, file groups C-E are different from file groups A-B from Container 1. Digests C-E for file groups C-E respectively are calculated by the software integrity tool when Container 2 is started, using cryptographic hash techniques such as described above.

[0142] Container 3 (1123) is instantiated from a different image than used to instantiate containers 1 and 2, but some layers in the image used for Container 3 are identical to layers of the images used for containers 1 and 2. In particular, file group A in Container 3’s file system are expected to be identical to file group A in Container l’s file system, and file group E in Container 3’s file system is expected to be identical to file group E in Container 2’s file system.

[0143] Thus, if measurements of containers 1 and 2 exist, the security level-specific techniques shown in Figures 8A-B can be used when measuring files from file groups A and E of Container 3. If the comparative measurement is successful, digest A for Container 1 and digest E for Container 2 can be used also as digests for the files in file groups A and E, respectively, of Container 3. Additionally, Container 3 includes another file group F that is not in common with either of containers 1-2, such that no measurement of file group F exists. In this case, the software integrity tool must perform a full measurement to obtain a digest for file group F of Container 3.

[0144] Figure 12 shows a block diagram of a host computing system (1200) that implements comparative measurements based on reference data according to these embodiments. In this arrangement, Container 1 does not need to be available when Container 2 (1222) is measured. Instead, the stored reference data (e.g., combination of references to physical files and digests of files) from the measurement of file group A of Container 1 is used by the software integrity tool (1210) for a comparative measurement of file group A of Container 2 in the manner discussed above. If the outcome is successful, digest A for file group A of Container 1 can also be used as a digest for file group A of Container 2. Since the reference data does not include a digest for file group B’ of Container 2 (which may be different than file group B of Container 1), a full measurement needs to be performed for file group B’ in the manner described above.

[0145] Note that the arrangements shown in Figures 7 and 9-12 are simplified examples. In actual implementations, a host computing system may host many containers that are created from the same image or from different images. Moreover, the file systems of the containers may include a relatively large number of files to be measured. In some embodiments, each file group of a container can be limited to files that are assigned the same security level. For example, all files assigned to security level 1 can be part of file group A with digest A, all files assigned to security level 2 can be part of file group B with digest B, and all files assigned to security level 3 can be part of file group C with digest C. In this way, each digest A, B, and C is associated with a single security level. In a variant, certain file groups can be limited to all files of a single security level that are also expected to be shared with other containers created from different images.

[0146] Figure 13 shows an example arrangement of reference data stored by a software integrity tool (1310), according to some embodiments of the present disclosure. For each file, the software integrity tool stores the associated security level and storage policy, along with the reference data for that file. For a file of security level 1, the stored reference data includes information identifying the corresponding physical file, which in a Linux system can be inode and dates / times of access, modify, change, and / or birth. This file reference data for security level 1 may also be referred to as a “file representation”. For a file assigned security level 2, a reference to a comparable file is stored. For a file assigned security level 3, a file digest is stored.

[0147] In some embodiments, the file digest may also be stored for files assigned security level 1, to facilitate using the technique for security level 3 as a backup method. Since calculating this digest consumes the extra time needed for the cryptographic hash calculations of the full measurement, it may be beneficial to limit the number of hashes stored for files assigned security level 1 or 2.

[0148] Figure 14 shows another example arrangement of reference data stored by a software integrity tool (1410), according to some embodiments of the present disclosure. In this example, all files are assigned either security level 2 or security level 3. For example, security level 1 may not be used if it is infeasible to obtain the file representations for the physical files on the host computing system.

[0149] In some embodiments, a presumed digest for a container’s file system can be created at the same time the container’s image is created on the host machine. If the container includes multiple file groups as discussed above, presumed digests for one or more of these file groups can be created at that time. After creation, the digest(s) are stored by the software integrity tool together with information used for comparative measurement and verification according to the respective security levels assigned to the files that make up the container’s file system.

[0150] At container execution time, the presumed digest can be used for the initial container started from a certain image, or the initial started container that includes a file group associated with the presumed digest. The digest is used after it is verified according to security levels assigned to the respective files that make up the container’s file system (or file groups therein). Implementation of presumed digest at image creation time can be preferable in cases where the image creation is less time-sensitive and / or resource-critical than container creation and startup.

[0151] The information used for comparative measurement and verification according to security levels assigned to files can be obtained in different ways in various embodiments. In some embodiments, the information is obtained when an initial container is started from an image. In other embodiments, the information is obtained when the image is created or loaded in the host computing system. In other embodiments, the information is obtained from files of a running container, e.g., the initial container started based on a certain image.

[0152] In some embodiments, the “reference container” associated with the information used for comparative measurement and verification can be fixed and / or non-transferable, such as the initial container started based on a certain image. In other embodiments, it may be beneficial to have a more dynamic choice of reference container. For example, the reference container for a given image can be the most recent container started from that image. This arrangement reduces the risk that one or more files of the reference container have been changed since the reference container was measured.

[0153] Post-measurement file changes in the reference container can cause various problems. For example, one or more of the files in the reference container may be changed during normal container execution, which can make the container useless as reference. Even without these expected file changes, a container becomes riskier to use as a reference as it ages, since it has been exposed longer to potential attacks that can corrupt the container’s files.

[0154] The following example illustrates some embodiments of the present disclosure in more detail. In this example, 100 containers are started from the same image on a Kubernetes multinode cluster, with 20 of these containers being assigned to the same worker node. Each container will need a key from the AVS to become functional. To obtain this key, the AVS will evaluate a measurement of each container as described above in relation to Figure 5. The software integrity tool running on the worker node obtains the digest for each of the 20 started containers, and sends each digest to the AVS for evaluation.

[0155] In embodiments of the present disclosure, the software integrity tool performs a full measurement of the initial one of the started containers by calculating a digest from SHA-256 hashes (also referred to as “SHA-256 digest”) from the files of the container’s file system in a predefined order. The digest calculation may exclude some files of the container’s file system, according to digest policy. The digest calculated for the initial container is sent to AVS for evaluation.

[0156] In addition, a security policy assigns a security level (e.g., 1-3) to each file to be measured in the container file system. For each measured file, the software integrity tool stores information corresponding to the security level assigned to the file. For a file of security level 1, the stored information (i.e., the file representation) identifies the corresponding physical file, which in a Linux system can be inode, dates / times of access / modify / change / birth, file size, file access rights, path to corresponding image file, etc. For a file assigned security level 2, a copy of the file or a reference to a comparable file is stored. For a file assigned security level 3, an SHA-256 digest of the file is stored.

[0157] When the next container created from the same image is started, the software integrity tool performs a comparative measurement based on the stored information associated with the initial container. Since the next container is created from the same image as the initial container, the software integrity tool assumes that the SHA-256 digest calculated for the initial container can also be used for this next container. To verify this, the software integrity tool compares each file to be included in the container measurement with the stored information for the corresponding file of the initial container.

[0158] If the file is assigned security level 1, the software integrity tool verifies that the file representation for the file is the same as the stored file representation for the corresponding file of the initial container. As an alternative, the software integrity tool verifies that the file representation for the file is the same as the corresponding information for the image used to create the initial container.

[0159] If the file is assigned security level 2, the software integrity tool performs a byte-by-byte comparison (“diff ’ operation) between the file and the stored copy (or comparable file) to verify that the files are identical. If the file is assigned security level 3, the software integrity tool verifies that the file’s SHA-256 digest is identical to the stored SHA-256 digest for the corresponding file of the initial container.

[0160] If the software integrity tool’s verification is successful for all files of the next container, the comparative measurement is considered successful and the SHA-256 digest calculated and stored for the initial container is considered valid also for the next container. The software integrity tool then sends this stored digest to the AVS for evaluation in relation to this next container.

[0161] In this example, the above-described procedure is repeated for each container started after the initial container but from the same image as used to start the initial container.

[0162] The following example illustrates other embodiments of the present disclosure in more detail. In this example, 10 containers are started from different images on the same worker node of a Kubernetes multi-node cluster. Each container will need a key from the AVS to become functional. To obtain this key, the AVS will evaluate a measurement of each container as described above in relation to Figure 5. The software integrity tool running on the worker node obtains the digest for each of the 10 started containers, and sends each digest to the AVS for evaluation. In this example, even though the containers are created from different images, they have some layers in common such that large parts of their respective file systems are identical. The existence of common layers can be determined, for example, by inspecting the images (e.g., Docker images) used for the containers. The identical parts of the file systems for 10 containers use the same physical files. In this example, the files of each container file system are divided into multiple groups, with a single group containing all files that are expected to be identical among the containers created from the different images. A digest policy may be used to assign files to file groups to be measured and / or to exclude files that are not measured. As an illustrative numerical example, the files of the initially started container are divided into three groups and the files of the next started container are divided into two group, with a first group for each container including files expected to be identical.

[0163] In embodiments of the present disclosure, the software integrity tool performs a full measurement of the initial one of the started containers by calculating three digests from SHA- 256 hashes (also referred to as “SHA-256 digest”) of the respective file groups of the container’s file system, in a predefined file order for each group. The three group digests calculated for the initial container are combined into a full digest and sent to AVS for evaluation.

[0164] In addition, a security policy assigns a security level (e.g., 1-3) to each file to be measured in the container file system. For each measured file, the software integrity tool stores information corresponding to the security level assigned to the file. For a file of security level 1, the stored information (i.e., the file representation) identifies the corresponding physical file, which in a Linux system can be inode, dates / times of access / modify / change / birth, file size, file access rights, path to corresponding image file, etc. For a file assigned security level 2, a copy of the file or a reference to a comparable file is stored. For a file assigned security level 3, an SHA-256 digest of the file is stored.

[0165] When the next container created from the same image is started, the software integrity tool performs a comparative measurement of the first group based on the stored information associated with the first group of the initial container. Since the first groups are expected to have identical files, the software integrity tool assumes that the SHA-256 digest calculated for the first group of the initial container can also be used for the first group of this next container. To verify this, the software integrity tool compares relevant information for each file in this next container’s first group with stored information for the corresponding file of the initial container’s first group.

[0166] If the file is assigned security level 1, the software integrity tool verifies that the file representation for the file is the same as the stored file representation for the corresponding file of the initial container’s first group. As an alternative, the software integrity tool verifies that the file representation for the file is the same as the corresponding information for the image used to create the initial container.

[0167] If the file is assigned security level 2, the software integrity tool performs a byte-by-byte comparison (“diff ’ operation) between the file and the stored copy (or comparable file) to verify that the files are identical. If the file is assigned security level 3, the software integrity tool verifies that the file’s SHA-256 digest is identical to the stored SHA-256 digest for the corresponding file of the initial container’s first group.

[0168] If the software integrity tool’s verification is successful for all files of the next container’s first group, the comparative measurement is considered successful and the SHA-256 digest calculated and stored for the initial container’s first group is considered valid also for the next container’s first group. The software integrity tool performs a full measurement to obtain an SHA- 256 digest of all files in the second group of this next container, since no stored information exists that would facilitate comparative measurement of the second group. The software integrity tool then then combines the two group digests into a full digest, which is sent to AVS for evaluation in relation to this next container.

[0169] In this example, the above-described procedure is repeated for each container that is started after the initial container and includes the first group of files that are common with the initial container.

[0170] The embodiments described above can be further illustrated with reference to Figure 15 (in parts A and B), which depicts an exemplary method (e.g., procedures) for a software integrity tool of a host computing system configured with a runtime environment arranged to execute containers that include applications. Put differently, various features of the operations described below correspond to various embodiments described above. The exemplary method shown in Figure 15 can be used cooperatively with other procedures described herein to provide benefits, advantages, and / or solutions to problems described herein. Although the exemplary method is illustrated in Figure 15 by specific blocks in a particular order, the operations corresponding to the blocks can be performed in different orders than shown and can be combined and / or divided into blocks and / or operations having different functionality than shown. Optional blocks and / or operations are indicated by dashed lines.

[0171] The exemplary method includes the operations of block 1520, where the software integrity tool obtains reference data associated with a first plurality of files. The first plurality of files are from respective file systems of at least one first container. For example, when only a single first container is used, the first plurality of files are from the file system of the single first container. As another example, when two first containers are used, the first plurality of files are from the respective file systems of the two first containers. For each of the first plurality of files, the reference data includes information needed for a comparative measurement according to a security level assigned to the file.

[0172] The exemplary method also includes the operations of blocks 1530-1540, where the software integrity tool obtains a second container having a file system with a second plurality of files and performs a comparative measurement of at least a portion of the second plurality of files based on the reference data associated with the first plurality of files. The exemplary method also includes the operations of block 1550, where when the comparative measurement indicates success, the software integrity tool uses at least one previous measurement performed on the respective at least one first container as a measurement of the least a portion of the file system of the second container.

[0173] In some embodiments, the first and second pluralities of files include one or more common file groups. In other embodiments, the first plurality of files are from a file system of a single first container, with the first container and the second container being based on a common container image.

[0174] In some embodiments, performing the comparative measurement based on the reference data comprises, for each file of the at least a portion of the second plurality of files, performing the comparative measurement of the file based on the following: a security level assigned to the file, and reference data for a corresponding file of the first plurality (i.e., a file of the first plurality that should be identical to the file of the second plurality that is subject to the comparative measurement). In some of these embodiments, for each file of the at least a portion of the second plurality of files, the assigned security level is obtained based on one of the following:

[0175] • an explicit indication in a container image associated with the second container;

[0176] • implicit from attributes of the file; or

[0177] • implicit from storage location of the file.

[0178] In some of these embodiments, performing the comparative measurement of the file based on the assigned security level in sub-block 1541 includes determining or calculating a representation of the file based on the assigned security level, and comparing the representation of the file to the reference data for the corresponding file of the first plurality.

[0179] In some variants of these embodiments, the comparative measurement of the at least a portion of the second plurality of files indicates success when the comparative measurement of each file indicates a match between the representation of the file and the reference data for the corresponding file of the first plurality. In some variants of these embodiments, the representation of the file based on the assigned security level is one of the following: • identifying information for a physical file in the host computing system, when a first security level is assigned;

[0180] • the file, when a second security level is assigned; or

[0181] • a digest for the file, when a third security level is assigned.

[0182] In some variants of these embodiments, the reference data for each file of the first plurality includes one of the following:

[0183] • identifying information for a physical file in the host computing system, when a first security level (e.g., security level 1) is assigned;

[0184] • a copy of the file or a reference to a corresponding file, when a second security level (e.g., security level 2) is assigned; or

[0185] • a digest for the file, when a third security level (e.g., security level 3) is assigned.

[0186] In some further variants, the identifying information for a physical file in the host computing system includes one or more of the following:

[0187] • an identifier of a storage device in the host computing system, on which the physical file is stored;

[0188] • a unique identifier of a position of the physical file on the storage device;

[0189] • dates and times for one or more of the following operations on the physical file: Access, Modify, Change, and Birth; and

[0190] • one or more of the following characteristics of the physical file: size, access rights, and metadata.

[0191] In some further variants, a match between the representation of the file and the reference data for the corresponding file of the first plurality is based on one of the following:

[0192] • when the first security level is assigned, the file and the corresponding file are the same physical file in the host computing system;

[0193] • when the second security level is assigned, a byte-by-byte match between contents of the file and contents of the corresponding file; or

[0194] • when the third security level is assigned, a byte-by-byte match between a digest for the file and the digest for the corresponding file.

[0195] In some further variants, when the second security level is assigned, a match between the representation of the file and the reference data for the corresponding file is further based on a match between one or more of the following properties of the file and the corresponding file: file type, and access rights.

[0196] In some embodiments, the exemplary method also includes the operations of block 1560, where when the comparative measurement indicates failure or does not indicate success, the software integrity tool performs a measurement of the at least a portion of the file system of the second container. In some of these embodiments, performing the measurement of the at least a portion of the file system of the second container in block 1560 includes the following operations, labelled with corresponding sub-block numbers:

[0197] • (1562) calculating respective cryptographic hashes of the files comprising the at least a portion of the file system of the second container; and

[0198] • (1563) combining the cryptographic hashes to derive, as the measurement, a digest for the at least a portion of the file system of the second container.

[0199] In some variants of these embodiments, performing the measurement of the at least a portion of the file system of the second container in block 1560 includes the operations of subblock 1561, where the software integrity tool selects the files, of which the cryptographic hashes are computed, based on a container digest policy of the host computing system.

[0200] In some embodiments, using the at least one previous measurement in block 1550 includes the operations of sub-block 1563, where the software integrity tool sends a completed measurement of the second container to an attestation verification system (AVS) for verification of the file system of the second container. The completed measurement includes the at least one previous measurement.

[0201] In some of these embodiments, the completed measurement of the second container is sent to the AVS together with a container locator tag for the second container. In such embodiments, using the at least one previous measurement in block 1550 also includes the operations of sub-block 1552, where the software integrity tool digitally signs the container locator tag and the completed measurement before sending to the AVS, based on key material that is accessible to the host computing system but is not accessible to containers configured to execute in the runtime environment. In some variants of these embodiments, the digital signing is performed by a Hardware-Mediated Execution Enclave (HMEE) associated with the software integrity tool, such as discussed above.

[0202] In some of these embodiments, the first plurality of files are from a file system of a single first container, with the first container and the second container being based on a common container image. In such case, the at least one previous measurement is a single previous measurement, which is performed on all files of the first plurality that are included based on a container digest policy associated with the host computing system. Also, when the comparative measurement indicates success, the previous measurement is used as a measurement of all files, of the second plurality, that are included based on the container digest policy.

[0203] In other of these embodiments, the first and second pluralities of files include one or more common file groups, and the at least one previous measurement is performed on the files of the one or more common file groups in the respective file systems of the at least one first container. In such case, when the comparative measurement indicates success, the at least one previous measurement is used as a measurement of the one or more common file groups in the file system of the second container. In some variants of these embodiments, the second container and the at least one first container are based on different container images.

[0204] In some variants of these embodiments, the at least one first container includes a plurality of first containers and the one or more common file groups include a plurality of common file groups. In such case, each of the common file groups is from a file system of one of the plurality of first containers. In this manner, previous measurements of common file groups from multiple containers can be used for a newly instantiated container, such as illustrated in Figure 11.

[0205] In some variants of these embodiments, the second plurality of files includes one or more second file groups in addition to the one or more common file groups. In such case, the exemplary method also includes the operations of block 1545, where the software integrity tool performs measurements (i.e., full measurements) on each of the second file groups. Moreover, using the at least one previous measurement in block 1550 includes the operations of sub-block 1551, where the software integrity tool obtaining the completed measurement of the second container (e.g., used in sub-blocks 1552-1553) based on combining the measurements performed on the one or more second file groups (e.g., in block 1545) with the at least one previous measurement of the one or more common file groups.

[0206] In some embodiments, obtaining the reference data associated with the first plurality of files in block 1520 is performed, for each first container of the at least one first container, according to one of the following:

[0207] • from the first container while it is running in in the runtime environment;

[0208] • from the first container when it was started in the runtime environment; or

[0209] • from a container image associated with the first container.

[0210] In some of these embodiments, one of the following applies for each first container of the at least one first container:

[0211] • the reference data is obtained from the first container when it is the initial container started in the runtime environment based on the container image;

[0212] • the reference data is obtained from the first container when it is the most recent container started in the runtime environment based on the container image; or

[0213] • the reference data is obtained from the container image when the container image is created or loaded in the host computing system.

[0214] In some embodiments, obtaining the second container in block 1530 includes the following operations, labelled with corresponding sub-block numbers: • (1531) monitoring for one or more events or patterns indicating that a container has been instantiated in the runtime environment; and

[0215] • (1532) in response to detecting the one or more events or patterns, obtaining an identifier of the second container which has been instantiated.

[0216] In some of these embodiments, the identifier of the second container is a process identifier (PID) and the file system of the second container has a pathname that includes the PID.

[0217] In some embodiments, the exemplary method also includes the operations of block 1510, wherein the software integrity tool performs the at least one previous measurement on the respective at least one first container (e.g., used in block 1550), based on the following operations for each first container of the at least one first container (labelled with corresponding sub-block numbers):

[0218] • (1511) calculating respective cryptographic hashes of the files of the first plurality that are from the file system of the first container; and

[0219] • (1512) combining the cryptographic hashes to derive a digest, which is the previous measurement on the first container.

[0220] Although Figure 15 describes a method (e.g., procedure), the operations corresponding to the method (including any blocks and sub-blocks) can also be embodied in a non-transitory, computer-readable medium storing computer-executable instructions. The operations corresponding to the method (including any blocks and sub-blocks) can also be embodied in a computer program (e.g., computer program product) comprising computer-executable instructions. In either case, when such instructions are executed by processing circuitry associated with a host computing system, they can configure the host computing system (or components thereof) to perform operations corresponding to the method.

[0221] Figure 16 is a block diagram illustrating a host computing system 1600 according to some embodiments of the present disclosure.

[0222] In some embodiments, some or all of the functions described herein can be implemented as components executed in runtime environment 1620 hosted by one or more of hardware nodes 1630. Such hardware nodes can be computing machines arranged in a cluster (e.g., such as in a data center or customer premise equipment (CPE)) where many hardware nodes work together and are managed via management and orchestration (MANO) 16100, which, among others, oversees lifecycle management of applications. Runtime environment 1620 can run on top of an operating system (OS) 1625, such as Linux or Windows, which runs directly on hardware nodes 1630. Alternately, OS 1625 may run on a virtual machine (VM), which abstracts one or more of the hardware nodes 1630. Hardware nodes 1630 can include processing circuitry 1660 and memory 1690. Memory 1690 contains instructions 1695 executable by processing circuitry 1660 whereby application 1640 can be operative for various features, functions, procedures, etc. of the embodiments disclosed herein. Processing circuitry 1660 can include general -purpose or special -purpose hardware devices such as one or more processors (e.g., custom and / or commercial off-the-shelf), dedicated Application Specific Integrated Circuits (ASICs), or any other type of processing circuitry including digital or analog hardware components or special purpose processors. Each memory 1690 of a hardware node can comprise memory 1690-1 which can be non-persistent memory for temporarily storing software code 1695 executed by processing circuitry 1660 of the hardware node. For example, software code 1695 can include program instructions (also referred to as a computer program product) that, when executed by processing circuitry 1660, can configure hardware node 1630 to perform operations corresponding to the methods / procedures described herein.

[0223] Each hardware node can comprise one or more network interface controllers (NICs) / network interface cards 1670, which include physical network interface 1680. Each memory 1690 of a hardware node can also include non-transitory, persistent, machine-readable storage media 1690-2 having stored therein software code 1695 executable by processing circuitry 1660. Software code 1695 can include any type of software including operating system 1625, runtime environment 1620, software integrity tool 1650, and containerized applications 1640.

[0224] Various applications 1642 (which can alternatively be called software instances, virtual appliances, network functions, virtual nodes, virtual network functions, containers, containerized applications, efc.) can be executed by host computing system 1600. Each application 1641 can be included in a corresponding container 1641, such as applications 1642a-b in containers 1641a-b shown in Figure 16. Note that in some instances applications 1642 can represent services. Each container 1641 can also include an attest client 1643, such as attest clients 1643a-b in containers 1641a-B shown in Figure 16.

[0225] In some embodiments, runtime environment 1620 can be used to abstract applications 1642 and containers 1641 from the underlying hardware nodes 1630. In such embodiments, processing circuitry 1660 executes software 1695 to instantiate runtime environment 1620, which can in some instances be a Docker Runtime. For example, runtime environment 1620 can appear like computing and / or networking hardware to containers and / or pods hosted by host computing system 1600.

[0226] In some embodiments, multiple application containers 1641 can be arranged in a pod 1640. In such embodiments, pod 1640 (e.g., a Kubernetes pod) can be a basic execution unit, i.e., the smallest and simplest unit that can be created and deployed in host computing system 1600. This may be the case, for instance, when multiple containers 1641 encapsulate services that are used are building blocks for a higher-level application, represented by pod 1640.

[0227] Each pod can include a plurality of resources shared by containers within the pod. For example, a pod can represent processes running on a cluster and can encapsulates container(s) (including applications / services therein), storage resources, a unique network IP address, and options that govern how the container(s) should run. In general, containers can be relatively decoupled from underlying physical or virtual computing infrastructure.

[0228] Attest clients 1643 can include, but are not limited to, various features, functions, structures, configurations, etc. of various attest client embodiments shown in various other figures and discussed in more detail above.

[0229] In addition to the applications 1640, a software integrity tool 1650 can also be run in the host computing system 1600 shown in Figure 16. Software integrity tool 1650 can include, but is not limited to, various features, functions, structures, configurations, etc. of various software integrity tool embodiments shown in various other figures and discussed in more detail above.

[0230] In some embodiments, the host computing system can include an attestation verification system (AVS) 1655. For example, AVS 1655 can be executed on hardware nodes 1630 of host computing system 1600. Alternately, the AVS can be executed on hardware external to host computing system 1600, which may be similar to the hardware shown in Figure 16. Moreover, AVS 1655 can include, but is not limited to, various features, functions, structures, configurations, etc. of various AVS embodiments shown in various other figures and discussed in more detail above.

[0231] The foregoing merely illustrates the principles of the disclosure. Various modifications and alterations to the described embodiments will be apparent to those skilled in the art in view of the teachings herein. It will thus be appreciated that those skilled in the art will be able to devise numerous systems, arrangements, and procedures that, although not explicitly shown or described herein, embody the principles of the disclosure and can be thus within the spirit and scope of the disclosure. Various embodiments can be used together with one another, as well as interchangeably therewith, as should be understood by those having ordinary skill in the art.

[0232] The term unit, as used herein, can have conventional meaning in the field of electronics, electrical devices and / or electronic devices and can include, for example, electrical and / or electronic circuitry, devices, modules, processors, memories, logic solid state and / or discrete devices, computer programs or instructions for carrying out respective tasks, procedures, computations, outputs, and / or displaying functions, etc., such as those that are described herein.

[0233] Any appropriate steps, methods, features, functions, or benefits disclosed herein may be performed through one or more functional units or modules of one or more virtual apparatuses. Each virtual apparatus may comprise a number of these functional units. These functional units may be implemented via processing circuitry, which may include one or more microprocessor or microcontrollers, as well as other digital hardware, which may include Digital Signal Processor (DSPs), special-purpose digital logic, and the like. The processing circuitry may be configured to execute program code stored in memory, which may include one or several types of memory such as Read Only Memory (ROM), Random Access Memory (RAM), cache memory, flash memory devices, optical storage devices, etc. Program code stored in memory includes program instructions for executing one or more telecommunications and / or data communications protocols as well as instructions for carrying out one or more of the techniques described herein. In some implementations, the processing circuitry may be used to cause the respective functional unit to perform corresponding functions according one or more embodiments of the present disclosure.

[0234] As described herein, device and / or apparatus can be represented by a semiconductor chip, a chipset, or a (hardware) module comprising such chip or chipset; this, however, does not exclude the possibility that a functionality of a device or apparatus, instead of being hardware implemented, be implemented as a software module such as a computer program or a computer program product comprising executable software code portions for execution or being run on a processor. Furthermore, functionality of a device or apparatus can be implemented by any combination of hardware and software. A device or apparatus can also be regarded as an assembly of multiple devices and / or apparatuses, whether functionally in cooperation with or independently of each other. Moreover, devices and apparatuses can be implemented in a distributed fashion throughout a system, so long as the functionality of the device or apparatus is preserved. Such and similar principles are considered as known to a skilled person.

[0235] Unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this disclosure belongs. It will be further understood that terms used herein should be interpreted as having a meaning that is consistent with their meaning in the context of this specification and the relevant art and will not be interpreted in an idealized or overly formal sense unless expressly so defined herein.

[0236] In addition, certain terms used in the present disclosure, including the specification and drawings, can be used synonymously in certain instances (e.g., “data” and “information”). It should be understood, that although these terms (and / or other terms that can be synonymous to one another) can be used synonymously herein, there can be instances when such words can be intended to not be used synonymously.

Claims

CLAIMS1. A method for a software integrity tool of a host computing system configured with a runtime environment arranged to execute containers that include applications, the method comprising: obtaining (1520) reference data associated with a first plurality of files, wherein: the first plurality of files are from respective file systems of at least one first container, and for each of the first plurality of files, the reference data includes information needed for a comparative measurement according to a security level assigned to the file; obtaining (1530) a second container having a file system with a second plurality of files; performing (1540) a comparative measurement of at least a portion of the second plurality of files based on the reference data associated with the first plurality of files; and when the comparative measurement indicates success, using (1550) at least one previous measurement performed on the respective at least one first container as a measurement of the least a portion of the file system of the second container.

2. The method of claim 1, wherein one of the following applies: the first and second pluralities of files include one or more common file groups, or the first plurality of files are from a file system of a single first container, with the first container and the second container being based on a common container image.3 The method of any of claims 1-2, wherein performing (1540) the comparative measurement based on the reference data comprises, for each file of the at least a portion of the second plurality of files, performing (1541) the comparative measurement of the file based on the following: a security level assigned to the file, and reference data for a corresponding file of the first plurality.

4. The method of claim 3, wherein for each file of the at least a portion of the second plurality of files, the assigned security level is obtained based on one of the following: an explicit indication in a container image associated with the second container; implicit from attributes of the file; orimplicit from storage location of the file.

5. The method of any of claims 3-4, wherein performing (1541) the comparative measurement of the file based on the assigned security level comprises: determining or calculating a representation of the file based on the assigned security level; and comparing the representation of the file to the reference data for the corresponding file of the first plurality.

6. The method of claim 5, wherein the comparative measurement of the at least a portion of the second plurality of files indicates success when the comparative measurement of each file indicates a match between the representation of the file and the reference data for the corresponding file of the first plurality.

7. The method of any of claims 5-6, wherein the representation of the file based on the assigned security level is one of the following: identifying information for a physical file in the host computing system, when a first security level is assigned; the file, when a second security level is assigned; or a digest for the file, when a third security level is assigned.

8. The method of any of claims 5-7, wherein the reference data for each file of the first plurality includes one of the following: identifying information for a physical file in the host computing system, when a first security level is assigned; a copy of the file or a reference to a corresponding file, when a second security level is assigned; or a digest for the file, when a third security level is assigned.

9. The method of any of claims 7-8, wherein the identifying information for a physical file in the host computing system includes one or more of the following: an identifier of a storage device in the host computing system, on which the physical file is stored; a unique identifier of a position of the physical file on the storage device;dates and times for one or more of the following operations on the physical file: Access, Modify, Change, and Birth; and one or more of the following characteristics of the physical file: size, access rights, and metadata.

10. The method of any of claims 7-9, wherein a match between the representation of the file and the reference data for the corresponding file of the first plurality is based on one of the following: when the first security level is assigned, the file and the corresponding file are the same physical file in the host computing system; when the second security level is assigned, a byte-by-byte match between contents of the file and contents of the corresponding file; or when the third security level is assigned, a byte-by-byte match between a digest for the file and the digest for the corresponding file.

11. The method of claim 10, wherein when the second security level is assigned, a match between the representation of the file and the reference data for the corresponding file is further based on a match between one or more of the following properties of the file and the corresponding file: file type, and access rights.

12. The method of any of claims 1-11, further comprising, when the comparative measurement indicates failure or does not indicate success, performing (1560) a measurement of the at least a portion of the file system of the second container.

13. The method of claim 12, wherein performing (1560) the measurement of the at least a portion of the file system of the second container comprises: calculating (1562) respective cryptographic hashes of the files comprising the at least a portion of the file system of the second container; and combining (1563) the cryptographic hashes to derive, as the measurement, a digest for the at least a portion of the file system of the second container.

14. The method of claim 13, wherein performing (1560) the measurement of the at least a portion of the file system of the second container comprises selecting (1561) the files, of which the cryptographic hashes are computed, based on a container digest policy of the host computing system.

15. The method of any of claims 1-14, wherein using (1550) the at least one previous measurement comprises sending (1553) a completed measurement of the second container to an attestation verification system, AVS, for verification of the file system of the second container, wherein the completed measurement includes the at least one previous measurement.

16. The method of claim 15, wherein: the completed measurement of the second container is sent to the AVS together with a container locator tag for the second container; and using (1550) the at least one previous measurement further comprises digitally signing (1552) the container locator tag and the completed measurement before sending to the AVS, based on key material that is accessible to the host computing system but is not accessible to containers configured to execute in the runtime environment.

17. The method of claim 16, wherein the digital signing is performed by a Hardware- Mediated Execution Enclave, HMEE, associated with the software integrity tool.

18. The method of any of claims 15-17, wherein: the first plurality of files are from a file system of a single first container, with the first container and the second container being based on a common container image; the at least one previous measurement is a single previous measurement; the previous measurement is performed on all files, of the first plurality, that are included based on a container digest policy associated with the host computing system; and when the comparative measurement indicates success, the previous measurement is used as a measurement of all files, of the second plurality, that are included based on the container digest policy.

19. The method of any of claims 15-17, wherein: the first and second pluralities of files include one or more common file groups; the at least one previous measurement is performed on the files of the one or more common file groups in the respective file systems of the at least one first container; andwhen the comparative measurement indicates success, the at least one previous measurement is used as a measurement of the one or more common file groups in the file system of the second container.

20. The method of claim 19, wherein the second container and the at least one first container are based on different container images.

21. The method of any of claims 19-20, wherein: the at least one first container includes a plurality of first containers; the one or more common file groups include a plurality of common file groups; and each of the common file groups is from a file system of one of the plurality of first containers.

22. The method of any of claims 19-21, wherein: the second plurality of files includes one or more second file groups in addition to the one or more common file groups; the method further comprises performing (1545) measurements on each of the second file groups; and using (1550) the at least one previous measurement further comprises obtaining (1551) the completed measurement of the second container based on combining the measurements performed on the one or more second file groups with the at least one previous measurement of the one or more common file groups.

23. The method of any of claims 1-22, wherein obtaining (1520) the reference data associated with the first plurality of files is performed, for each first container of the at least one first container, according to one of the following: from the first container while it is running in in the runtime environment; from the first container when it was started in the runtime environment; or from a container image associated with the first container.

24. The method of claim 23, wherein for each first container of the at least one first container, one of the following applies: the reference data is obtained from the first container when it is the initial container started in the runtime environment based on the container image;the reference data is obtained from the first container when it is the most recent container started in the runtime environment based on the container image; or the reference data is obtained from the container image when the container image is created or loaded in the host computing system.

25. The method of any of claims 1-24, wherein obtaining (1530) the second container comprises: monitoring (1531) for one or more events or patterns indicating that a container has been instantiated in the runtime environment; and in response to detecting the one or more events or patterns during the monitoring (1531), obtaining (1532) an identifier of the second container which has been instantiated.

26. The method of claim 25, wherein the identifier of the second container is a process identifier, PID, and the file system of the second container has a pathname that includes the PID.

27. The method of any of claims 1-26, further comprising performing (1510) the at least one previous measurement on the respective at least one first container, based on the following operations for each first container of the at least one first container: calculating (1511) respective cryptographic hashes of the files of the first plurality that are from the file system of the first container; and combining (1512) the cryptographic hashes to derive a digest, which is the previous measurement on the first container.

28. A host computing system (310, 510, 700, 900, 1000, 1100, 1200, 1600) configured with a runtime environment (330, 1620) arranged to execute containers (340, 520, 721, 722, 922, 1021, 1022, 1121, 1122, 1123, 1222, 1641) that include applications (522, 1642), the host computing system comprising: memory (1690) storing computer-executable software code (1695) for a software integrity tool (530, 710, 910, 1010, 1110, 1210) and for the runtime environment; and processing circuitry (1660) configured to execute the software code, wherein execution of the software code configures the host computing system to: obtain reference data associated with a first plurality of files, wherein:the first plurality of files are from respective file systems of at least one first container (721, 1021, 1121, 1122), and for each of the first plurality of files, the reference data includes information needed for a comparative measurement according to a security level assigned to the file; obtain a second container (722, 922, 1022, 1123, 1222) having a file system with a second plurality of files; perform a comparative measurement of at least a portion of the second plurality of files based on the reference data associated with the first plurality of files; and when the comparative measurement indicates success, use at least one previous measurement performed on the respective at least one first container as a measurement of the least a portion of the file system of the second container.

29. The host computing system of claim 28, wherein execution of the software code further configures the host computing system to perform operations corresponding to any of the methods of claims 2-27.

30. A host computing system (310, 510, 700, 900, 1000, 1100, 1200, 1600) configured with a runtime environment (330, 1620) arranged to execute containers (340, 520, 721, 722, 922, 1021, 1022, 1121, 1122, 1123, 1222, 1641) that include applications (522, 1642), the host computing system being further configured to: obtain reference data associated with a first plurality of files, wherein: the first plurality of files are from respective file systems of at least one first container (721, 1021, 1121, 1122), and for each of the first plurality of files, the reference data includes information needed for a comparative measurement according to a security level assigned to the file; obtain a second container (722, 922, 1022, 1123, 1222) having a file system with a second plurality of files; perform a comparative measurement of at least a portion of the second plurality of files based on the reference data associated with the first plurality of files; andwhen the comparative measurement indicates success, use at least one previous measurement performed on the respective at least one first container as a measurement of the least a portion of the file system of the second container.

31. The host computing system of claim 30, being further configured to perform operations corresponding to any of the methods of claims 2-27.

32. A non-transitory, computer-readable medium (1690-2) storing software code for a software integrity tool (530, 710, 910, 1010, 1110, 1210) of a host computing system (310, 510, 700, 900, 1000, 1100, 1200, 1600) configured with a runtime environment (330, 1620) arranged to execute containers (340, 520, 721, 722, 922, 1021, 1022, 1121, 1122, 1123, 1222, 1641) that include applications (522, 1642), wherein execution of the software code by processing circuitry (1660) of the host computing system configures the software integrity tool to perform operations corresponding to any of the methods of claims 1-27.

33. A computer program product (1695) comprising software code for a software integrity tool (530, 710, 910, 1010, 1110, 1210) of a host computing system (310, 510, 700, 900, 1000, 1100, 1200, 1600) configured with a runtime environment (330, 1620) arranged to execute containers (340, 520, 721, 722, 922, 1021, 1022, 1121, 1122, 1123, 1222, 1641) that include applications (522, 1642), wherein execution of the software code by processing circuitry (1660) of the host computing system configures the software integrity tool to perform operations corresponding to any of the methods of claims 1-27.

Images