Systems and methods for testing sandboxed in-kernel programs

Abstract

Systems and methods for testing sandboxed in-kernel programs are provided. A method of testing a program includes: obtaining a Berkeley Packet Filter (BPF) program to test; obtaining a test to run on the BPF program; performing the test on the BPF program for a plurality of Linux kernels and reporting a result of performing the test on the BPF program for a plurality of Linux kernels. This enables a generic testing solution for black box testing of BPF programs. The embodiments can integrate with existing testing suites and frameworks and can integrate into CI / CD pipelines. Some embodiments allow testing using any kernel from user space. Host kernel testing of BPF programs is enabled by a developer using the developer's own workstation and version of the Linux kernel. Guest kernel testing of BPF programs is enabled using specific kernel versions by providing the framework a specification of the target kernel environment.

Description

SYSTEMS AND METHODS FOR TESTING SANDBOXED IN-KERNEL PROGRAMSTechnical Field

[0001] The current disclosure relates generally to testing programs.Background

[0002] Berkeley Packet Filter (BPF)

[0003] BPF is a technology that runs in the Linux kernel without tweaking source code of the kernel or loading a module. Commonly, changing the kernel is done through two options, namely, native support or kernel modules. However, with BPF, a reprogramming of the kernel can be done without altering its source code or course of its running. BPF is a well-documented instrumenting and executing code technology inside the Linux kernel. BPF bytecode must pass strict validation to be executable inside the Linux kernel as required by the kernel verifier. BPF programs must have an end.

[0004] In the literature, some patents were proposed to tackle diverse use cases using BPF, for the sake of illustration, in (Gill, T. S., Gill, H. S., Arnoux, J., Nguyen, C. T. N., Soundararajan, S., Haolin, L. U., & Nguyen, A. T. N. (2018). Systems and methods for networked microservice modeling and visualization. U.S. Patent Application No. 15 / 963,079), BPF is considered a source to collect packets for the sake of microservice modeling and visualization. In (KONG, Seok Hwan et SAIKIA, Dipjyoti. Method for controlling of accelerating edge platform network and electronic device using the same. U.S. Patent Application No 17 / 383,331, 27 janv. 2022), authors proposed a method to build a device for controlling of accelerating edge platform network, it loads BPF user programs in BPF byte code to use in a data plane. In (DALY, Daniel, JAIN, Anjali Singhai, LI, Yadong, et al. Transport and cryptography offload to a network interface device. U.S. Patent Application No 17 / 544,699, 31 mars 2022), Address Family of the express Data Path (AF_XDP) Linux socket built at the top of BPF program is used for packets reception and attached to an interface driver without landing into the user space application. In (Anil Vasudevan, Grzegorz JERECZEK, Parthasarathy Sarangam. Management of port congestion, Intel Corp (2022). U.S. Patent Application No US17 / 838,703), a redirection manager is proposed to manage redirection queues on ports based on eBPF and XDP. In (Shaopeng He, Cunming LIANG, Haitao Kang, Hongjun NI, Jiang YuZiye Yang, Anjali Singhai Jain, Daniel Daly, Yadong Li, Ping Yu, BoCui, Jingjing WU, Liang Ma, Changpeng LIU. Service mesh offload to network devices. PCT / US2022 / 021795, 2021 US, 2022 WO), AF_XDP sockets may enable the possibility for XDP programs to redirect frames to a memory buffer accessible to a userspace application, where the service mesh is offloaded to network devices. In (Srinivas Akkipeddi, Narendranath Karjala Subramanyam, Sachchidanand Vaidya, Mahesh SIVAKUMAR, Pavan Kumar Kurapati, Philip M. Goddard, Sivakumar Ganapathy, Shailender Sharma, Kiran K N, Pranavadatta D N, Vinay K Nallamothu, Yuvaraja Mariappan, Ashutosh K. Grewal. Containerized router with virtual networking. US17 / 649,632, 2022), BPF and XDP are coined to support Data Plane Development Kit (DPDK) and virtual Router forwarding planes.Data message processing method and system, intelligent terminal and storage medium. CN202110370689.7A, 2021), BPF and XDP are proposed to be used optionally for data packet forwarding. InData processing method and device based on policy routing, equipment and storage medium.CN202211369071.XA, 2022), a routing strategy proposed through eBPF the strategy routing, which bypasses Netfi Iter and part of kernel protocol stack, to reduce the processing path of data packet. Based on patents, BPF / XDP technology stack is of interest mainly on introducing the advantage of using it within Kernel land to fulfill service and networking functions requirements.BPF itself specifies a bytecode that any compatible Linux kernel will JIT (Just In Time) compile to machine code. While such code may be written as is using tools like Linux's header files (the approach taken by tcpdump with cBPF), reasonable people intending to use BPF will do by compiling C code to eBPF (other programming languages are supported, but the concepts stay the same). In a standard development lifecycle for eBPF, a program is written and compiled into an object file, an intermediate ELF representation of the BPF code and metadata. The ELF object file is given to a libbpf tool to generate a skeleton boilerplate code to include a requested programming language. In the programming language, the skeleton is used to load the BPF code into the kernel. Once loaded, the code is attached to an event source for it to start execution. Testing BPF program is done by reusing the generated skeleton to craft a program that loads and attaches the BPF program and will then use some method togenerate events that the kernel will use to trigger our BPF program. Improved systems and methods for software testing are needed.Summary

[0005] Systems and methods for testing of sandboxed in-kernel programs are provided. In some embodiments, a method of testing a program includes: obtaining a Berkeley Packet Filter (BPF) program to test; obtaining a test to run on the BPF program; performing the test on the BPF program for a plurality of Linux kernels and reporting a result of performing the test on the BPF program for a plurality of Linux kernels. The plurality of Linux kernels includes a plurality of Linux kernel versions and / or a plurality of Linux kernel configuration options.

[0006] Some advantages of the embodiments include: Generic testing solution for black box testing of BPF programs. The embodiments can integrate with existing testing suites and frameworks like unittest or GoogleTest. Can integrate into CI / CD pipelines by, e.g., leveraging integration. Allows testing using any kernel from user space. Usable from Linux or Windows Subsystem for Linux using, e.g., testing using any kernel from user space. Host kernel testing of BPF programs by a developer using the developer's own workstation and version of the Linux kernel. Guest kernel testing of BPF programs using specific kernel versions by providing the framework a specification of the target kernel environment.

[0007] In some embodiments, performing the test on the BPF program comprises performing the test on the BPF program by running the plurality of Linux kernels in a corresponding plurality of virtualization environments.

[0008] In some embodiments, running the plurality of Linux kernels in the corresponding plurality of virtualization environments comprises: running each of the plurality of Linux kernels in a corresponding User Mode Linux (UML) process.

[0009] In some embodiments, running the plurality of Linux kernels in the corresponding plurality of virtualization environments comprises: running each of the plurality of Linux kernels in one of the group consisting of: Quick Emulator (QEMU); Kernel-based Virtual Machine (KVM); Linux Containers (LXC); Xen; and VMWare.

[0010] In some embodiments, the method also includes, prior to performing the test on the BPF program: compiling one or more of the plurality of Linux kernels with the corresponding Linux kernel versions and Linux kernel configurations.

[0011] In some embodiments, the method also includes, prior to performing the test on the BPF program: retrieving from a cache one or more of the plurality of Linux kernels with the corresponding Linux kernel versions and Linux kernel configurations.

[0012] In some embodiments, one or more of the Linux kernels retrieved from the cache comprises: checkpoint information to speed the boot process.

[0013] In some embodiments, the cache includes one or more of the group consisting of: a local cache for a quick boot of already booted kernels; a remote cache that already contains checkpoints for commonly used kernels; and a Content Delivery Network (CDN) cache to use which already contains checkpoints for the most used kernel versions and configuration options.

[0014] In some embodiments, the checkpoint information comprises: Checkpoint! ng / Restore In Userspace (CRIU) information.

[0015] In some embodiments, performing the test on the BPF program comprises: determining whether the BPF program passes BPF validation for the plurality of Linux kernels.

[0016] In some embodiments, performing the test on the BPF program comprises: determining whether the outputs of the BPF program corresponds to expected outputs for the plurality of Linux kernels.

[0017] In some embodiments, performing the test on the BPF program comprises: providing input context, and input data; and receiving output context, output data, return code, and / or execution time of the BPF program.

[0018] In some embodiments, performing the test on the BPF program comprises: testing the plurality of Linux kernels from user space.

[0019] In some embodiments, the method is integrated with an existing testing suite and / or framework.

[0020] In some embodiments, the existing testing suite and / or framework comprises: unittest or GoogleTest.

[0021] Some embodiments in the current disclosure differ from Cilium's approach in many ways. For example, some improvements over Cilium's approach include: allowing black box testing of BPF programs without needing to know about program internals, but solely on their interface; allowing testing on different versions of the Linux kernel by paravirtualization of hardware interfaces; and allowing testing on a kernel as a non-root user.

[0022] Some embodiments in the current disclosure allow BPF development to be integrated into tests, allowing a test suite to be run using any kernel version (that supports BPF) with any kernel compilation options. Some embodiments in the current disclosure are easily integrated into CI / CD pipelines and existing test frameworks and allows BPF programs to be treated like any other kind of source code in a modern development process.Brief Description of the Drawings

[0023] The accompanying drawing figures incorporated in and forming a part of this specification illustrate several aspects of the disclosure, and together with the description serve to explain the principles of the disclosure.

[0024] Figure 1 illustrates a method of testing a program, according to some embodiments of the present disclosure;

[0025] Figure 2 shows such a typical application structure;

[0026] Figure 3 illustrates a Framework for Host Testing in accordance with some embodiments of the current disclosure;

[0027] Figure 4 illustrates a proposed framework for guest testing, according to some embodiments of the current disclosure;

[0028] Figure 5 illustrates a sequence diagram explaining how a kernel compilation is carried out, according to some embodiments of the current disclosure;

[0029] Figure 6 illustrates an excerpt from an example / proc / config.gz;

[0030] Figure 7 illustrates how CRIU can be leveraged to fast boot a UML kernel if it was already booted once before, according to some embodiments of the current disclosure;

[0031] Figure 8 is a schematic block diagram of a compute node according to some embodiments of the present disclosure;

[0032] Figure 9 is a schematic block diagram that illustrates a virtualized embodiment of the compute of Figure 8 according to some embodiments of the present disclosure; and

[0033] Figure 10 is a schematic block diagram of the compute node of Figure 8 according to some other embodiments of the present disclosure.Detailed Description

[0034] The embodiments set forth below represent information to enable those skilled in the art to practice the embodiments and illustrate the best mode of practicing the embodiments. Upon reading the following description in light of the accompanying drawing figures, those skilled in the art will understand the concepts of the disclosure and will recognize applications of these concepts not particularly addressed herein. It should be understood that these concepts and applications fall within the scope of the disclosure.

[0035] As a recent technology that is presently seeing rapid adoption in a wide range of fields, BPF presents many opportunities for data collection, monitoring, networking, etc. BPF being code executed inside the Linux kernel presents a known problem, being that Linux internal data structures have no guaranteed ABI (Application Binary Interface). Depending on the kernel version and compilation options, field order, size, and presence is not at all guaranteed by the kernel, and rightfully so. BPF provides facilities for relocatable reads and writes to kernel data structures, but in the real world, development and production machines often run different versions of the kernel with differing kernel compilation options.

[0036] Moreover, dealing with BPF requires one to write a loader program that will be compiled to bytecode and then loaded into the kernel. Once loaded, the program must then be attached to an attach point, like a socket, a system call, a kprobe, etc., before it can start to execute. In literature, many such narrow-purpose programs are built to test BPF programs. While these loader programs may work, they are not a good method of testing according to software engineering principles due to the high number of variables present.

[0037] While working with BPF ensuring program security is done by the Kernel, but ensuring program correctness is hard, as it requires integration of the loader program in another application so that adequate inputs are generated, and even then, validation of outputs is hard and prone to interpretation errors.

[0038] Out of all the literature and tools currently available, Cilium already has a usable testing framework for BPF code. Their framework, however, only works for host testing using BPF_PROG_RUN, and for their use case. They do introduce interesting ideas, like mocks, but they require all their tests and the framework itself to be run in BPF. In contrast, some embodiments disclosed herein aim to keep the BPF code itself asa black box and test the BPF programs from the outside. Some aspects of the Cilium approach could however be incorporated into the embodiments described herein.

[0039] Systems and methods for testing sandboxed in-kernel programs are provided. Figure 1 illustrates a method of testing a program, according to some embodiments of the present disclosure. In some embodiments, the testing framework obtains (100) a BPF program to test. The testing framework obtains (102) a test to run on the BPF program. The testing framework performs (104) the test on the BPF program for a plurality of Linux kernels and reports (106) a result of performing the test on the BPF program for a plurality of Linux kernels. The plurality of Linux kernels includes a plurality of Linux kernel versions and / or a plurality of Linux kernel configuration options.

[0040] Some advantages of the embodiments include: Generic testing solution for black box testing of BPF programs. The embodiments can integrate with existing testing suites and frameworks like unittest or GoogleTest. Can integrate into CI / CD pipelines by, e.g., leveraging integration. Allows testing using any kernel from user space. Usable from Linux or Windows Subsystem for Linux using, e.g., testing using any kernel from user space. Host kernel testing of BPF programs by a developer using the developer's own workstation and version of the Linux kernel. Guest kernel testing of BPF programs using specific kernel versions by providing the framework a specification of the target kernel environment.

[0041] Some embodiments in the current disclosure differ from Cilium's approach in many ways. For example, some improvements over Cilium's approach include: allowing black box testing of BPF programs without needing to know about program internals, but solely on their interface; allowing testing on different versions of the Linux kernel by paravirtualization of hardware interfaces; and allowing testing on a kernel as a non-root user.

[0042] Some embodiments in the current disclosure allow BPF development to be integrated into tests, allowing a test suite to be run using any kernel version (that supports BPF) with any kernel compilation options. Some embodiments in the current disclosure are easily integrated into CI / CD pipelines and existing test frameworks and allows BPF programs to be treated like any other kind of source code in a modern development process.

[0043] Some embodiments in the current disclosure introduce a framework that allows developers to put BPF programs under test, like any other kind of source code.

[0044] The Linux kernel provides a BPF_PROG_TEST_RUN operation (kernel version 3.19) to do test runs of programs, including sending fake inputs and receiving the outputs of the program. Some embodiments in the current disclosure integrate this operation into the scope of a testing framework focused on BPF and its functionalities.

[0045] User Mode Linux (UML) is a kernel compilation mode that started as a separate project for version 2.2 and integrated in the kernel for version 2.6. Since Linux is built on drivers to allow interaction with hardware, UML leverages specialized drivers and kernel support to run the Linux kernel inside another Linux kernel as a regular process. It is a form of software virtualization often referred to as paravirtualization (see, Perens, B. (n.d.). User Mode Linux (world). Guide Books. https: / / doi.org / 10.5555 / 1137797).

[0046] The kernel which is connected to the hardware is called the host kernel, while the kernel running inside the host kernel is called the guest kernel. Since the guest kernel is run as a standard Linux process, it does not require special privileges on the host kernel while the guest kernel can be securely run as root in a safe manner for the host kernel.

[0047] Some embodiments in the current disclosure use UML as an alternate testing environment over the host kernel. Provided with the kernel version and compilation options, this allows anybody to test BPF programs in a UML kernel that is indistinguishable from the target kernel (apart from CPU architecture).

[0048] Checkpointing / Restore In Userspace (CRIU) started as a project by Virtuozzo, a cloud and virtualization software development company. It was later released for Linux on a GNU GPLv2 license. CRIU is now integrated in many cloud offerings like Kubernetes, Docker, Podman, etc.

[0049] CRIU serves as a way to quickly preserve the state of a process tree on Linux into a checkpoint on the file system. A checkpoint consists of all non-read-only states of a process tree, including the mapped memory, the registers, the open file descriptors, etc. A checkpoint also links to the read-only data by referencing the executable from which a process was started. With a checkpoint available, CRIU then allows the restoration of said checkpoint into the original process tree (see, Bozyigit, M., & Wasiq, M. (2001). User-level process checkpoint and restore for migration. ACM SIGOPS Operating Systems Review, 35(2), 86-96. https: / / doi.org / 10.1145 / 377069.377091).

[0050] At a high-level, CRIU is a way to serialize and deserialize processes to and from a file system. The main goal of CRIU is to improve boot time of containers by checkpointing a base container after boot and restoring that checkpoint for later use to improve boot time.

[0051] Some embodiments in the current disclosure use process checkpointing and restore. Slow tests can have a negative impact on software development. Since a UML kernel is just a Linux process, the system can checkpoint it after it finishes booting, and cache the checkpoint for future use. The next time tests are run with that specific version of the kernel, instead of rebooting the kernel from scratch, it can then restore the previously cached checkpoint to reduce the time it takes for tests to run.

[0052] Some advantages of the embodiments include: Generic testing solution for black box testing of BPF programs. Can integrate with existing testing suites and frameworks like unittest or GoogleTest. Can integrate into CI / CD pipelines by, e.g., leveraging integration. Allows testing using any kernel from user space. Usable from Linux or Windows Subsystem for Linux using, e.g., testing using any kernel from user space. Host kernel testing of BPF programs by a developer using the developer's own workstation and version of the Linux kernel. Guest kernel testing of BPF programs using specific kernel versions by providing the framework a specification of the target kernel environment.

[0053] In a typical BPF development pipeline, the BPF program is developed at the same time as another loader program. The loader program's job is to load the BPF program into the kernel and manage its maps. A map is a dictionary with which a BPF program can interact with the outside world.

[0054] Figure 2 shows such a typical application structure. While the loader program appears simple, it can be part of another application, like a routing control application. The routing control application would load the maps with route information, and a BPF program is loaded onto all network interfaces to route packets according to the route information.

[0055] In the current situation, testing a program requires creating a dedicated loader program that will load and attach the BPF program, and then generate some activity to trigger execution of the BPF program. Such loader programs are seldom reused, and hard to reuse.

[0056] In Figure 2, some operations are indicated over the arrows. In some embodiments, the loader program loads and attaches the maps. Then, the maps are read to and written in parallel by: the eBPF program and the loader program. As used herein, classical Berkeley Packet Filter (cBPF) refers to the original implementation of BPF used only in the network stack by tcpdump and al., now superseded by eBPF. In most instances, the kernel still accepts eBPF, but will translate it to eBPF on the fly. As used herein, BPF has the same meaning as eBPF, and its use is recommended over eBPF.

[0057] In some embodiments, the loader program is written with a tool like libbpf or a similar tool. Libbpf acts as a BPF program loader. Libbpf can load, check, and relocate BPF programs. Libbpf can also sort out maps and hooks. In some embodiments, this will take care of initializing the maps during the load operation.

[0058] In this scenario, testing the BPF program would require end-to-end testing of the whole application stack, including other devices connected to the network interfaces. Such a testing process can be error prone, and quite costly in both time and money to implement. Improved systems and methods for software testing are needed.

[0059] Figure 3 illustrates a Framework for Host Testing in accordance with some embodiments of the current disclosure. In Figure 3, the test framework takes the place of the loader program. Instead of attaching the BPF program to an attach point in the kernel, the test framework will instead use the BPF_PROG_TEST_RUN feature added to the 3.19 version of the kernel, or a similar mechanism. The test framework is able to provide input context, and input data. The test framework is also able to receive output context, output data, return code, and / or execution time of the BPF program. In some embodiments, the test framework is also in charge of initializing and / or validating the maps that the BPF program under test uses.

[0060] In Figure 3, the operations indicated over the arrows could happen in the following order. In step 200, the test framework initializes the maps. In step 202, the test framework loads the BPF program. In step 204, the test framework test runs the BPF program. In step 206, the eBPF program will read and write maps. In step 208, the test framework validates the results of the maps and the test run.

[0061] Figure 4 illustrates a proposed framework for guest testing, according to some embodiments of the current disclosure. Figure 4 shows an example of the test framework where the loader program can be reused inside a guest UML kernel, or othersuitable virtualization system. UML is a virtualization system based on an architectural port of a Linux kernel to its own system call interface. This enables multiple virtual Linux kernel-based operating systems to run as an application in a Linux system. The virtual operating systems are known as guests while the system is known as the host kernel.

[0062] The guest UML kernel can be configured to run any version of the Linux kernel with any configuration options. In practice, testing can be done on the host kernel, or as many UML kernels as required. In some embodiments, the steps described in relation to Figure 3 are also applicable to Figure 4.

[0063] In some embodiments, the testing framework exposes a C API to be used as part of a larger test suite in a programming language agnostic fashion since C is the lingua franca of programming languages and foreign function interface (FFI) support is ubiquitous. This allows the framework to be integrated into larger applications and other testing frameworks (i.e.: Python and unittest, C++ and GoogleTest, etc.).

[0064] As the testing framework aims to be usable from a variety of different programming languages and testing frameworks, the exposed C API allows the use of the testing framework in a traditional testing framework workflow: Setup(); Test(); TeardownO-

[0065] The embodiments herein can be implemented in many ways. Herein, libbpf is the de facto standard way of working with BPF on Linux since it is directly supported by kernel developers. For the examples described herein, the testing framework uses libbpf to load BPF programs compatible with libbpf for code relocation purposes and map management. Such integration enables the testing framework to be as close as possible to existing BPF development workflows.

[0066] In the context of BPF, any kernel built with the same version and compilation options are indistinguishable from each other independently of CPU architecture and Linux distribution. A program compiled on Ubuntu will work the same on a CentOS server with the same kernel with the same compilation options, and vice versa. This correlation also applies for UML kernels.

[0067] Building the kernel is a straightforward but slow process. Figure 5 illustrates a sequence diagram explaining how such compilation is carried out. Most Linux systems encode the compilation options used in a compressed text file called / proc / config.gz. Figure 6 illustrates an excerpt from an example / proc / config.gz. This configuration filecontains the compilation options for the currently running kernel that can be ingested by the Linux build tools at compile time.

[0068] In some embodiments, to enable reproducible testing of BPF programs, booting a UML kernel from scratch implies fetching the kernel source for that version and compiling it using the above-mentioned options. The / proc / config.gz is usually available for kernels compiled by most distributions, but there are other ways to get that information in case the configuration file is not available.

[0069] Booting any Linux kernel requires a valid filesystem initialized with programs, configurations, etc. following the Filesystem Hierarchy Standard. Normally, the step of creating and initializing the filesystem is done by the installer of a Linux distribution. However, in the embodiments described herein, the kernels are without distributions. In this situation, the rootfs distributed with container specifications can be leveraged to get a valid filesystem to work with the kernel without having to install a distribution. This allows for less time and resources per test. The UML rootfs can be obtained by creating a file at least as big as the container's rootfs, then formatting the file to ext4, or another appropriate format, mounting the file, extracting the container's rootfs to the mounted path, and finally unmounting the file. The resulting file will contain a fully working rootfs that can be mapped to UML as a host filesystem.

[0070] UML kernels under some distributions suffer from slow starts that can take a few minutes even on modern top-of-the-line hardware. While testing on the host kernel is fast, the slow boot time of a UML kernel might discourage some developers from testing on guest kernels, even in a continuous integration / continuous delivery (CI / CD) pipeline. To alleviate such a problem, a tool like Checkpoint / Restore In Userspace (CRIU) can be used to checkpoint the UML kernel process after boot in a cache. The next time the UML kernel is requested to boot, it can be done so from the checkpoint much more quickly (full boot once, checkpoint boot the rest of the time). This can save substantial time and computation resources during testing, especially when there are multiple kernels to test against.

[0071] Figure 7 illustrates how CRIU can be leveraged to fast boot a UML kernel if it was already booted once before. Unlike the quick bootFromCheckpoint operation, bootFromScratch implies a full boot of a Linux kernel and all its drivers. By checkpointing the UML kernel process after it has finished booting, and caching thatcheckpoint on the filesystem, it will allow a subsequent boot to happen from the checkpoint, which takes a drastically smaller amount of time.

[0072] Figure 7 also interfaces something referred to herein as a Checkpoint Cache. A Checkpoint Cache is a service that allows the storage and retrieval of checkpoints for specific kernel versions and configuration options. There are multiple ways to implement such a service, including: a local cache for a quick boot of already booted kernels; a remote cache for an organization that already contains checkpoints for commonly used kernels; and a Content Delivery Network (CDN) cache to use which already contains checkpoints for the most used kernel versions and configuration options.

[0073] The examples discussed herein use UML as it provides a quick software dependent based approach to virtualization. However, the embodiments disclosed herein are not restricted to using UML as a choice of virtualization technology. Any other suitable virtualization technology could be used.

[0074] UML presents a significant advantage compared to other virtualization technologies in the sense that it requires neither hardware support nor super-user (e.g., "root") access to the host machine. While UML has some limitations, like forced singlethreading and slightly slower execution speed, these limitations do not have an impact on embodiments disclosed herein due to the nature of BPF programs being singlethreaded and Just-in-time (JIT) compiled to machine code for execution directly inside the Linux kernel.

[0075] Use of Quick Emulator (QEMU) / Kernel-based Virtual Machine (KVM), Linux Containers (LXC), Xen, VMWare SX (and ESX, ESXi, etc.), etc. as a means of virtualization is possible using libvirt. libvirt is an open-source API and management tool for managing platform virtualization, libvirt can be used to manage and of these virtualization technologies. In some embodiments, process checkpoints will be replaced with virtual machine snapshots. Creating a bootable environment will require more work, as the approach for booting a UML kernel by creating a rootfs from a container's specification will not work. There is now a need to find and install a bootloader and install the kernel into the rootfs of the virtual machine.

[0076] VirtualBox is another means of virtualization that could also be used while suffering the same problems as libvirt. The API for VirtualBox is however more difficult to use considering it is based on the Component Object Model from Microsoft (see Cross Platform Component Object Model or XPCOM).

[0077] In some embodiments, the testing framework is available as Software as a Service (SaaS). In some embodiments, a version of the testing framework can implement essentially "BPF Testing as a Service". Some embodiments include the ability to test on different kernel versions with different compilation options as a cloud offering.

[0078] The development could be tested by companies using the host kernel tester, and as part of a CI / CD pipeline, they could push their programs and tests to the testing framework to run the tests on a matrix (e.g., a variety) of kernel versions and compilation options.

[0079] Since building a kernel takes some time, some embodiments provide a kernel that is already checkpointed for regular users. Some embodiments then gate the use of custom kernel testing for specific users of the service. In such a scenario, a regular user could test on a Long-Term Support (LTS) kernel but would need to unlock the ability to use a custom kernel that mimics their production environment (which might not even be on LTS).

[0080] Figure 8 is a schematic block diagram of a compute node 800 according to some embodiments of the present disclosure. Any of the methods or steps described herein could be performed by one or more of the compute nodes 800 described herein. Specifically, the steps related to the testing framework could be performed by one or more compute nodes 800. As illustrated, the compute node 800 includes a control system 802 that includes one or more processors 804 (e.g., Central Processing Units (CPUs), Application Specific Integrated Circuits (ASICs), Field Programmable Gate Arrays (FPGAs), and / or the like), memory 806, and a network interface 808. The one or more processors 804 are also referred to herein as processing circuitry. The one or more processors 804 operate to provide one or more functions of a compute node 800 as described herein. In some embodiments, the function(s) are implemented in software that is stored, e.g., in the memory 806 and executed by the one or more processors 804.

[0081] Figure 9 is a schematic block diagram that illustrates a virtualized embodiment of the compute node 800 according to some embodiments of the present disclosure. This discussion is equally applicable to other types of compute nodes. Further, other types of compute nodes may have similar virtualized architectures. Again, optional features are represented by dashed boxes.

[0082] As used herein, a "virtualized" compute node is an implementation of the compute node 800 in which at least a portion of the functionality of the compute node 800 is implemented as a virtual component(s) (e.g., via a virtual machine(s) executing on a physical processing node(s) in a network(s)). As illustrated, in this example, the compute node 800 may include the control system 802 as described above. The compute node 800 includes one or more processing nodes 900 coupled to or included as part of a networks) 902. If present, the control system 802 is connected to the processing node(s) 900 via the network 902. Each processing node 900 includes one or more processors 904 (e.g., CPUs, ASICs, FPGAs, and / or the like), memory 906, and a network interface 908.

[0083] In this example, functions 910 of the compute node 800 described herein are implemented at the one or more processing nodes 900 or distributed across the one or more processing nodes 900 and the control system 802 in any desired manner. In some particular embodiments, some or all of the functions 910 of the compute node 800 described herein are implemented as virtual components executed by one or more virtual machines implemented in a virtual environment(s) hosted by the processing node(s) 900. As will be appreciated by one of ordinary skill in the art, additional signaling or communication between the processing node(s) 900 and the control system 802 is used in order to carry out at least some of the desired functions 910.

[0084] In some embodiments, a computer program including instructions which, when executed by at least one processor, causes the at least one processor to carry out the functionality of compute node 800 or a node (e.g., a processing node 900) implementing one or more of the functions 910 of the compute node 800 in a virtual environment according to any of the embodiments described herein is provided. In some embodiments, a carrier comprising the aforementioned computer program product is provided. The carrier is one of an electronic signal, an optical signal, a radio signal, or a computer readable storage medium (e.g., a non-transitory computer readable medium such as memory).

[0085] Figure 10 is a schematic block diagram of the compute node 800 according to some other embodiments of the present disclosure. The compute node 800 includes one or more modules 1000, each of which is implemented in software. The module(s) 1000 provide the functionality of the compute node 800 described herein. This discussion is equally applicable to the processing node 900 of Figure 9 where themodules 1000 may be implemented at one of the processing nodes 900 or distributed across multiple processing nodes 900 and / or distributed across the processing node(s) 900 and the control system 802.

[0086] Any appropriate steps, methods, features, functions, or benefits disclosed herein may be performed through one or more functional units or modules of one or more virtual apparatuses. Each virtual apparatus may comprise a number of these functional units. These functional units may be implemented via processing circuitry, which may include one or more microprocessor or microcontrollers, as well as other digital hardware, which may include Digital Signal Processors (DSPs), special-purpose digital logic, and the like. The processing circuitry may be configured to execute program code stored in memory, which may include one or several types of memory such as Read Only Memory (ROM), Random Access Memory (RAM), cache memory, flash memory devices, optical storage devices, etc. Program code stored in memory includes program instructions for executing one or more telecommunications and / or data communications protocols as well as instructions for carrying out one or more of the techniques described herein. In some implementations, the processing circuitry may be used to cause the respective functional unit to perform corresponding functions according to one or more embodiments of the present disclosure.

[0087] While processes in the figures may show a particular order of operations performed by certain embodiments of the present disclosure, it should be understood that such order is exemplary (e.g., alternative embodiments may perform the operations in a different order, combine certain operations, overlap certain operations, etc.).

[0088] At least some of the following abbreviations may be used in this disclosure. If there is an inconsistency between abbreviations, preference should be given to how it is used above. If listed multiple times below, the first listing should be preferred over any subsequent listing(s).• 3GPP Third Generation Partnership Project• 5G Fifth Generation• 5GC Fifth Generation Core• 5GS Fifth Generation System• ABI Application Binary Interface• AF Application FunctionAMF Access and Mobility FunctionAN Access NetworkAP Access PointAPI Application Programming InterfaceASIC Application Specific Integrated CircuitAUSF Authentication Server FunctionBPF Berkeley Packet Filter cBPF classical Berkeley Packet FilterCDN Content Delivery NetworkCPU Central Processing UnitCRIU Checkpoint! ng / Restore In UserspaceDCI Downlink Control InformationDN Data NetworkDSP Digital Signal Processor eBPF extended Berkeley Packet Filter eNB Enhanced or Evolved Node BEPS Evolved Packet SystemE-UTRA Evolved Universal Terrestrial Radio AccessFFI Foreign Function InterfaceFPGA Field Programmable Gate Array gNB New Radio Base Station gNB-DU New Radio Base Station Distributed UnitHSS Home Subscriber Server loT Internet of ThingsIP Internet ProtocolKVM Kernel-based Virtual MachineLTE Long Term EvolutionLXC Linux ContainerMAC Medium Access ControlMME Mobility Management EntityMTC Machine Type CommunicationNEF Network Exposure FunctionNF Network Function• NR New Radio• NRF Network Function Repository Function• NSSF Network Slice Selection Function• OTT Over-the-Top• PC Personal Computer• PCF Policy Control Function• PDSCH Physical Downlink Shared Channel• P-GW Packet Data Network Gateway• PRS Positioning Reference Signal• QEMU Quick Emulator• QoS Quality of Service• RAM Random Access Memory• RAN Radio Access Network• ROM Read Only Memory• RP Reception Point• RRH Remote Radio Head• RTT Round Trip Time• SaaS Software as a Service• SCEF Service Capability Exposure Function• SMF Session Management Function• TCI Transmission Configuration Indicator• TP Transmission Point• TRP Transmission / Reception Point• UDM Unified Data Management• UE User Equipment• UML User Mode Linux• UPF User Plane Function

[0089] Those skilled in the art will recognize improvements and modifications to the embodiments of the present disclosure. All such improvements and modifications are considered within the scope of the concepts disclosed herein.

Claims

Claims1. A method of testing a program comprising: obtaining (100) a Berkeley Packet Filter, BPF, program to test; obtaining (102) a test to run on the BPF program; performing (104) the test on the BPF program for a plurality of Linux kernels, where the plurality of Linux kernels comprises one or more of: a plurality of Linux kernel versions; and a plurality of Linux kernel configuration options; and reporting (106) a result of performing the test on the BPF program for a plurality of Linux kernels.

2. The method of claim 1 wherein performing the test on the BPF program comprises performing the test on the BPF program by running the plurality of Linux kernels in a corresponding plurality of virtualization environments.

3. The method of claim 2 wherein running the plurality of Linux kernels in the corresponding plurality of virtualization environments comprises: running each of the plurality of Linux kernels in a corresponding User Mode Linux, UML, process.

4. The method of claim 2 wherein running the plurality of Linux kernels in the corresponding plurality of virtualization environments comprises: running each of the plurality of Linux kernels in one of the group consisting of: Quick Emulator, QEMU; Kernel-based Virtual Machine, KVM; Linux Containers, LXC; Xen; and VMWare.

5. The method of any of claims 1-4 further comprising, prior to performing the test on the BPF program: compiling one or more of the plurality of Linux kernels with the corresponding Linux kernel versions and Linux kernel configurations.

6. The method of any of claims 1-5 further comprising, prior to performing the test on the BPF program: retrieving from a cache one or more of the plurality of Linux kernels with the corresponding Linux kernel versions and Linux kernel configurations.

7. The method of claim 6 wherein one or more of the Linux kernels retrieved from the cache comprises: checkpoint information to speed the boot process.

8. The method of any of claims 6-7 wherein the cache comprises one or more of the group consisting of: a local cache for a quick boot of already booted kernels; a remote cache that already contains checkpoints for commonly used kernels; and a Content Delivery Network, CDN, cache to use which already contains checkpoints for the most used kernel versions and configuration options.

9. The method of any of claims 7-8 wherein the checkpoint information comprises: Checkpoint! ng / Restore In Userspace, CRIU, information.

10. The method of any of claims 1-9 wherein performing the test on the BPF program comprises: determining whether the BPF program passes BPF validation for the plurality of Linux kernels.

11. The method of any of claims 1-9 wherein performing the test on the BPF program comprises: determining whether the outputs of the BPF program corresponds to expected outputs for the plurality of Linux kernels.

12. The method of any of claims 1-11 wherein performing the test on the BPF program comprises: providing input context, and input data; and receiving output context, output data, return code, and / or execution time of the BPF program.

13. The method of any of claims 1-12 wherein performing the test on the BPF program comprises: testing the plurality of Linux kernels from user space.

14. The method of any of claims 1-13 wherein the method is integrated with an existing testing suite and / or framework.

15. The method of claim 14 wherein the existing testing suite and / or framework comprises: unittest or GoogleTest.

16. A compute node (800) comprising one or more processors (804) and memory (806), the memory (806) comprising instructions to cause the compute node (800) to: obtain a Berkeley Packet Filter, BPF, program to test; obtain a test to run on the BPF program; perform the test on the BPF program for a plurality of Linux kernels, where the plurality of Linux kernels comprises one or more of: a plurality of Linux kernel versions; and a plurality of Linux kernel configuration options; and report a result of performing the test on the BPF program for a plurality of Linux kernels.

17. The compute node (800) of claim 16 wherein performing the test on the BPF program comprises performing the test on the BPF program by running the plurality of Linux kernels in a corresponding plurality of virtualization environments.

18. The compute node (800) of claim 17 wherein running the plurality of Linux kernels in the corresponding plurality of virtualization environments comprises: running each of the plurality of Linux kernels in a corresponding User Mode Linux, UML, process.

19. The compute node (800) of claim 17 wherein running the plurality of Linux kernels in the corresponding plurality of virtualization environments comprises: running each of the plurality of Linux kernels in one of the group consisting of: Quick Emulator, QEMU; Kernel-based Virtual Machine, KVM; Linux Containers, LXC; Xen; and VMWare.

20. The compute node (800) of any of claims 16-19 further comprising, prior to performing the test on the BPF program: compiling one or more of the plurality of Linux kernels with the corresponding Linux kernel versions and Linux kernel configurations.

21. The compute node (800) of any of claims 16-21 further comprising, prior to performing the test on the BPF program: retrieving from a cache one or more of the plurality of Linux kernels with the corresponding Linux kernel versions and Linux kernel configurations.

22. The compute node (800) of claim 21 wherein one or more of the Linux kernels retrieved from the cache comprises: checkpoint information to speed the boot process.

23. The compute node (800) of any of claims 21-22 wherein the cache comprises one or more of the group consisting of: a local cache for a quick boot of already booted kernels; a remote cache that already contains checkpoints for commonly used kernels; and a Content Delivery Network, CDN, cache to use which already contains checkpoints for the most used kernel versions and configuration options.

24. The compute node (800) of any of claims 22-23 wherein the checkpoint information comprises: Checkpoint! ng / Restore In Userspace, CRIU, information.

25. The compute node (800) of any of claims 16-24 wherein performing the test on the BPF program comprises: determining whether the BPF program passes BPF validation for the plurality of Linux kernels.

26. The compute node (800) of any of claims 16-24 wherein performing the test on the BPF program comprises: determining whether the outputs of the BPF program corresponds to expected outputs for the plurality of Linux kernels.

27. The compute node (800) of any of claims 16-26 wherein performing the test on the BPF program comprises: providing input context, and input data; and receiving output context, output data, return code, and / or execution time of the BPF program.

28. The compute node (800) of any of claims 16-27 wherein performing the test on the BPF program comprises: testing the plurality of Linux kernels from user space.

29. The compute node (800) of any of claims 16-28 wherein the method is integrated with an existing testing suite and / or framework.

30. The compute node (800) of claim 29 wherein the existing testing suite and / or framework comprises: unittest or GoogleTest.

31. A computer-readable medium comprising instructions which, when executed on at least one processor (804), cause the at least one processor (804) to carry out the method according to any one of claims 1-15.

Images