System and method for testing sandboxed kernel-in-process

Abstract

Systems and methods for testing sandboxed kernel-based programs are provided. One method for testing a program includes: acquiring a Berkeley Packet Filter (BPF) program to be tested; acquiring tests to run on the BPF program; executing tests on the BPF program against multiple Linux kernels; and reporting the results of executing tests on the BPF program against multiple Linux kernels. This provides a general testing scheme for black-box testing of BPF programs. These embodiments can be integrated with existing test suites and frameworks and can be integrated into CI / CD pipelines. Some embodiments allow testing from user space using any kernel. Host kernel testing of BPF programs can be implemented by developers using their own workstations and Linux kernel versions. Guest kernel testing of BPF programs can be implemented using specific kernel versions by providing the framework with specifications of the target kernel environment.

Description

Technical Field

[0001] This disclosure generally pertains to program testing. Background Technology

[0002] Berkeley Packet Filter (BPF)

[0003] BPF is a technology that runs within the Linux kernel without requiring modifications to the kernel source code or loading modules. Typically, kernel modifications are made in two ways: native support or kernel modules. However, with BPF, the kernel can be reprogrammed without altering its source code or its execution. BPF is a well-documented technology for code instrumentation and execution within the Linux kernel. BPF bytecode must undergo rigorous verification by the kernel verifier before it can be executed within the Linux kernel. BPF programs must be able to terminate.

[0004] Several patents have been proposed in the literature to utilize BPF to address various use cases. For example, in (Gill, TS, Gill, HS, Arnoux, J., Nguyen, CTN, Soundararajan, S., Haolin, LU, & Nguyen, ATN (2018), Systems and methods for networked microservice modeling and visualization, U.S. Patent Application No. 15 / 963,079), BPF is presented as a packet capture source for microservice modeling and visualization. In (KONG, Seok Hwan and SAIKIA, Dipjyoti, Method for controlling of accelerating edge platform network and electronic device using the same, U.S. Patent Application No. 17 / 383,331, January 27, 2022), the authors propose a method for constructing a device for controlling an accelerated edge platform network by loading a BPF user program in the form of BPF bytecode into the data plane for use. In (DALY, Daniel, JAIN, Anjali Singhai, LI, Yadong, et al., Transport and cryptography offload to a network interface device, U.S. Patent Application No. 17 / 544,699, March 31, 2022), a Fast Data Path Address Family (AF_XDP) Linux socket built on top of a BPF program is used for packet reception and attached to the interface driver without entering the user-space application. In (Anil Vasudevan, Grzegorz JERECZEK, Parthasarathy Sarangam, Management of port congestion, Intel Corp. (2022), U.S. Patent Application No. US17 / 838,703), a redirection manager is proposed for managing redirection queues on eBPF and XDP-based management ports.In (Shaopeng He, Cunming LIANG, Haitao Kang, Hongjun NI, Jiang YuZiye Yang, Anjali SinghaiJain, Daniel Daly, Yadong Li, Ping Yu, Bo Cui, Jingjing WU, Liang Ma, Changpeng LIU, Service mesh offload to network devices, PCT / US2022 / 021795, 2021 US, 2022 WO), the AF_XDP socket enables XDP programs to redirect frames to a memory buffer accessible to user-space applications, thereby offloading the service mesh to network devices. In (Srinivas Akkipeddi, Narendranath Karjala Subramanyam, Sachchidanand Vaidya, Mahesh SIVAKUMAR, Pavan Kumar Kurapati, Philip M. Goddard, Sivakumar Ganapathy, ShailenderSharma, Kiran KN, Pranavadatta DN, Vinay K Nallamothu, Yuvaraja Mariappan, Ashutosh K. Grewal, Containerized router with virtual networking, US17 / 649,632, 2022), BPF and XDP are used to support the Data Plane Development Kit (DPDK) and the virtual router forwarding plane. In (Ma Zengxie, Zhou Jianer, Tu Weijian, Li Qing, A data packet processing method, system, smart terminal and storage medium, CN202110370689.7A, 2021), it is proposed to optionally use BPF and XDP for packet forwarding. In (Wang Chuanguo, Cui Shiwei, Xu Xin, Han Chunchao, Xu Guozhen, Wu Baoxi, Data Processing Method, Apparatus, Device and Storage Medium Based on Policy Routing, CN202211369071.XA, 2022), a policy routing method implemented through eBPF was proposed. This policy routing bypasses Netfilter and part of the kernel protocol stack to shorten the data packet processing path. Based on these patents, the BPF / XDP technology stack has attracted attention primarily for its advantages in using the kernel space to meet service and network function requirements.

[0005] BPF itself defines a bytecode that can be compiled into machine code by any compatible Linux kernel via JIT (Just-In-Time) compilation. While this code can be written directly using tools like Linux header files (the method used by tcpdump for cBPF), developers who genuinely intend to use BPF will compile C code into eBPF (other programming languages ​​are also supported, but the concept is the same). In the standard development lifecycle of eBPF, a program is written and compiled into an object file, which is an intermediate ELF representation of the BPF code and metadata. The ELF object file is provided to the libbpf tool to generate skeleton boilerplate code containing the requested programming language. In that programming language, the skeleton is used to load the BPF code into the kernel. Once loaded, the code is attached to an event source to begin execution. Testing of BPF programs is accomplished by reusing the generated skeleton to build a program that loads and attaches to the BPF program, then uses some method to generate an event that the kernel uses to trigger the BPF program. Improvements are needed in software testing systems and methods. Summary of the Invention

[0006] Systems and methods for testing sandboxed kernel-based programs are provided. In some embodiments, a method for testing a program includes: acquiring a Berkeley Package Filter (BPF) program to be tested; acquiring tests to run on the BPF program; performing tests on the BPF program against multiple Linux kernels; and reporting the results of performing tests on the BPF program against multiple Linux kernels. The multiple Linux kernels include multiple Linux kernel versions and / or multiple Linux kernel configuration options.

[0007] Some advantages of this implementation include: a general testing scheme for black-box testing of BPF programs; integration with existing testing suites and frameworks such as unittest or GoogleTest; integration into CI / CD pipelines, for example; allowing testing from user space using any kernel; availability from Linux or Windows Subsystem for Linux, for example, using any kernel from user space; host kernel testing of BPF programs using the developer's own workstation and Linux kernel version; and guest kernel testing of BPF programs using a specific kernel version by providing the framework with specifications of the target kernel environment.

[0008] In some embodiments, performing tests on a BPF program includes running multiple Linux kernels in corresponding virtualization environments to perform tests on the BPF program.

[0009] In some embodiments, running multiple Linux kernels in corresponding virtualization environments includes running each of the multiple Linux kernels in a corresponding user-mode Linux (UML) process.

[0010] In some embodiments, running multiple Linux kernels in corresponding virtualization environments includes running each of the multiple Linux kernels in one of the following groups: Quick Emulator (QEMU), Kernel-based Virtual Machine (KVM), Linux Container (LXC), Xen, and VMware.

[0011] In some embodiments, the method further includes: compiling one or more of a plurality of Linux kernels using the corresponding Linux kernel version and Linux kernel configuration before performing tests on the BPF program.

[0012] In some embodiments, the method further includes: retrieving one or more Linux kernels with corresponding Linux kernel versions and Linux kernel configurations from a cache of multiple Linux kernels before performing tests on the BPF program.

[0013] In some embodiments, one or more Linux kernels retrieved from the cache include checkpoint information for speeding up the boot process.

[0014] In some embodiments, the cache includes one or more of the following: a local cache for quickly starting a kernel that has already been started; a remote cache that contains checkpoints for commonly used kernels; and a content delivery network (CDN) cache for use that contains checkpoints for the most commonly used kernel versions and configuration options.

[0015] In some embodiments, checkpoint information includes user space checkpoint / recovery (CRIU) information.

[0016] In some embodiments, performing tests on a BPF program includes determining whether the BPF program passes BPF verification against multiple Linux kernels.

[0017] In some embodiments, performing tests on a BPF program includes determining whether the output of the BPF program corresponds to the expected output for multiple Linux kernels.

[0018] In some embodiments, performing a test on a BPF program includes: providing an input context and input data; and receiving the output context, output data, return code, and / or execution time of the BPF program.

[0019] In some embodiments, performing tests on a BPF program includes testing multiple Linux kernels from user space.

[0020] In some embodiments, the method is integrated with existing test suites and / or frameworks.

[0021] In some embodiments, existing test suites and / or frameworks include unittest or GoogleTest.

[0022] Some embodiments of this disclosure differ from the Cilium method in many respects. For example, some improvements over the Cilium method include: allowing black-box testing of BPF programs without knowing the program's internal structure, based solely on its interface; allowing testing on different versions of the Linux kernel through semi-virtualization of the hardware interface; and allowing non-root users to perform testing on the kernel.

[0023] Some embodiments of this disclosure allow BPF development to be integrated into tests, thereby allowing test suites to be run using any (BPF-enabled) kernel version and any kernel compilation options. Some embodiments of this disclosure are easily integrated into CI / CD pipelines and existing testing frameworks, and allow BPF programs to be treated as any other type of source code in modern development processes. Attached Figure Description

[0024] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate several aspects of this disclosure and, together with the specification, serve to explain the principles of this disclosure.

[0025] Figure 1 Methods of a test procedure according to some embodiments of this disclosure are shown;

[0026] Figure 2 This illustrates a typical application structure;

[0027] Figure 3 A host testing framework according to some embodiments of this disclosure is shown;

[0028] Figure 4 A client testing framework proposed according to some embodiments of this disclosure is illustrated;

[0029] Figure 5 A sequence diagram illustrating how kernel compilation is performed according to some embodiments of this disclosure is shown;

[0030] Figure 6 An excerpt from the example / proc / config.gz is shown;

[0031] Figure 7 This illustrates how, according to some embodiments of the present disclosure, the UML kernel can be quickly started using CRIU when it has been previously started once;

[0032] Figure 8 This is a schematic block diagram of a computing node according to some embodiments of the present disclosure;

[0033] Figure 9 This illustrates some embodiments according to the present disclosure. Figure 8 A schematic block diagram of a virtualization embodiment of the computing node shown; and

[0034] Figure 10 Other embodiments according to this disclosure Figure 8 The diagram shows a schematic block diagram of the computation nodes. Detailed Implementation

[0035] The embodiments described below provide information enabling those skilled in the art to implement these embodiments and illustrate the best mode of implementation. Those skilled in the art will understand the concepts of this disclosure upon reading the following description in conjunction with the accompanying drawings, and will recognize the applications of these concepts not specifically described herein. It should be understood that these concepts and applications fall within the scope of this disclosure.

[0036] As a rapidly evolving technology in numerous fields, BPF offers many opportunities for data acquisition, monitoring, and networking. However, as code executing within the Linux kernel, BPF has a known limitation: Linux internal data structures do not have a guaranteed ABI (Application Binary Interface). Depending on the kernel version and compilation options, the kernel does not guarantee the order, size, or existence of fields, and this is generally true. While BPF provides a mechanism for relocatable reading and writing of kernel data structures, in real-world environments, development and production machines typically run different kernel versions and use different kernel compilation options.

[0037] Furthermore, handling BPF requires writing a loader program, which is compiled into bytecode and then loaded into the kernel. Once loaded, the program must be attached to a point (such as a socket, system call, kprobe, etc.) before execution can begin. Many such narrowly defined programs have been built in the literature to test BPF programs. While these loader programs may be effective, they are not good testing methods according to software engineering principles due to the large number of variables involved.

[0038] When using BPF, program security is guaranteed by the kernel, but program correctness is difficult to guarantee because the loader program needs to be integrated into another application to generate sufficient input; even so, verifying the output is difficult and prone to interpretation errors.

[0039] Of all the literature and tools currently available, Cilium already has a usable BPF code testing framework. However, this framework is only applicable to host testing using BPF_PROG_RUN and its own use cases. While the framework does introduce meaningful concepts such as mock objects, it requires all tests, as well as the framework itself, to run within BPF. In contrast, some embodiments disclosed herein aim to treat the BPF code itself as a black box and test BPF programs from the outside. However, certain aspects of the Cilium approach can also be incorporated into the embodiments described herein.

[0040] Systems and methods for testing sandboxed kernel programs are provided. Figure 1 A method for a testing procedure according to some embodiments of the present disclosure is illustrated. In some embodiments, the testing framework acquires (100) a BPF program to be tested. The testing framework acquires (102) tests to be run on the BPF program. The testing framework performs (104) tests on the BPF program against multiple Linux kernels and reports (106) the results of the tests performed on the BPF program against multiple Linux kernels. The multiple Linux kernels include multiple Linux kernel versions and / or multiple Linux kernel configuration options.

[0041] Some advantages of this embodiment include: a general testing scheme for black-box testing of BPF programs; integration with existing testing suites and frameworks such as unittest or GoogleTest; integration into CI / CD pipelines, for example; allowing testing from user space using any kernel; use from Linux or Windows Subsystem for Linux, for example, testing from user space using any kernel; host kernel testing of BPF programs using the developer's own workstation and Linux kernel version; and guest kernel testing of BPF programs using a specific kernel version by providing the framework with specifications of the target kernel environment.

[0042] Some embodiments of this disclosure differ from the Cilium method in many respects. For example, some improvements over the Cilium method include: allowing black-box testing of BPF programs without knowing the program's internal structure, based solely on its interface; allowing testing on different versions of the Linux kernel through semi-virtualization of the hardware interface; and allowing non-root users to perform testing on the kernel.

[0043] Some embodiments of this disclosure allow BPF development to be integrated into tests, thereby allowing test suites to be run using any (BPF-enabled) kernel version and any kernel compilation options. Some embodiments of this disclosure are easily integrated into CI / CD pipelines and existing testing frameworks, and allow BPF programs to be treated as any other type of source code in modern development processes.

[0044] Some embodiments of this disclosure introduce a framework that enables developers to test BPF programs as they would any other type of source code.

[0045] The Linux kernel provides the BPF_PROG_TEST_RUN operation (kernel version 3.19) for testing programs, including sending pseudo-inputs and receiving program outputs. Some embodiments of this disclosure integrate this operation into a testing framework focused on BPF and its functionality.

[0046] User Mode Linux (UML) is a kernel compilation mode that started as a standalone project in version 2.2 and was integrated into the kernel in version 2.6. Because Linux is built based on drivers for interacting with hardware, UML utilizes dedicated drivers and kernel support to run the Linux kernel as a regular process within another Linux kernel. This is a form of software virtualization, often referred to as paravirtualization (see Perens, B. (nd). User Mode Linux(world). Guide Books. https: / / doi.org / 10.5555 / 1137797).

[0047] The kernel connected to the hardware is called the host kernel, while the kernel running inside the host kernel is called the guest kernel. Because the guest kernel runs as a standard Linux process, it does not require special privileges on the host kernel, and it can run securely as root in a manner that is secure to the host kernel.

[0048] Some embodiments of this disclosure use UML as an alternative testing environment for the host kernel. Given a kernel version and compilation options, anyone can test BPF programs in a UML kernel (except for the CPU architecture) that is indistinguishable from the target kernel.

[0049] User Space Checkpoint / Restoration (CRIU) is a project initiated by Virtuozzo, a cloud and virtualization software development company. It was later released under the GNU GPLv2 license for Linux. CRIU is now integrated into many cloud services, such as Kubernetes, Docker, and Podman.

[0050] CRIU provides a way to quickly save the process tree state on Linux as a checkpoint on the file system. A checkpoint includes all non-read-only states of the process tree, including mapped memory, registers, open file descriptors, etc. The checkpoint also links read-only data by referencing the executable file that launched the process. When a checkpoint is available, CRIU allows the checkpoint to be restored to the original process tree (see Bozyigit, M., & Wasiq, M. (2001), User-level process checkpoint and restore for migration, ACM SIGOPS Operating Systems Review, 35(2), 86–96, https: / / doi.org / 10.1145 / 377069.377091).

[0051] At a high level, CRIU is a method for serializing processes to and deserializing processes from a file system. The main goal of CRIU is to improve startup time by setting checkpoints for the base container after startup and restoring those checkpoints for later use, thereby reducing container startup time.

[0052] Some embodiments of this disclosure use process checkpointing setup and recovery. Slow testing can negatively impact software development. Since the UML kernel is simply a Linux process, the system can set checkpoints for it after it has finished booting and cache these checkpoints for future use. The next time tests are run using that particular version of the kernel, the previously cached checkpoints can be restored without having to reboot the kernel from scratch, thus reducing the time required to run tests.

[0053] Some advantages of this implementation include: a general testing scheme for black-box testing of BPF programs; integration with existing testing suites and frameworks such as unittest or GoogleTest; integration into CI / CD pipelines, for example; allowing testing from user space using any kernel; availability from Linux or Windows Subsystem for Linux, for example, testing from user space using any kernel; host kernel testing of BPF programs using the developer's own workstation and Linux kernel version; and guest kernel testing of BPF programs using a specific kernel version by providing the framework with specifications of the target kernel environment.

[0054] In a typical BPF development pipeline, the BPF program is developed concurrently with another loader program. The loader program's task is to load the BPF program into the kernel and manage its mappings. Mappings are dictionaries that the BPF program can use to interact with the outside world.

[0055] Figure 2 This illustrates a typical application structure. While the loader program appears simple, it can be part of another application, such as a routing control application. The routing control application loads routing information into the mapping, and the BPF program is loaded onto all network interfaces to route packets based on the routing information.

[0056] In the current situation, the test program needs to create a dedicated loader program, which loads and attaches the BPF program, and then generates some activities to trigger the execution of the BPF program. Such loader programs are rarely reused and are difficult to reuse.

[0057] exist Figure 2 In the diagram, the arrows indicate certain operations. In some embodiments, the loader program loads and attaches the mapping. Then, the eBPF program and the loader program read and write the mapping in parallel. The Classic Berkeley Packet Filter (cBPF) used in this article refers to the original implementation of BPF, which was only used in the network stack by programs like tcpdump and has now been superseded by eBPF. In most cases, the kernel still accepts cBPF but will convert it to eBPF on demand. The BPF used in this article has the same meaning as eBPF, and it is recommended to use BPF instead of eBPF.

[0058] In some embodiments, the loader program is written using libbpf or a similar tool. libbpf acts as a BPF program loader. libbpf can load, inspect, and relocate BPF programs. libbpf can also organize mappings and hooks. In some embodiments, this is responsible for initializing mappings during the loading operation.

[0059] In this scenario, testing BPF programs requires end-to-end testing of the entire application stack, including other devices connected to the network interface. This testing process is error-prone and costly in terms of both time and money. Improved software testing systems and methods are needed.

[0060] Figure 3 A host testing framework according to some embodiments of this disclosure is illustrated. Figure 3In this framework, the test framework replaces the loader program. Instead of attaching the BPF program to an attachment point in the kernel, the test framework uses the BPF_PROG_TEST_RUN feature added in kernel version 3.19 or a similar mechanism. The test framework can provide input context and input data. It can also receive the BPF program's output context, output data, return code, and / or execution time. In some embodiments, the test framework is also responsible for initializing and / or verifying the mappings used by the BPF program under test.

[0061] exist Figure 3 In the diagram, the operations indicated by the arrows can occur in the following order: In step 200, the test framework initializes the mapping. In step 202, the test framework loads the BPF program. In step 204, the test framework runs the BPF program. In step 206, the eBPF program reads and writes the mapping. In step 208, the test framework verifies the mapping and the results of the trial run.

[0062] Figure 4 A client testing framework proposed according to some embodiments of this disclosure is shown. Figure 4 An example of a test framework is shown, in which the loader program can be reused in a guest UML kernel or other suitable virtualization system. UML is a virtualization system based on a system call interface that ports the Linux kernel architecture to itself. This allows multiple operating systems based on virtual Linux kernels to run as applications on the Linux system. The virtual operating system is called the guest, and the system is called the host kernel.

[0063] The guest UML kernel can be configured to run any version of the Linux kernel and any configuration options. In practice, testing can be performed on the host kernel or on a desired number of UML kernels. In some embodiments, combined with Figure 3 The steps described also apply to Figure 4.

[0064] In some embodiments, the testing framework exposes a C API for use as part of a larger test suite in a programming language-agnostic manner, because C is a universal language for programming and external function interface (FFI) support is ubiquitous. This allows the framework to be integrated into larger applications and other testing frameworks (i.e., Python and unittest, C++ and GoogleTest, etc.).

[0065] Because the testing framework is designed to be used by a variety of different programming languages ​​and testing frameworks, the exposed CAPI allows its use within traditional testing framework workflows: Setup(); Test(); Teardown().

[0066] The embodiments described in this article can be implemented in several ways. In this article, libbpf is the de facto standard way to use BPF on Linux because it receives direct support from kernel developers. In the examples described in this article, the test framework uses libbpf to load a libbpf-compatible BPF program for code relocation and mapping management. This integration allows the test framework to closely approximate existing BPF development workflows.

[0067] In the BPF context, any kernel built with the same version and compilation options is indistinguishable from one another, regardless of CPU architecture or Linux distribution. A program compiled on Ubuntu will work in the same way on a CentOS server with the same kernel and compilation options, and vice versa. This relevance also applies to UML kernels.

[0068] Building the kernel is a simple but slow process. Figure 5 The diagram illustrates the sequence of steps involved in performing this compilation. Most Linux systems encode the compilation options used in a compressed text file named / proc / config.gz. Figure 6 The image shows an excerpt from a sample ` / proc / config.gz` file. This configuration file contains the compilation options used by the currently running kernel, which Linux build tools can read during compilation.

[0069] In some embodiments, to enable reproducible testing of BPF programs, booting the UML kernel from scratch means obtaining the kernel source code for that version and compiling it using the options described above. For most distributions, the ` / proc / config.gz` file is typically available, but if this configuration file is unavailable, other methods can be used to obtain this information.

[0070] Booting any Linux kernel requires a valid file system that conforms to file system hierarchy standards and is initialized using programs and configurations. Typically, the steps of creating and initializing the file system are performed by the Linux distribution's installer. However, in the embodiment described herein, the kernel is not bundled with a distribution. In this case, a valid file system for use with the kernel can be obtained using the root file system (rootfs) distributed with the container specification, without requiring the distribution to be installed. This reduces the time and resources required for each test. The UML root file system can be obtained by creating a file at least as large as the container's root file system, formatting that file to ext4 or another suitable format, mounting that file, unpacking the container's root file system to the mount path, and finally unmounting the file. The resulting file will contain a fully functional root file system that can be mapped to UML as the host file system.

[0071] Some distributions suffer from slow startup times for their UML kernels, which can take several minutes even on modern, high-end hardware. While testing on the host kernel is faster, the slow startup time of the UML kernel can deter some developers from testing on the guest kernel, even in continuous integration / continuous delivery (CI / CD) pipelines. To mitigate this issue, tools like User-Space Checkpoint / Resume (CRIU) can be used to cache checkpoints for the UML kernel process after startup. The next time the UML kernel is requested to start, it can be started from the checkpoint more quickly (a full startup on the first attempt, followed by checkpoint startups). This can save significant time and computational resources during testing, especially when testing across multiple kernels.

[0072] Figure 7 This demonstrates how to quickly boot a UML kernel using CRIU when the kernel has already been booted once. Unlike the fast "bootFromCheckpoint" operation, "bootFromScratch" means a complete boot of the Linux kernel and all its drivers. By setting a checkpoint for the UML kernel process after it has finished booting and caching that checkpoint in the file system, subsequent boots can start from that checkpoint, significantly reducing the required time.

[0073] Figure 7 Also shown is what this article refers to as a checkpoint cache. A checkpoint cache is a service that allows the storage and retrieval of checkpoints for specific kernel versions and configuration options. This service can be implemented in several ways, including: a local cache for quickly booting kernels that have already been booted; a remote cache for use by an organization that already contains checkpoints for commonly used kernels; and a content delivery network (CDN) cache for use that already contains checkpoints for the most commonly used kernel versions and configuration options.

[0074] The examples discussed herein use UML because UML provides a fast, software-based virtualization approach. However, the embodiments disclosed herein are not limited to the choice of UML as the virtualization technology. Any other suitable virtualization technology can be used.

[0075] Compared to other virtualization technologies, UML has significant advantages, requiring neither hardware support nor superuser (e.g., "root") access to the host. Although UML has some limitations, such as being strictly single-threaded and slightly slower execution, these limitations do not affect the embodiments disclosed herein because BPF programs are single-threaded and are compiled into machine code via Just-In-Time (JIT) compilation for direct execution within the Linux kernel.

[0076] Libvirt can be used to virtualize various virtualization technologies, including Quick Emulator (QEMU), Kernel-based Virtual Machine (KVM), Linux Containers (LXC), Xen, VMware SX (and ESX, ESXi, etc.). Libvirt is an open-source API and management tool for managing platform virtualization. It can be used to manage any of these virtualization technologies. In some embodiments, virtual machine snapshots are used instead of process checkpoints. Creating a bootable environment will require more work because the method of booting a UML kernel by creating a root filesystem according to the container specification is no longer applicable. At this point, a bootloader needs to be found and installed, and the kernel needs to be mounted to the virtual machine's root filesystem.

[0077] VirtualBox is another available virtualization method, but it suffers from the same problems as libvirt. However, given that VirtualBox's API is based on Microsoft's component object model (see Cross-Platform Component Object Model or XPCOM), it is more difficult to use.

[0078] In some embodiments, the testing framework is provided as Software as a Service (SaaS). In some embodiments, a version of the testing framework may essentially implement "BPF Test as a Service". Some embodiments include the ability to perform tests as a cloud service on different kernel versions and different compilation options.

[0079] Companies can use host kernel testers to test their development content and, as part of a CI / CD pipeline, can push their programs and tests to a test framework to run tests on a matrix of multiple (e.g., various different) kernel versions and compilation options.

[0080] Because building a kernel takes time, some implementations provide regular users with a kernel that has checkpoints already set. Other implementations then control the testing of specific service users using custom kernels. In this scenario, regular users can test on a Long-Term Support (LTS) kernel, but if they want to use a custom kernel that simulates their production environment (which may not even be based on an LTS kernel), they need to unlock the corresponding capabilities.

[0081] Figure 8 This is a schematic block diagram of a computing node 800 according to some embodiments of this disclosure. Any methods or steps described herein can be performed by one or more computing nodes 800 described herein. Specifically, steps related to the test framework can be performed by one or more computing nodes 800. As shown, the computing node 800 includes a control system 802, which includes one or more processors 804 (e.g., a central processing unit (CPU), application-specific integrated circuit (ASIC), field-programmable gate array (FPGA), etc.), memory 806, and network interface 808. The one or more processors 804 are also referred to herein as processing circuitry. The one or more processors 804 are used to provide one or more functions of the computing node 800 described herein. In some embodiments, the functions are implemented in software, which is stored, for example, in memory 806 and executed by one or more processors 804.

[0082] Figure 9 This is a schematic block diagram illustrating a virtualization embodiment of a computing node 800 according to some embodiments of the present disclosure. This discussion also applies to other types of computing nodes. Furthermore, other types of computing nodes may have similar virtualization architectures. Similarly, optional features are indicated by dashed boxes.

[0083] The “virtualized” computing node used in this paper is one implementation of computing node 800, wherein at least a portion of the functionality of computing node 800 is implemented as a virtual component (e.g., by a virtual machine executing on a physical processing node in the network). As shown in the figure, in this example, computing node 800 may include the control system 802 described above. Computing node 800 includes one or more processing nodes 900 coupled to or part of network 902. If control system 802 is present, control system 802 is connected to processing nodes 900 via network 902. Each processing node 900 includes one or more processors 904 (e.g., CPU, ASIC, FPGA, etc.), memory 906, and network interface 908.

[0084] In this example, the functionality 910 of the computing node 800 described herein is implemented at one or more processing nodes 900 in any desired manner, or distributed between one or more processing nodes 900 and the control system 802. In some specific embodiments, some or all of the functionality 910 of the computing node 800 described herein is implemented as virtual components executed by one or more virtual machines, which are implemented in a virtual environment hosted by the processing node 900. Those skilled in the art will understand that additional signaling or communication is used between the processing node 900 and the control system 802 in order to perform at least some of the desired functionality 910.

[0085] In some embodiments, a computer program including instructions is provided that, when executed by at least one processor, causes the at least one processor to perform the functions of computing node 800, or to perform the functions of a node (e.g., processing node 900) that implements one or more functions 910 of computing node 800 in a virtual environment according to any embodiment described herein. In some embodiments, a carrier including the above-described computer program product is provided. The carrier is one of electronic signals, optical signals, radio signals, or computer-readable storage media (e.g., non-transitory computer-readable media such as memory).

[0086] Figure 10 This is a schematic block diagram of a computing node 800 according to other embodiments of the present disclosure. The computing node 800 includes one or more modules 1000, each implemented in software. Modules 1000 provide the functionality of the computing node 800 described herein. This discussion also applies to... Figure 9 The processing node 900 shown can be implemented at one of the processing nodes 900, or distributed among multiple processing nodes 900, and / or distributed between the processing node 900 and the control system 802.

[0087] Any suitable steps, methods, features, functions, or benefits disclosed herein may be performed by one or more functional units or modules of one or more virtual devices. Each virtual device may include multiple such functional units. These functional units may be implemented by processing circuitry, which may include one or more microprocessors or microcontrollers and other digital hardware, including digital signal processors (DSPs), application-specific digital logic devices, etc. The processing circuitry may be configured to execute program code stored in memory, which may include one or more types of memory, such as read-only memory (ROM), random access memory (RAM), cache memory, flash memory devices, optical storage devices, etc. The program code stored in memory includes program instructions for executing one or more telecommunications and / or data communication protocols and instructions for executing one or more technologies described herein. In some implementations, the processing circuitry may be used to cause corresponding functional units to perform corresponding functions according to one or more embodiments of this disclosure.

[0088] Although the processes in the accompanying drawings may illustrate a particular sequence of operations performed by certain embodiments of this disclosure, it should be understood that such sequence is merely exemplary (e.g., alternative embodiments may perform operations in a different order, combine certain operations, overlap certain operations, etc.).

[0089] At least some of the following abbreviations may be used in this disclosure. In the event of inconsistencies between abbreviations, the usage listed above shall prevail. If the same abbreviation is listed multiple times below, the usage listed first shall take precedence over the usage listed subsequently.

[0090] 3GPP Third Generation Partnership Project

[0091] 5G (Fifth Generation)

[0092] 5GC fifth-generation core network

[0093] 5GS fifth-generation system

[0094] ABI Application Binary Interface

[0095] AF application functions

[0096] AMF access and mobility management functions

[0097] AN access network

[0098] AP access point

[0099] API Application Programming Interface

[0100] ASIC (Application-Specific Integrated Circuit)

[0101] AUSF Authentication Server Functionality

[0102] BPF Berkeley Package Filter

[0103] cBPF Classic Berkeley Packet Filter

[0104] CDN (Content Delivery Network)

[0105] CPU Central Processing Unit

[0106] CRIU Userspace Checkpoint / Restore

[0107] DCI downlink control information

[0108] DN Data Network

[0109] DSP Digital Signal Processor

[0110] eBPF Extended Berkeley Packet Filter

[0111] eNB Enhanced or Evolved Node B

[0112] EPS Evolution Grouping System

[0113] E-UTRA Evolutionary Universal Terrestrial Radio Access

[0114] FFI External Function Interface

[0115] FPGA Field Programmable Gate Array

[0116] gNB New Radio Base Station

[0117] gNB-DU New Radio Base Station Distributed Unit

[0118] HSS belongs to user server

[0119] IoT

[0120] IP Internet Protocol

[0121] KVM is a kernel-based virtual machine.

[0122] LTE Long Term Evolution

[0123] LXC Linux containers

[0124] MAC Media Access Control

[0125] MME (Mobility Management Entity)

[0126] MTC Machine Type Communication

[0127] NEF Network Open Functions

[0128] NF Network Functions

[0129] NR New Radio

[0130] NRF Network Functions Repository Functions

[0131] NSSF Network Slice Selection Function

[0132] OTT over-the-top service

[0133] PC Personal Computer

[0134] PCF strategy control function

[0135] PDSCH Physical Downlink Shared Channel

[0136] P-GW Packet Data Network Gateway

[0137] PRS positioning reference signal

[0138] QEMU Quick Simulator

[0139] QoS (Quality of Service)

[0140] RAM (Random Access Memory)

[0141] RAN (Radio Access Network)

[0142] ROM (Read-Only Memory)

[0143] RP receiver point

[0144] RRH remote radio head

[0145] RTT round trip time

[0146] SaaS (Software as a Service)

[0147] SCEF service capability opening function

[0148] SMF Session Management Function

[0149] TCI Transport Configuration Indicator

[0150] TP transmission point

[0151] TRP Sending and Receiving Points

[0152] UDM Unified Data Management

[0153] UE User Equipment

[0154] UML User Mode Linux

[0155] UPF User Face Functions

[0156] Those skilled in the art will recognize improvements and modifications to the embodiments of this disclosure. All such improvements and modifications are considered to fall within the scope of the concept disclosed herein.

Claims

1. A method for testing a program, comprising: Obtain (100) the Berkeley Packet Filter (BPF) program to be tested; Obtain (102) the test to be run on the BPF program; The test described in (104) is performed on the BPF program against multiple Linux kernels, wherein the multiple Linux kernels include one or more of the following: Multiple Linux kernel versions; as well as Multiple Linux kernel configuration options; and Report (106) presents the results of the tests performed on the BPF program against the aforementioned multiple Linux kernels.

2. The method according to claim 1, wherein, Performing the test on the BPF program includes: running the multiple Linux kernels in the corresponding multiple virtualization environments to perform the test on the BPF program.

3. The method according to claim 2, wherein, Running the multiple Linux kernels in the corresponding multiple virtualization environments includes running each of the multiple Linux kernels in a corresponding user-mode Linux UML process.

4. The method according to claim 2, wherein, Running the plurality of Linux kernels in the corresponding plurality of virtualization environments includes running each of the plurality of Linux kernels in one of the following groups: QEMU fast emulator, KVM kernel-based virtual machine, LXC Linux container, Xen, and VMware.

5. The method according to any one of claims 1 to 4, further comprising: Before performing the test on the BPF program, one or more of the plurality of Linux kernels are compiled using the corresponding Linux kernel version and Linux kernel configuration.

6. The method according to any one of claims 1 to 5, further comprising: Before performing the test on the BPF program, one or more Linux kernels with corresponding Linux kernel versions and Linux kernel configurations are retrieved from the cache.

7. The method according to claim 6, wherein, The one or more Linux kernels retrieved from the cache include checkpoint information for speeding up the boot process.

8. The method according to any one of claims 6 to 7, wherein, The cache includes one or more of the following: A local cache used for quickly launching kernels that have already been launched; It already includes a remote cache of checkpoints for commonly used kernels; as well as The content delivery network (CDN) cache provided for use already includes checkpoints for the most commonly used kernel versions and configuration options.

9. The method according to any one of claims 7 to 8, wherein, The checkpoint information includes user space checkpoint / recovery CRIU information.

10. The method according to any one of claims 1 to 9, wherein, Performing the test on the BPF program includes determining whether the BPF program passes BPF verification for the plurality of Linux kernels.

11. The method according to any one of claims 1 to 9, wherein, Performing the test on the BPF program includes determining whether the output of the BPF program corresponds to the expected output for the plurality of Linux kernels.

12. The method according to any one of claims 1 to 11, wherein, Performing the test on the BPF program includes: Provide input context and input data; and Receive the output context, output data, return code, and / or execution time of the BPF program.

13. The method according to any one of claims 1 to 12, wherein, Performing the tests on the BPF program includes testing the multiple Linux kernels from user space.

14. The method according to any one of claims 1 to 13, wherein, The method integrates with existing test suites and / or frameworks.

15. The method according to claim 14, wherein, The existing test suites and / or frameworks include unittest or GoogleTest.

16. A computing node (800) including one or more processors (804) and a memory (806), the memory (806) including instructions for causing the computing node (800) to perform the following operations: Obtain the Berkeley Package Filter (BPF) program to be tested; Obtain the tests to be run on the BPF program; The test was performed on the BPF program against multiple Linux kernels, wherein, The plurality of Linux kernels includes one or more of the following: Multiple Linux kernel versions; as well as Multiple Linux kernel configuration options; and The report presents the results of the tests performed on the BPF program against the aforementioned multiple Linux kernels.

17. The computing node (800) according to claim 16, wherein, Performing the test on the BPF program includes: running the multiple Linux kernels in the corresponding multiple virtualization environments to perform the test on the BPF program.

18. The computing node (800) according to claim 17, wherein, Running the multiple Linux kernels in the corresponding multiple virtualization environments includes running each of the multiple Linux kernels in a corresponding user-mode Linux UML process.

19. The computing node (800) according to claim 17, wherein, Running the plurality of Linux kernels in the corresponding plurality of virtualization environments includes running each of the plurality of Linux kernels in one of the following groups: QEMU fast emulator, KVM kernel-based virtual machine, LXC Linux container, Xen, and VMware.

20. The computing node (800) according to any one of claims 16 to 19 further comprises: Before performing the test on the BPF program, one or more of the plurality of Linux kernels are compiled using the corresponding Linux kernel version and Linux kernel configuration.

21. The computing node (800) according to any one of claims 16 to 21, further comprising: Before performing the test on the BPF program, one or more Linux kernels with corresponding Linux kernel versions and Linux kernel configurations are retrieved from the cache.

22. The computing node (800) according to claim 21, wherein, The one or more Linux kernels retrieved from the cache include checkpoint information for speeding up the boot process.

23. The computing node (800) according to any one of claims 21 to 22, wherein, The cache includes one or more of the following: A local cache used for quickly launching kernels that have already been launched; It already includes a remote cache of checkpoints for commonly used kernels; as well as The content delivery network (CDN) cache provided for use already includes checkpoints for the most commonly used kernel versions and configuration options.

24. The computing node (800) according to any one of claims 22 to 23, wherein, The checkpoint information includes user space checkpoint / recovery CRIU information.

25. The computing node (800) according to any one of claims 16 to 24, wherein, Performing the test on the BPF program includes determining whether the BPF program passes BPF verification for the plurality of Linux kernels.

26. The computing node (800) according to any one of claims 16 to 24, wherein, Performing the test on the BPF program includes determining whether the output of the BPF program corresponds to the expected output for the plurality of Linux kernels.

27. The computing node (800) according to any one of claims 16 to 26, wherein, Performing the test on the BPF program includes: Provide input context and input data; and Receive the output context, output data, return code, and / or execution time of the BPF program.

28. The computing node (800) according to any one of claims 16 to 27, wherein, Performing the tests on the BPF program includes testing the multiple Linux kernels from user space.

29. The computing node (800) according to any one of claims 16 to 28, wherein, The method integrates with existing test suites and / or frameworks.

30. The computing node (800) according to claim 29, wherein, The existing test suites and / or frameworks include unittest or GoogleTest.

31. A computer-readable medium comprising instructions that, when executed on at least one processor (804), cause the at least one processor (804) to perform the method according to any one of claims 1 to 15.

Images