Fine-grained real-time preemption of small code based on runtime thresholds

Abstract

Examples are described for providing fine-grained real-time preemption of small code based on runtime thresholds. The controller inserts a checkpoint into an extended Berkley packet filter (eBPF) bytecode of the third-party small code prior to verifying the third-party small code. The device executes the code group at the hook point of the application. The inserted checkpoint determines the runtime of the small code. In response to the run-time exceeding the threshold, the device terminates the code group. The application may be a virtualized radio access network (vRAN) network function, and the small code may control the vRAN function or derive network metrics. The application may be executed in a container management system that modifies a container for the application to install code including a function associated with a hook point of the application to the container; detecting an annotation identifying a container of the code group; and symbologically linking the code to the hook point.

Description

Background Art

[0001] A radio access network (RAN) can provide wireless access to a network to multiple user devices. User devices can communicate wirelessly with a base station, which forwards the communication to a core network. Typically, a base station in a RAN is implemented by dedicated processing hardware (e.g., an embedded system) located near a radio unit including an antenna. A base station can perform lower layer processing including physical (PHY) layer and medium access control (MAC) layer processing for one or more cells. There may be costs associated with deploying dedicated processing hardware for each base station in a RAN, particularly for a RAN that includes small cells with a relatively small coverage area. Additionally, the dedicated processing hardware may be a single point of failure for a cell.

[0002] The virtualized radio access network can utilize an edge data center with general computing resources to perform RAN processing for one or more cells. That is, instead of performing PHY and MAC layer processing locally on dedicated hardware, the virtualized radio access network can forward radio signals from the radio unit to the edge data center for processing, and similarly forward signals from the edge data center to the radio unit for wireless transmission. In a specific example, a cloud computing environment can be used to provide mobile edge computing (MEC), where certain functions of the mobile network can be provided as workloads on nodes in the cloud computing environment. In the MEC, a centralized unit (CU) can be implemented in a backend node, one or more distributed units (DU) can be implemented in an intermediate node, and various remote units (RU) that can provide at least the PHY and / or MAC layers of other RAN nodes of a base station or mobile network can be deployed at an edge server. The RU can communicate with the CU via one or more DUs. In an example, the DU can provide higher network layer functionality for the RAN, such as radio link control (RLC) or packet data convergence protocol (PDCP) layer functions. The RU can facilitate access to the CU for various downstream devices, such as user equipment (UE), Internet of Things (IoT) devices, and the like.

[0003] Because edge data centers utilize general-purpose computing resources, virtualized RAN can provide scalability and fault tolerance for base station processing. For example, edge data centers can allocate a variable number of computing resources (e.g., servers) to perform PHY layer processing for radio units associated with edge data centers based on workload. In addition, virtualized RAN can implement multiple layers of RAN processing at the data center, making the collection of multiple data feeds possible. Summary of the invention

[0004] A simplified summary of one or more aspects is given below to provide a basic understanding of these aspects. This summary is not an extensive overview of all contemplated aspects, and is not intended to identify key elements or important elements of all aspects, nor is it intended to delineate the scope of any or all aspects. Its sole purpose is to present some concepts of one or more aspects in a simplified form as a prelude to a more detailed description that is presented later.

[0005] In some aspects, the technology described herein relates to a method comprising: inserting a checkpoint into an extended Berkeley Packet Filter (eBPF) bytecode of a third-party codelet before verification of the third-party codelet; executing the third-party codelet at a hook point of an application; determining a running time of the third-party codelet through the inserted checkpoint; and terminating the third-party codelet in response to the running time exceeding a threshold.

[0006] In some aspects, the technology described herein relates to a method in which the application is a virtualized radio access network (vRAN) network function and the codelet is configured to control the vRAN function or derive network metrics.

[0007] In some aspects, the technology described herein relates to a method in which inserting checkpoints into eBPF bytecode includes: at the beginning of the eBPF bytecode, adding initialization code including a function call to a first function, the first function storing a time when execution of a small code starts; and at various locations of the eBPF bytecode, adding checkpoint code including a function call to a second function, the second function comparing the time of execution of the second function with the stored time when execution of the small code starts.

[0008] In some aspects, the techniques described herein relate to a method in which various locations of eBPF bytecode include the following: after a function call to a function marked as long-lasting by an application; for each logical path through a codelet, at every N instructions along the logical path from a last checkpoint; and within each loop of the codelet, regardless of the distance from the last checkpoint.

[0009] In some aspects, the techniques described herein relate to a method in which initialization code or checkpoint code is configured to: store register state to a stack; call a first function or a second function; and restore the register state from the stack.

[0010] In some aspects, the techniques described herein relate to a method in which checkpoint code is configured to skip a function call to a second function based on a specified frequency of execution of the checkpoint code.

[0011] In some aspects, the techniques described herein relate to a method in which inserting a checkpoint into eBPF bytecode further comprises updating a jump offset to account for the inserted checkpoint code.

[0012] In some aspects, the techniques described herein relate to a method, further comprising statically verifying, by a verifier, that the eBPF bytecode including the checkpoint is safe.

[0013] In some aspects, the technology described herein relates to a method and also includes modifying an application to include a plurality of hook points, the plurality of hook points including a hook point for a third-party codelet.

[0014] In some aspects, the technology described herein relates to a method in which an application is executed in a container of a container orchestration system, and further includes: mounting code including functions associated with a hook point of the application to the container; during execution of the application, detecting annotations for the container, the annotations identifying third-party small code; and symbolically linking the third-party small code indicated by the annotations to the hook point of the application.

[0015] In some aspects, the technology described herein relates to a method in which mounting code including a function includes: modifying an entry point of a container to execute a script to dynamically load a library capable of executing a small code at a hook point.

[0016] In some aspects, the techniques described herein relate to a method further comprising making available to a container a location containing the compiled and verified codelet.

[0017] In some aspects, the technology described herein relates to a method that also includes adding a component to a container, the component configured to: monitor whether a codelet is executed at a hook point, and select which codelet to execute at the hook point.

[0018] In some aspects, the technology described herein relates to a system for injecting telemetry collection capabilities in an application running on a container orchestration system, including: an application programming interface (API) server, configured to: receive a user configuration for a container image for an application, the user configuration including a header indicating dynamic loading of a small code, and when the application is executed in a main container, receive a configuration map for the small code; a mutating network hook, configured to modify the user configuration in response to the header: add a sidecar container to the container image to run with the main container, mount one or more libraries into the main container, wherein the one or more libraries define functions associated with hook points within the application, modify an entry point for the main container to run a script, the script loads one or more libraries, and mounts a directory for compiled and verified small code into the main container; a node agent, configured to: monitor the API server for the configuration map, compile and verify the small code, and place the small code in the directory; and a controller, configured to: receive annotations to the user configuration via the API server, wherein the sidecar container is configured to symbolically link the small code to the hook point within the application based on the annotations.

[0019] In some aspects, the technology described herein relates to a system in which a node agent is configured to insert checkpoints into the bytecode of a small code before verification, and wherein a main container is configured to: execute the small code at a hook point of an application; determine the running time of the small code through the inserted checkpoints; and terminate the small code in response to the running time exceeding a threshold.

[0020] In some aspects, the techniques described herein relate to a system in which the application is a vRAN network function and the codelet is configured to control the vRAN function or derive network metrics.

[0021] In some aspects, the technology described herein relates to a system in which, in order to insert a checkpoint into a bytecode, a node agent is configured to: at the beginning of the bytecode, add initialization code including a function call to a first function, the first function storing the time when the execution of the small code starts; and at various locations in the bytecode, add checkpoint code including a function call to a second function, the second function comparing the time of execution of the second function with the stored time when the execution of the small code starts.

[0022] In some aspects, the techniques described herein relate to a system in which various locations of bytecode include the following: after a function call to a function that is marked as long-lasting by an application; for each logical path through a codelet, at every N instructions along the logical path from a last checkpoint; and within each loop of the codelet, regardless of the distance from the last checkpoint.

[0023] In some aspects, the techniques described herein relate to a system in which initialization code or checkpoint code is configured to: store a register state to a stack; call a first function or a second function; and restore the register state from the stack.

[0024] In some aspects, the techniques described herein relate to a system in which checkpoint code is configured to skip a function call to a second function based on a specified frequency of execution of the checkpoint code.

[0025] In some aspects, the techniques described herein relate to a system in which the node agent is further configured to update the jump offset to account for the inserted checkpoint code.

[0026] In some aspects, the techniques described herein relate to a system in which the node agent is further configured to statically verify that the bytecode including the checkpoint includes memory checks, includes only bounded loops, and includes only calls to allowed helper functions.

[0027] In some aspects, the technology described herein relates to an apparatus comprising: a memory storing computer-executable instructions; and at least one processor, the processor being configured to execute instructions to: insert a checkpoint into the eBPF bytecode of a third-party small code before verification of the third-party small code; and load the third-party small code at a hook point of an application, wherein the small code is configured to: determine the running time of the third-party small code through the inserted checkpoint; and terminate the third-party small code in response to the running time exceeding a threshold.

[0028] To achieve the foregoing and related ends, one or more aspects include features fully described below and particularly pointed out in the claims. The following description and the accompanying drawings set forth in detail certain illustrative features of one or more aspects. However, these features are merely indicative of some of the various ways in which the principles of the various aspects may be employed, and this description is intended to include all such aspects and their equivalents. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] Figure 1 is a diagram of an example virtualized radio access network (vRAN) connecting user equipment (UE) to a core network.

[0030] Figure 2 is a diagram of an example gate system that provides a code execution framework that allows secure dynamic loading of custom code in a sandbox environment at different locations in a vRAN.

[0031] Figure 3 An example architecture for loading codelets in a container orchestration system is shown.

[0032] Figure 4is an example code listing for codelets that may be executed in a vRAN using the example gate system.

[0033] Figure 5 is a diagram illustrating patching a checkpoint into a codelet 50 to create a patched codelet.

[0034] Figure 6 is a schematic diagram of an example of an apparatus for implementing a doorway controller.

[0035] Figure 7 is a flow diagram of an example of a method for executing a codelet within an application.

[0036] Figure 8 is a flow diagram of an example of a method for loading a codelet into an application executing in a container.

[0037] Fig. 9 Shown include Figure 6 The components shown are examples of devices with additional optional components detailing the components. DETAILED DESCRIPTION

[0038] The detailed description set forth below in conjunction with the accompanying drawings is intended as a description of various configurations and is not intended to represent the only configuration in which the concepts described herein can be practiced. The detailed description includes specific details for providing a thorough understanding of the various concepts. However, it will be apparent to those skilled in the art that these concepts can be practiced without these specific details. In some instances, well-known components are shown in block diagram form to avoid making such concepts difficult to understand.

[0039] The present disclosure describes various examples related to the use of small codes for analysis and control. In particular, the small code can be limited to a runtime threshold that can be set at the microsecond level to provide fine-grained, real-time preemption of the small code. For example, such preemption can be useful in real-time environments such as virtualized radio access networks (vRANs) with strict timing control for network functions. Any system with real-time requirements for loading small codes can benefit from the fine-grained preemption of small codes using the techniques described herein. For example, a small code executed within the Linux kernel can utilize preemption to meet timing requirements.

[0040] The key shift in the radio access network (RAN) in 5G is the migration to an open RAN architecture, which virtualizes and disaggregates the 5G RAN across multiple open interfaces. This approach promotes innovation by allowing multiple vendors to propose unique solutions for different components at a faster pace. In addition, a new component introduced in the open RAN architecture, called the Radio Intelligent Controller (RIC), allows third parties to build new, vendor-agnostic monitoring and optimization use cases through O-RAN standardized interfaces.

[0041] Despite this compelling vision, opportunities for innovation remain largely untapped because of two major challenges. The first challenge is related to flexible data collection for monitoring and telemetry applications. RAN functions can generate large amounts of telemetry data at high frequencies (e.g., gigabytes per second). Collecting, transmitting, and processing this data can put pressure on computing and network capacity. The conventional approach standardized by the Third Generation Partnership Project (3GPP) defines a small set of aggregated cell key performance indicators (KPIs) collected every few seconds or minutes. The O-RAN RIC extends this idea by providing new KPIs with finer time granularity. Each KPI is defined by a service model (in the form of an API), most of which are standardized by O-RAN. However, this approach evolves slowly and does not scale well because of the limited number of initial use cases and the need to standardize new proposals. The second challenge is due to the real-time nature of many RAN control and data plane operations. In order to support new service models, any new functions added to these operations must be completed within deadlines, typically ranging from nanoseconds (μs) to milliseconds (ms). Deadline violations can result in performance degradation or even vRAN crashes. Any changes to these critical paths can create significant design challenges and make RAN vendors reluctant to add new functionality.

[0042] In one aspect, the present disclosure describes a system that provides dynamic monitoring and control of vRAN functionality through a provably secure execution environment. The system can provide access and security for the execution environment and can be referred to as a gate system. The system extends the RIC concept by allowing vRAN operators to write their own monitoring and control code without any assistance from suppliers. The monitoring and control code, referred to as a small code, can be deployed at different vRAN components at runtime without interrupting vRAN operations. Small codes are usually executed inline on the vRAN critical path, so the small code can directly access all important internal vRAN data structures and can collect arbitrary statistics. Existing and new O-RAN service models can be implemented as small codes, and small codes can be used to implement fast control operations such as radio resource allocation. The small code model can be extended to other computing environments. For example, small codes may be useful in industrial automation or robotic control where strict timing requirements also exist. In one aspect, small codes can be deployed into applications for executing workloads via a container orchestration system, which loads the small code into a container and symbolically links the small code to a hook point within the application.

[0043] In order to provide the above functionality, the system design pays special attention to security, efficiency and flexibility. The system utilizes the extended Berkeley packet filter (eBPF) technology, which solves similar problems in the Linux kernel. The Linux community has observed that the predefined kernel API limits its observability and controllability. For this reason, eBPF is developed as a sandbox execution environment, in which system operators can deploy custom code that can access and modify selected kernel structures. Before loading the small code, the eBPF execution environment statically verifies the small code and only allows safe small code to run. eBPF has created a surge in new applications at all levels of the Linux kernel.

[0044] Gateway strives to achieve similar goals in the RAN space and other environments. Gateway builds on a userspace implementation of eBPF, called Userspace Berkeley Packet Filter (uBPF), while also leveraging existing tooling of the eBPF toolchain such as compilers and verifiers. At a high level, Gateway runs a small code inside a uBPF virtual environment that is inlined in the control and data paths of the deployed system (e.g., vRAN). Two main sets of challenges are unique to vRAN.

[0045] The first set of challenges is about flexibility. It is unclear how much and which RAN monitoring or control data is required to build useful applications. The system focuses on key locations, interfaces, and abstractions within the standard vRAN architecture, which provides rich and diverse data from only a few collection points. The system includes a toolchain that allows developers to define arbitrary output data structures that are shipped to the RIC. This data structure contrasts with the existing Linux output data model that is constrained to a specific format. The gate system can provide a variety of applications. For example, four categories of applications have been implemented and tested: flexible statistics collection, interference detection, network slicing scheduler, and real-time ML prediction. These categories include 17 applications that add flexibility to vRAN. These applications (in addition to those defined in the RIC data model) cannot be deployed using the standard O-RAN RIC.

[0046] The second set of challenges is about security. Strict microsecond (μs) execution latency requirements are not typically imposed within the Linux kernel, but are important for vRAN operations. On one hand, the system includes an eBPF patching mechanism that preempts small code that exceeds a certain runtime threshold. In addition, the gate system includes multiple features to ensure a lock-free and non-preemptive design in the fast path (where the small code runs), minimizing the impact of the system on the performance of the vRAN. The system can also include static verification to cover the newly introduced flexible output data structures.

[0047] In one aspect, to enforce the runtime threshold, the system may insert a checkpoint into the eBPF bytecode of the third-party codelet prior to verification of the third-party codelet. The system may execute the third-party codelet at the hook point of the application. The system may determine the runtime of the third-party codelet through the inserted checkpoint. The system may terminate the third-party codelet in response to the runtime exceeding the threshold.

[0048] On the other hand, the present disclosure provides a system for the injection of small codes in running applications on a cluster of a container orchestration system. Small codes can provide additional capabilities, such as telemetry collection, configuration, or control. The system includes an application programming interface (API) server, which is configured to receive a user configuration for a container image of an application. The user configuration includes a header indicating the dynamic loading of the small code. The API service is configured to receive a configuration map for the small code when the application is being executed in the main container. The system includes a mutation hook configured to modify the user configuration in response to the header. The mutation hook can, for example, add a sidecar container to the container image to run with the main container, mount one or more libraries in the main container, wherein one or more libraries define functions associated with hook points within the application, modify the entry point for the main container to run a script, which loads one or more libraries, and mounts a directory for compiled and verified small codes in the main container. The system includes a node agent, which is configured to monitor the API server for the configuration map, compile and verify the small code, and place the small code in the directory. The system includes a controller configured to receive annotations to a user configuration via an API server. The sidecar container is configured to symbolically link a small code to a hook point within the application based on the annotation.

[0049] In one aspect, inserting checkpoints into the bytecode of a codelet to monitor the runtime of the codelet can ensure the security of the codelet executed within another application by forcing the codelet to stop if it would violate the timing requirements of the application. In addition, the placement of checkpoints within specific locations in the bytecode can provide a guarantee that the runtime is checked frequently enough to meet the timing requirements. In addition, loading the codelet into the hook point of the application via the container management system allows for runtime customization of the application and dynamic switching of application behavior. Therefore, the aspects described herein improve the performance of a computer system including a vRAN system.

[0050] Now turn to Figures 1 to 6 , examples are depicted with reference to one or more components and one or more methods that can perform the actions or operations described herein, where the components and / or actions / operations in dashed lines may be optional. Figure 7 and 8 The operations described in the description are presented in a particular order and / or performed by example components, but in some examples, the ordering of the actions and the execution of the actions by the components may vary depending on the implementation. In addition, in some examples, one or more of the actions, functions, and / or described components may be performed by a specially programmed processor, a processor executing specially programmed software or computer-readable media, or by any other combination of hardware components and software components capable of performing the described actions or functions.

[0051] Figure 1 1 is a diagram of an example vRAN 100 that connects a user equipment (UE) 110 to a core network 160. The vRAN 100 may include a radio unit 120 that transmits and receives wireless signals with the UE 110. The vRAN 100 may include a virtual distributed unit (vDU) 130 that performs processing at, for example, a physical (PHY) layer, a medium access control (MAC) layer, and a radio link control (RLC) layer. The vRAN 100 may include a virtual central unit (vCU) 140 that performs processing at higher layers of the radio protocol stack. The vRAN 100 may include a RAN intelligent control (RIC) that performs autonomous configuration and optimization of the vRAN 100.

[0052] The functional division between vDU 130 and vCU 140 may depend on the functional partitioning architecture. vCU 140 may be divided into a central unit control plane (CU-CP) and a central unit user plane (CU-UP). CU-UP may include a packet data convergence protocol (PDCP) layer and a service data adaptation (SDAP) layer and a radio resource control (RRC) layer. Different components or layers may have different latency and throughput requirements. For example, the PHY layer may have a latency requirement between 125μs and 1ms and a throughput requirement greater than 1Gbps, the MAC and RLC layers may have a latency requirement between 125μs and 1ms and a throughput requirement greater than 100Mbps, and the higher layers at the vCU may have a latency requirement greater than 125μs and a throughput requirement greater than 100Mbps.

[0053] Programmability in the open RAN components can be facilitated by a near real-time RIC 150. Network operators can install applications (Apps 152, such as xApps in open RAN) on top of the RIC 150 that can collect detailed telemetry and can use the telemetry to optimize network performance or report problems in near real-time (e.g., >10ms to several seconds). Data collection and control of vRAN components can be facilitated by a service model that must be embedded in the vRAN functionality by the vendor. The service model can explicitly define the type and frequency of data reporting for each App 152, as well as a list of control policies that the RIC can use to modify RAN behavior. Each App 152 can specify its own service model, requiring the collection of different telemetry data.

[0054] The initial xApps proposed by Open RAN for standardization focus on optimizing handover, self-organizing network (SON) functionality, anomaly detection, and coarse-grained radio resource allocation. In these use cases, important network events occur at low speeds (100ms to seconds). The amount of telemetry data generated is low, and all data is transmitted to the RIC. This allows App 152 to have complete insight into the domain and tune vRAN functionality through a preset set of control policies. Unfortunately, this approach has not been extended to many other use cases for RAN monitoring and control for two main reasons.

[0055] First, for many low-level applications (e.g., PHY and MAC), due to the huge amount of data (e.g., Figure 1 As shown in the figure, it is inefficient to transmit all data to the RIC. For example, applications like localization, channel estimation, interference detection, and beamforming may require uplink IQ samples from the physical layer. Transmitting all IQ samples to the RIC requires more than 5 Gbps per cell for 100 MHz 4′4 MIMO. The standard RIC design overcomes this problem by specifying the required data in terms of type and frequency (e.g., in raw or preprocessed format) in the service model of each xApp. The form of preprocessing (e.g., subsampling, averaging, or data subsets) depends greatly on the service model, which imposes severe limitations on interoperability because vRAN vendors must implement and support each proprietary service model.

[0056] Secondly, many real-time control loops (e.g., packet scheduling and power control) have very tight timing constraints (<10ms). Such timing constraints cannot be met by standard RIC designs with expected latency on the order of hundreds of milliseconds. As in the case of telemetry, the choice of control strategy by the xApp is constrained by a set of (standardized) strategies provided by the appropriate service model for the RIC. The vRAN vendor has to implement and integrate such algorithms in its stack while ensuring that they do not impact the runtime performance of the vRAN components. This framework of the service model implanted by the vRAN vendor makes it very difficult to implement new custom variants of tight control loops in practice.

[0057] Figure 2 1 is an example diagram of an example gate system 200 that provides a code execution framework that allows secure dynamic loading of custom code in a sandbox environment at different locations in a vRAN. The gate system may include a gate controller 210 that may be implemented at a RIC 150 and a gate device 230 that may be implemented at a network node such as a vDU 130 or a vCU 140.

[0058] The gate device 230 can be any application 232 that allows the execution of custom code, such as a vRAN component (e.g., a vCU or vDU process). The gate system 200 can execute custom code inside a user-mode version of eBPF called uBPF. An application 232, such as a vRAN component, is modified to introduce a gate call point or hook point 234 at a selected location in the vRAN function where an eBPF virtual machine (VM) with custom code can be called. The call is inlined with the vRAN code and gives the VM read-only access to a selected internal vRAN context, which includes, for example, various 3GPP-defined data structures. Table 1 below provides exemplary hook points in the vRAN. Table 1

[0059] At runtime, the codelets 272, including custom code, may be dynamically loaded and unloaded from the hook point 234 of the gateway device 230. In some implementations, the codelets 272 may not affect the performance of the gateway device 230 with respect to performing vRAN functions. For example, the codelets 272 may simply collect data. In other implementations, the codelets 272 may control vRAN functions (e.g., make scheduling decisions).

[0060] The gatelet code 272 is custom code that can be deployed at a single hook point 234 on the gate device 230 at runtime. Developers can write the codelet 272 in a programming language such as C and compile the codelet into eBPF bytecode using the eBPF compiler 252. Similar to any eBPF program, the gatelet codelet 272 must be written in a way that allows the codelet 272 to be statically verifiable (for example, the code must introduce memory checks and can only have bounded loops). Any operations required by the codelet 272 that may be potentially unsafe (for example, accessing memory outside of an area that the code group is allowed to access) can only be performed by a set of whitelist helper functions 240 that are considered safe by the framework. The codelet 272 does not maintain any state between two calls. All states must be stored in an external location called a graph 236. For example, the codelet 272 may send its telemetry data through the special map 236 to the gate output thread 238 running at the gate device 230, which forwards the telemetry data to the gate controller 210. In some implementations, the codelet 272 calls the protobuf encoder to write to the special map 236. For each output stream (map of type JANUS_MAP_TYPE_RINGBUF), a single-producer / single-consumer lockless ring buffer is created to push data from the codelet to the output thread 238. This ensures that the codelet will never be preempted and may only drop excess packets in the worst case.

[0061] A codelet set 270 may refer to an integration of coordinated codelets that operate across multiple doorway hook points of doorway devices. Different codelets 272 in a codelet set 270 may be coordinated with very low latency via a shared graph 236. Codelets across devices may be coordinated via a doorway controller 210 if desired.

[0062] The gate controller 210 is responsible for controlling the gate devices 230 and the codelets 270. In the context of O-RAN, the gate controller 210 may be implemented as an App 152 on the RIC 150. Developers may upload codelets 270 to the controller 210 with instructions for loading / unloading them to one or more gate devices 230. For example, instructions may be provided as described below with respect to Figure 3 The described container orchestration system. The controller 210 includes a bytecode verifier 214 that verifies the safety and termination of each codelet before the controller 210 allows the codelet 270 to be loaded. For this purpose, an eBPF verifier (such as the open source PREVAIL eBPF verifier) ​​can be extended with verification for gateway helper functions and output modes. The controller 210 includes a bytecode patcher 212, which is configured to further modify (or patch) the byte code of the uploaded codelet with additional control code, and if the runtime of the codelet exceeds a certain threshold, the additional control code preempts the codelet. The controller 210 includes a just-in-time (JIT) compiler 216 that compiles the patched codelet. The controller 210 includes a network API 218, which is configured to push the compiled codelet to the gateway device 230 over a network. The controller 210 may also include a metadata file generator 220, which is configured to provide metadata files required for enabling flexible output of data using protobuf output mode. The controller 210 may also provide a data collector 222 that collects and deserializes data 224 sent from the gatelet code.

[0063] The gate system can also provide a gate SDK 250 that allows developers to locally test gatelet code early in the development cycle. The SDK 250 includes a compiler 252, a validator 254, and a debugger 256, as well as a header file 258 that includes definitions of all helper functions and graph types supported by the gate device 230.

[0064] The gatelet codeset 270 is JIT-compiled and loaded to the gate device 230 through the network API 218 provided by the gate controller 210. Developers can use the tools provided by the gate SDK 250 to upload the developed codelet codeset 270 to the gate controller 210 and load the codelet codeset 270 to the gate device 230. The instructions required to upload the developed codelet codeset 270 to the gate controller 210 and load the codelet codeset 270 to the gate device 230 are encoded in a descriptor file written in the yet-another-markup-language (YAML) format.

[0065] At a minimum, the YAML file specifies the names of all codelets that are part of the codelet set (eBPF byte code object files in ELF format), the name of the hook in the gate device 230 to which the codelets should be linked, and the calling priority of the codelets. Multiple codelets can be linked to the same hook. The YAML file provides additional optional configuration parameters for further configuring the codelets such as runtime thresholds.

[0066] Codelets 272 of the same codelet set 270 can share state and coordinates. This sharing may be particularly useful when monitoring or control operations are required to be performed with correlation of events and data across different layers of the vRAN stack (for example, in the case of interference detection). State sharing between codelets of a codelet set is performed using a shared graph 236. For this mechanism to work, codelets that want to share state must use exactly the same definition of the shared graph 236 in terms of its type, key_size, value_size, and max_entry, while the name of the graph may be different. Developers can then specify the graph to be linked in the linked_maps section of the YAML descriptor file they use to load the codelet set 270. When the codelet set 270 is loaded, the memory for the shared graph 236 is allocated only once on the gate device 230, and all codelets 272 that share the graph 236 obtain a pointer to the same memory, which the codelets 272 can then call through the helper function 240 to store and load state.

[0067] Figure 3 An example architecture 300 for loading codelets in a container orchestration system 302 is shown. For example, the container orchestration system 302 can be a Kubernetes system that executes applications within containers grouped into workload pods. In some embodiments, the application can be a vRAN function, and the gate device 230 can be implemented in a container. The architecture 300 includes a container API server 310, a mutation hook 320, a controller 330, a node agent 340, and a workload group 370.

[0068] The container API server 310 can interact with the user 312 to configure the container for execution. For example, the user 312 can provide a configuration for the container image. In some embodiments, the configuration can be a yet-another-markup-language (YAML) file or a JavaScript Object Notation (JSON) file that specifies container properties and resources. The configuration may include a header indicating dynamic loading of a small code. The container API server 310 can control the controller 330 and the node agent 340 to instantiate a main container 380 within the workload group 370 and provide resources for the container.

[0069] The mutation hook 320 may include computer executable code for modifying a user configuration of a container image. For example, the mutation hook 320 may modify a user configuration to add the ability for the main container 380 to dynamically load a small code. For example, the mutation hook 320 may add a sidecar container to a container image (e.g., within a workload group 370) to run with the main container. The mutation hook 320 may mount one or more libraries 382 into the main container 380. The library 382 may define functions associated with a hook point 392 within an application 390. The mutation hook 320 may modify an entry point for the main container to run a script that loads one or more libraries 382. For example, the script and library 382 may be defined by a common script and library 350. The mutation hook 320 may mount a directory 386 for compiled and verified small code into the main container.

[0070] The node agent 340 may include computer executable code for dynamically loading a codelet into the main container 380. The codelet may be dynamically defined by the user 312 via a configuration map for the codelet. The container API server 310 may receive the configuration map while the application 390 is executing in the main container 380. The node agent 340 may be used to monitor the API server 310 for the configuration map. The node agent 340 may compile and verify the codelet to generate a compiled and verified codelet 360. The node agent 340 may load the compiled and verified codelet 360 into the codelet directory 386.

[0071] The controller 330 may include computer executable code for controlling a workload group 370 and containers, including a main container 380 and a sidecar container 372. The controller 330 may receive user-configured annotations to the container via the container API server 310. The annotation may associate a codelet with a hook point 392 of an application 390. The controller 330 may provide the annotation to the sidecar container 372. The sidecar container 372 may be configured to symbolically link the codelet indicated by the annotation to the hook point 392 within the application 390. For example, the sidecar container 372 may symbolically link the codelet to a function in a library 382 associated with the hook point. Once the compiled and verified codelet 360 stored in the codelet directory 386 is symbolically linked to the hook point, the codelet may be considered as a mapped codelet 384. When the execution of the application 390 reaches the hook point and calls a function in the library 382, ​​the mapped codelet 384 may be executed via the symbolic link. Accordingly, codelets may be dynamically loaded and executed within application 390 during execution.

[0072] In another implementation, codelets can be dynamically loaded into hook point 234 using a user space implementation of a read-copy-update mechanism (RCU). RCU allows multiple uncoordinated readers to access shared data structures at the expense of longer write / update times. For example, the reader can be a fast vRAN thread that executes the codelets, and the writer is a (non-real-time) thread that updates the list of hook point codelets.

[0073] Figure 4 An example code listing 400 is shown for a small code 410 that can be executed in a vRAN using the system 200. In the example code listing, the system 200 and its components may be referred to as Janus. The gate system 200 and / or architecture 300 activate new monitoring and control capabilities. The example code listing 400 refers to a small code developed for OAI's vDU, which is an open source implementation of the 4G and 5G vRAN stack. At a high level, the example small code 410 is called through a hook introduced at the functional API (FAPI) interface, which is used for communication between the PHY layer and the MAC layer of the vDU. The FAPI packet is a C structure that contains information used to schedule radio resources to a mobile device. In the example small code, a counter maintaining the number of captured FAPI events is sent to the data collector 222 after every 1000 events. Although simple, this small code captures important features that demonstrate the power of the gate system over conventional RIC designs.

[0074] Figure 4The state parameter in line 22 of is the context (type janus_ran_fapi_ctx, lines 28 to 29) which is passed to the function by the vRAN via hook point 234. The state parameter contains a pointer to a FAPIDL configuration request message of type nfapi_dl_config_request_pdu_t (lines 31 to 32). This message describes the complete scheduling allocation for a specific downlink timeslot for each mobile user including over 20 fields including modulation and coding scheme, transport block size, allocated resource blocks, MIMO layers, etc. The validator 24 ensures read-only access to the context information to enable secure zero-copy access to the internal vRAN context. Due to the modular vRAN design, a small number of small interfaces are specified in different standards (3GPP, Small Cell Forum) that carry all relevant states across vRAN components. By adding hook points at these interfaces, the gateway controller can give the gateway code developer access to a large amount of vRAN telemetry. The hooks identified in Table 1 have been implemented. The hooks in Table 1 can be used to activate multiple RIC applications without modifying a single line of code inside the vRAN function except for the addition of the hook point. In some cases, the implementation of the interface may differ across vRAN vendors (e.g., in terms of C-structure memory layout). Due to the flexibility of the gate system, small changes to small code can be used to adjust for observed differences while maintaining the same small code functionality.

[0075] Gatelet codelets rely on a shared memory area called a graph 236 in eBPF to store state across consecutive calls, and to exchange state with other codelets 272 of the same codelet set 270. The gatelet system provides a variety of graph types for storing data, including arrays, hash maps, and bloom filters. As an example, for Figure 4 Codelet 236 is a small code that maintains the counter of the observed FAPI grouping to export to RIC, rather than exporting the entire original captured grouping. The counter is implemented by defining an array diagram (rows 6-11) containing a single element (max_entry). Each time one or more FAPI packets are received, the counter can be restored from memory by a helper function (row 37), incremented by a new number of groups (row 39), and saved again using a helper function (row 45) to be used for the next call to codelet 410. Various security checks are required to activate static verification (e.g., rows 34 and 38). In addition, Figure 236 can be shared between codelets 272 in codelet set 270 to coordinate actions.

[0076] In a similar manner, the gatelet code can use a flexible output scheme to send arbitrary telemetry (e.g., network metrics) data to the data collector 222 through a special type of ring buffer map (JANUS_MAP_TYPE_RINGBUF). The map is linked to a codelet-specific protobuf schema defined by the codelet developer. In the example of Listing 400, the definition of the output map for the custom output protobuf schema called output_msg (line 26) can be seen in lines 13 to 19, along with a single counter field (line 41). The data can be exported to the data collector 222 through another helper function 240 (line 42). This flexibility allows, after the hook point 234 is in place, the implementation of the data model currently available in the O-RAN RIC specification without modifying a single line of code in the vCU 140 and vDU 130.

[0077] Validation ensures that the gatelet code 272 cannot directly modify the state of the vRAN. Alternatively, modification of the vRAN behavior requires a control helper function 240 provided by the vRAN vendor. The helper function 240 can be associated with a specific hook point 234. The verifier 214 can verify that calls to the helper function are allowed for the associated hook point 234. For example, a vendor can allow a codelet to control scheduling by providing a helper function called allocate_resource_blocks() that can only be called at the MAC hook for radio resource allocation. Similar helper functions can be added for other operations (e.g., switching, flow priority). Although the control helper functions are more dependent on the internal vRAN software architecture, they provide a way for developers to customize vRAN behavior at runtime.

[0078] The gate system 200 can provide several example categories of applications for various vRAN use cases. In the first example, the gate system 200 can provide flexible monitoring of the vRAN. As described above, the O-RAN RIC 150 allows the collection of telemetry data for network optimization through the specification of service models. Each model undergoes a long standardization process and then needs to be implemented by each RAN vendor that decides to support it. The gate system 200 can be used to implement more flexible monitoring without relying on the service model. The gate system 200 can be used to implement a small code that extracts key performance indicators (KPIs) specified in the O-RAN key performance measurement (KPM) model and raw scheduling data. The implementation and collection of these KPIs can be implemented in the system 200 without having to change a single line of code in the vRAN function that includes the hook point. Therefore, the new data model can be implemented by developers without the need for further vendor modifications of the vRAN function or standardization between vendors. For example, the system 200 can collect the downlink total physical resource block (PRB) usage KPI by accessing the FAPI hook of Table 1 and capturing the nfapi_dl_config_request_pdu_t structure described in Listing 400, and the structure contains the number of PRBs allocated to each user at each scheduling decision. The contents of the structure can be stored in the gate map 236, averaged over a period of 0.5ms and sent out to the data collector 222 by the small code 272.

[0079] Another example use case for system 200 is external interference detection. Detecting external interferers in cellular networks (e.g., repeaters or jammers with improperly adjusted gain, stray radiation from electronic devices) is a difficult problem that conventionally involves sending experts and dedicated hardware to the site. The gatelet code 270 can define a spectrum analyzer to detect external interference through a 5G radio unit (e.g., RU 120) that has been deployed and operated without dispatching. The small code 270 may include two small codes 272 for coordination using a diagram. When there is no 5G transmission, the first small code (installed at the FAPI hook of Table 1) detects idle time slots. The second small code (installed at the IQ sample hook of Table 1) samples interference during the detected idle time slot. By specifying multiple parameters in terms of which antenna ports and symbols, at what frequency (e.g., every receive time slot or every 10ms), and at what granularity (e.g., the average energy of the original IQ sample relative to each resource block) to collect IQ samples, the flexibility of the gate system allows adjustment of the fidelity and overhead of the spectrum analyzer as needed. Similar approaches can be used to implement other use cases that require radio channel telemetry data in different formats (e.g., wireless positioning and wireless channel estimation).

[0080] Another example use case for system 200 is external interference detection. Detecting external interferers in a cellular network (e.g., repeaters or jammers with improperly adjusted gain, stray radiation from electronic devices) is a difficult problem that conventionally involves sending experts and dedicated hardware to the site. The doorway codelet set 270 can define a spectrum analyzer to detect external interference through a 5G radio unit (e.g., RU 120) that has been deployed and operated without dispatching. The codelet set 270 can include two codelets 272 for coordination using a diagram. When there is no 5G transmission, the first codelet (installed at the FAPI hook of Table 1) detects idle time slots. The second codelet (installed at the IQ sample hook of Table 1) samples interference during the detected idle time slot. By specifying multiple parameters in terms of which antenna ports and symbols, at what frequency (e.g., every receive slot or every 10ms), and at what granularity (e.g., raw IQ samples versus average energy per resource block) to collect IQ samples, the flexibility of the gate system allows for tuning of the fidelity and overhead of the spectrum analyzer as required. A similar approach can be used to implement other use cases that require radio channel telemetry data in different formats (e.g., wireless positioning and wireless channel estimation).

[0081] Another example use case for the system 200 is network slicing. Many vertical and enterprise applications may rely on network slicing for stricter service QoS guarantees. For example, a robotic warehouse may require very accurate tail communication latency. Existing solutions based on the O-RAN service model allow a set of predefined slice scheduling policies that control scheduling at a granularity of 10 milliseconds. The gate system 200 can provide real-time slicing of arbitrary scheduling policies at a granularity of 0.5-10ms, which is faster than the capabilities of the standard RIC model. The gate system 200 can introduce a control hook (slice_scheduler()) at the beginning of the MAC downlink scheduling function of the vDU 130, which is responsible for the allocation of radio resources to the UE 110. The MAC downlink scheduling function can pass a structure used by the MAC scheduler as a hook context. The structure contains the scheduling state of the base station (e.g., the number of devices, the slices they belong to, buffer size, signal quality, etc.). A helper function 240 (allocate_resource_block()) dedicated to the slice_scheduler() hook can be configured to allow the small code to modify the allocation of radio resources for each slice. Using this hook, the small code 272 can implement a network slice scheduler that can be dynamically loaded.

[0082] Another example use case for the gate system 200 is real-time machine learning prediction. Real-time parameter prediction has been proposed for the RAN control loop because prediction freshness has a direct impact on network performance. Due to O-RAN RIC latency, conventional App 152 provides predictions with a lag of 10 milliseconds. Using the gate system 200, the ML predictor codelet can achieve prediction latency under 10ms. The first example ML predictor is an autoregressive integrated moving average (ARIMA) time series model for prediction of user signal quality. The second example ML predictor is a quantized decision tree for predicting the running time of signal processing tasks. More complex models (such as random forests) are more difficult to implement in gatelet codes using C code because such complex models result in a large number of bytecode instructions (>100K), making the verification process slow (>20 minutes), and the codelet may be unverifiable. In one aspect, the gate system 200 can support complex ML models in the form of a graph 236. The pre-trained, serialized complex model can be passed to the gate system 200 as a configuration graph and linked to the defined graph during codelet loading. The gate system 200 can parse the serialized model and rebuild the model in memory. The model can then be accessed by a small code for inference using a helper function 240 (e.g., janus_model_predict()). This approach can be extended to other commonly used ML models (e.g., long short-term memory (LSTM) networks).

[0083] On the one hand, the use of codelets presents additional technical issues that potentially impact the runtime of the host application. For example, running codelets inside a vRAN may require guarantees of memory safety and bounded runtime. The eBPF verifier can assert memory safety and termination. That is, if the codelet does not provably terminate (e.g., due to an unbounded loop), the codelet is rejected by the verifier. However, existing verifiers do not give sufficiently strict guarantees on the worst-case execution time of the codelet. This is important for the vRAN, which is very sensitive to any jitter.

[0084] A simple way to estimate the worst case execution time is to analyze the maximum number of eBPF instructions that a small code can execute. This information can be inferred by static analysis of the longest path for the small code in the case of considering bounded loops. However, converting the number of instructions to the expected runtime may be difficult because the actual runtime may depend on many factors, including CPU clock, memory and cache hierarchy, conversion of eBPF instructions to JIT code, etc. An additional challenge of the gate system 200 is the helper function, whose execution time can vary widely between functions and across parameter values. For example, although the helper function can be analyzed as a single instruction, the helper function can perform more complex operations and incur more significant overhead compared to several basic instructions that indicate that the maximum number of instructions per small code is not a good proxy for the maximum runtime.

[0085] In one aspect, the gate system 200 can enforce runtime through bytecode patching. To address the challenges of runtime control, the system 200 includes a bytecode patcher 212 configured to inject additional instructions into the eBPF bytecode that measure the codelet execution time while the codelet is running. If a threshold is exceeded, the additional instructions preempt the codelet.

[0086] Figure 5 5 is a diagram 500 showing patching of a checkpoint 510 into a codelet 520 to create a patch codelet 530. Figure 5 As shown, the initialization code 540 includes a first helper function 542 (e.g., mark_init_time()) that stores the current time (using the processor's read timestamp counter (rdtsc) instruction) in a thread local variable, and the initialization code 540 is added at the beginning of the patched codelet 530 (e.g., at line 1). The bytecode patcher 212 then introduces a checkpoint 510 in the selected location, which calls a second helper function 550 (e.g., run_limit_exceeded()) that checks the runtime of the codelet 530 that has elapsed since the first helper function 542 and compares the elapsed runtime to a threshold. If the elapsed runtime exceeds the threshold, the checkpoint 510 forces the codelet to exit and returns an error code indicating that the codelet was preempted. The runtime threshold can be specified for each codelet during loading of the codelet set. Checkpoint patching of the codelet allows the verifier 214 to verify the bytecode of the patched codelet 530 for security, which ensures that any modifications made by the patcher 212 do not affect the security of the loaded codelet 270.

[0087] The time check is implemented as helper functions 542 and 550 because the processor rdtsc instruction may not have a counterpart in the eBPF instruction set. A call to helper function 542 or 550 invalidates the eBPF registers r0-r5, which may store state from the normal codelet execution flow. The bytecode patcher 212 adds code within the initialization code 540 or checkpoint 510 that stores and reloads those registers (e.g., Figure 5 17 and 22 in ), thereby ensuring that the codelet 530 is still verifiable. The bytecode patcher 212 also updates all jump offsets to account for the injected instructions.

[0088] The problem for the bytecode patcher 212 is where to inject the checkpoint 510. Limiting the maximum number of instructions N between two consecutive checkpoints can reduce the impact of runtime jitter. However, each checkpoint incurs overhead, including calls to helper functions 550 ( Figure 5 Line 18 in the code to check if the small code 530 ( Figure 5 19) must be preempted, and the necessary registers saved and restored ( Figure 5 17 and 22 in ). On currently available processors, the overhead of checkpoints can be on the order of tens of nanoseconds per checkpoint. In order to keep the overhead low while limiting the number of instructions N, the bytecode patcher 212 can use a greedy algorithm in the following algorithm 1 to diffuse the checkpoints. Initially, after the call of the helper function has been marked by the vendor of the application as long-lasting and therefore potentially unsafe, the bytecode patcher 212 adds a checkpoint immediately if it is used frequently (line 2). Next, the bytecode patcher 212 uses the static analysis of the verifier 214 to enumerate (from shortest to longest) all simple paths from the first instruction to the last of the small code 520, as well as all loops. The static analysis can be, for example, a control flow graph of the small code 520. For each path, the bytecode patcher 212 adds a checkpoint 510 for every N instructions (lines 11 to 14). The bytecode patcher 212 considers checkpoints 510 that have already been added during the traversal of other paths. If a checkpoint 510 is found, the count of instructions for the new checkpoint is reset using the existing checkpoint 510 as the starting point (lines 8-9). Finally, at least one checkpoint is added at each loop, even if the distance is less than N (lines 17-19). This approach guarantees that a checkpoint 510 can always be reached every N instructions. For a finer level of control, the bytecode patcher 212 allows RAN vendors to use the RUNTIME_LIMIT_EXCEDED macro (which performs the same Figure 5 ) to insert checkpoints in the source code of their long-lasting helper functions 240.

[0089]

[0090] Even for a small number of checkpoints, the overhead may become significant (e.g., in small code with tight loops). To further reduce the overhead, the checkpointer 510 performs the check with a sampling frequency (one hit in M ​​checkpoints). The bytecode patcher 212 adds a 32-bit counter in the eBPF stack and performs the check only when the counter reaches a certain value ( Figure 5 Otherwise, the execution flow jumps back to the original bytecode instruction. In this way, it is guaranteed that at least one check is performed every M*N instructions.

[0091] In one aspect, the gate system 200 allows definition of arbitrary output modes for codelets. The modes are user defined and loaded with the codelets at runtime. The output data is transmitted over a network to a centralized controller. The gate system 200 uses protobuf to serialize the data. However, adding arbitrary modes may compromise security. Specifically, the gate system 200 must deal with two challenges.

[0092] The first challenge is to ensure that the small code cannot generate arbitrarily large serializable messages, which may violate memory safety. Protocol buffer messages are defined by the developer of the small code and may contain fields of variable size (e.g., repeated fields or strings). The verifier 214 does not know the actual size of the message at compile time, so the verifier 214 cannot statically verify the message size. To overcome this problem, the system 200 requires an upper limit for all protocol buffer modes that have fields of variable size in the message specification. The system 200 allocates a maximum message size for the C representation exposed to the small code and reports this size to the verifier 214. This allocated size allows static verification at the expense of slightly increased memory consumption (which is not a bottleneck in the vRAN system).

[0093] The second challenge is to ensure that incorrectly formatted messages cannot cause memory violations. Consider the following case, in which the codelet is configured to allocate a 30B memory block, project it as an example message, set the number of elements to 16, and call the protocol buffer encoder. Because the memory block is too small for 16 elements, the encoder will try to encode from memory outside the allocated block, which may cause a segmentation fault. In order to ensure memory access security, the verifier 214 can be configured to assert that the memory passed to the protocol buffer encoder is always equal to the maximum possible size. Errors like the above are still functionally incorrect and send garbled data, but do not violate security.

[0094] Figure 66 is a schematic diagram of an example of an apparatus 600 (e.g., a computing device) for implementing a gateway controller 210. The apparatus 600 can be an example of a RIC 150. The apparatus 600 can reside in a data center, which can be an edge data center. The apparatus 600 can be connected to other servers in the data center via a switch 640 and a wired connection. For example, the apparatus 600 can be connected to a gateway device 230, such as a vCU 140 or a vDU 130.

[0095] In an example, the apparatus 600 may include a processor 602 and / or a memory 604 configured to execute or store instructions or other parameters related to providing an operating system 606, which may execute one or more applications or processes, such as, but not limited to, the gate controller 210. For example, the processor 602 and the memory 604 may be separate components communicatively coupled via a bus (e.g., on a motherboard or other portion of a computing device, on an integrated circuit such as a system on a chip (SoC), etc.), components integrated with each other (e.g., the processor 602 may include the memory 604 as an onboard component), etc. The memory 604 may store instructions, parameters, data structures, etc. for use / execution by the processor 602 to perform the functions described herein.

[0096] In an example, the gate controller may include one or more of a bytecode patcher 212 , a bytecode verifier 214 , a JIT compiler 216 , a network API 218 , a metadata file generator 220 , and a data collector 222 .

[0097] Figure 7 is a flow chart of an example of a method 700 for executing a codelet within an application. For example, the method 700 may be performed by the apparatus 600 and / or one or more components of the apparatus 600 to implement a runtime threshold for execution of a codelet within an application.

[0098] At box 710, method 700 includes inserting a checkpoint into the eBPF bytecode of the third-party small code before verification of the third-party small code. In an example, the bytecode patcher 212, for example in combination with the processor 602, the memory 604, and the operating system 606, can insert the checkpoint 510 into the eBPF bytecode of the third-party small code 520 before verifying the third-party small code (e.g., the patched small code 530). For example, the third party can be a developer who writes the small code 520 for execution within an application provided by a supplier (second party) and executed by an operator (first party). In some embodiments, at subbox 712, box 710 can optionally include adding an initialization code 540 including a function call to a first function (e.g., an auxiliary function 542) at the beginning of the eBPF bytecode, the first function storing the time when the execution of the small code 530 starts. At sub-block 714, block 710 may optionally include adding checkpoint code (e.g., checkpoint 510) at various locations of the eBPF bytecode including function calls to the second function 550, the second function 550 comparing the execution time of the second function to the time when the execution of the stored codelet began. At sub-block 716, block 710 may optionally include updating the jump offsets to account for the inserted checkpoint code.

[0099] At block 720, method 700 includes loading the third-party codelet to the hook point of the application. In the example, network API 218, for example in conjunction with processor 602, memory 604, and operating system 606, can load the third-party codelet 272 to the hook point 234 of application 232. In some embodiments, where the application is executed in a container orchestration system, block 720 can include method 800 discussed below.

[0100] At block 730 , the method 700 includes executing the third-party codelet at the hook point of the application. In an example, the gate device 230 , for example in conjunction with the processor 602 , the memory 604 , and the operating system 606 , executes the third-party codelet 272 at the hook point 234 of the application 232 .

[0101] At box 740, method 700 includes determining the runtime of the third-party small code by the inserted checkpoint. In the example, the gate device 230 executing the application 232, for example in combination with the processor 602, the memory 604 and the operating system 606, can determine the runtime of the third-party small code 272 by the inserted checkpoint 510. In some embodiments, at sub-box 742, box 740 can optionally include storing the register state to the stack. For example, the initialization code 540 or the checkpoint 510 can store the register state to the stack. At sub-box 744, box 740 can optionally include calling the first function 542 or the second function 550 (for example, to compare the current time with the start of the execution of the small code). At sub-box 746, box 740 can optionally include restoring the register state from the stack. For example, the initialization code 540 or the checkpoint 510 can restore the register state from the stack. In some implementations, at sub-block 748 , block 740 may include skipping a function call to the second function (eg, sub-block 744 ) based on a specified execution frequency of the checkpoint code.

[0102] At block 750, method 700 includes terminating the third-party codelet in response to the runtime exceeding a threshold. In an example, gate device 230 executing codelet 272, such as in conjunction with processor 602, memory 604, and operating system 606, may terminate the third-party codelet 272 in response to the runtime exceeding a threshold.

[0103] Figure 8 800 is a flow chart of an example of a method 800 for loading a codelet into an application executing in a container. For example, method 800 may be performed by apparatus 600 and / or one or more components of apparatus 600 to load codelet 270 or codelet 272 into application 232. In some embodiments, method 800 may be used to load code into any application executing on a container orchestration system. Method 800 may be performed in conjunction with method 700. For example, method 800 may correspond to block 720.

[0104] At block 810 , the method 800 optionally includes modifying the application to include a plurality of hook points. In an example, the gateway controller 210 or the container API server 310 , for example in conjunction with the processor 602 , the memory 604 , and the operating system 606 , may modify the application 390 to include a plurality of hook points 392 .

[0105] At block 820, method 800 includes receiving a user configuration for a container image for an application, the user configuration including a header indicating dynamic loading of a codelet. In an example, container API server 310, e.g., in conjunction with processor 602, memory 604, and operating system 606, receives a user configuration (e.g., a YAML file) for a container image for application 390, the user configuration including a header indicating dynamic loading of a codelet.

[0106] At block 830, method 800 includes receiving a configuration map for a small code while the application is executing in the main container. In an example, container API server 310 (e.g., in conjunction with processor 602, memory 604, and operating system 606) can receive a configuration map for a small code while the application is executing in main container 380.

[0107] At block 840, method 800 includes installing code including functions associated with the hook point of the application to the container. In the example, variant hook 320, for example in conjunction with processor 602, memory 604, and operating system 606, can install code including functions associated with hook point 392 of application 390 (e.g., library 382) to container 380.

[0108] At block 850, the method 800 includes installing code including functions associated with the hook point of the application to the container. In an example, the variant hook 320, for example in conjunction with the processor 602, the memory 604, and the operating system 606, can install code including functions associated with the hook point 392 of the application 390 (e.g., the library 382) to the container 380. For example, at sub-block 842, the variant hook 320 can modify the entry point of the container 380 to execute a script to dynamically load the library 382 capable of executing a small code at the hook point 392.

[0109] At block 860, method 800 includes making a location available to a container containing compiled and verified codelets. In an example, mutation hook 320, for example in conjunction with processor 602, memory 604, and operating system 606, can make a location (e.g., codelet directory 386) containing compiled and verified codelets 360 available to container 380.

[0110] At block 870, method 800 includes detecting, during execution of the application, an annotation for the container, the annotation identifying the third-party codelet. In an example, controller 330, e.g., in conjunction with processor 602, memory 604, and operating system 606, can detect, during execution of the application, an annotation for the container, the annotation identifying the third-party codelet.

[0111] At block 880, the method 800 includes symbolically linking the third-party codelet indicated by the annotation to the hook point of the application. In the example, the sidecar container 372, for example in conjunction with the processor 602, the memory 604, and the operating system 606, can symbolically link the third-party codelet indicated by the annotation to the hook point 392 of the application 390.

[0112] Fig. 9 An example of a device 900 is shown including additional optional component details, such as Figure 6 As shown. In one aspect, the device 900 may include a processor 902, which may be similar to the processor 602 for performing processing functions associated with one or more of the components and functions described herein. The processor 902 may include a single group or multiple groups of processors or a multi-core processor. In addition, the processor 902 may be implemented as an integrated processing system and / or a distributed processing system.

[0113] The device 900 may also include a memory 904, which may be similar to the memory 604, such as for storing a local version of an operating system (or components thereof) and / or applications executed by the processor 902, such as the gate controller 210, the gate device 230, etc. The memory 904 may include the type of memory usable by a computer, such as random access memory (RAM), read-only memory (ROM), tapes, magnetic disks, optical disks, volatile memory, non-volatile memory, and any combination thereof.

[0114] Additionally, the device 900 may include a communication component 906 that provides for establishing and maintaining communications with one or more other devices, parties, entities, etc. utilizing hardware, software, and services as described herein. The communication component 906 may transmit communications between components on the device 900 and between the device 900 and external devices, such as devices located across a communication network and / or devices serially or locally connected to the device 900. For example, the communication component 906 may include one or more buses and may also include transmit chain components and receive chain components associated with wireless or wired transmitters and receivers, respectively, operable to interface with external devices.

[0115] In addition, the device 900 may include a data store 908, which may be any suitable combination of hardware and / or software that provides mass storage of information, databases, and programs used in conjunction with the aspects described herein. For example, the data store 908 may be or may include a data repository for an operating system (or components thereof), application programs, related parameters, etc., that are not currently executed by the processor 902. Furthermore, the data store 908 may be a data repository for the doorway controller 210, the doorway device 230, and / or one or more other components of the device 900.

[0116] The device 900 may optionally include a user interface component 910 that is operable to receive input from a user of the device 900 and further operable to generate output for presentation to the user. The user interface component 910 may include one or more input devices, including but not limited to a keyboard, a numeric keypad, a mouse, a touch-sensitive display, navigation keys, function keys, a microphone, a voice recognition component, a gesture recognition component, a depth sensor, a gaze tracking sensor, a switch / button, any other mechanism capable of receiving input from a user, or any combination thereof. In addition, the user interface component 910 may include one or more output devices, including but not limited to a display, a speaker, a tactile feedback mechanism, a printer, any other mechanism capable of presenting output to a user, or any combination thereof.

[0117] The device 900 may further include a gate controller 210 for providing execution of codelets within applications and gate devices 230, as described herein. In some embodiments, the gate controller 210 may be implemented as a container orchestration system 302, and the gate device 230 may be implemented as a workload pod 370.

[0118] As an example, any part of an element or any combination of elements can be implemented with a "processing system" including one or more processors. Examples of processors include microprocessors, microcontrollers, digital signal processors (DSPs), field programmable gate arrays (FPGAs), programmable logic devices (PLDs), state machines, gated logic, discrete hardware circuits, and other suitable hardware configured to perform various functions described throughout the present disclosure. One or more processors in a processing system can execute software. Software should be broadly interpreted as meaning instructions, instruction sets, codes, code segments, program codes, programs, subroutines, software modules, applications, software applications, software packages, routines, subroutines, objects, executable files, threads of execution, processes, functions, etc., whether referred to as software, firmware, middleware, microcode, hardware description language, or other.

[0119] Therefore, in one or more aspects, one or more of the functions described can be implemented in hardware, software, firmware, or any combination thereof. If implemented in software, the function can be stored on a computer-readable medium or encoded as one or more instructions or codes on a computer-readable medium. Computer-readable media include computer storage media. The storage medium can be any available medium that can be accessed by a computer. As an example and not limitation, such a computer-readable medium may include RAM, ROM, EEPROM, CD-ROM or other optical disk storage devices, disk storage devices or other magnetic storage devices, or any other medium that can be used to carry or store the desired program code in the form of an instruction or data structure and can be accessed by a computer. As used herein, disks and optical disks include optical disks (CDs), laser optical disks, optical disks, digital versatile disks (DVDs), and floppy disks, wherein disks generally reproduce data magnetically, and optical disks reproduce data optically with lasers. The above combination should also be included in the scope of computer-readable media.

[0120] The foregoing description is provided to enable those skilled in the art to practice the various aspects described herein. Various modifications to these aspects will be apparent to those skilled in the art, and the general principles defined herein may be applied to other aspects. Therefore, the claims are not intended to be limited to the aspects shown herein, but to the full range consistent with the claim language. Unless specifically stated otherwise, references to singular elements are not intended to mean "one and only one", but "one or more". Unless otherwise specifically stated, the term "some" refers to one or more. In addition, the term "or" is intended to mean an inclusive "or" rather than an exclusive "or". That is, unless otherwise stated, it is clear from the context that the phrase "X adopts A or B" is intended to mean any natural inclusive replacement. That is, the phrase "X adopts A or B" is satisfied by any of the following examples: X adopts A; X adopts B; or X adopts both A and B. In addition, the articles "a" and "an" as used in the present application and the appended claims should generally be interpreted as meaning "one or more", unless otherwise stated or clearly specified as a singular form from the context. Furthermore, nothing disclosed herein is intended to be dedicated to the public regardless of whether such disclosure is explicitly recited in the claims.A claim element is not to be construed as a means-plus-function unless the element is explicitly recited using the phrase "means for."

Claims

1. A method comprising: inserting a checkpoint into an extended Berkeley Packet Filter (eBPF) bytecode of the third-party codelet prior to verification of the third-party codelet; Executing the third-party small code at the hook point of the application; Determining the runtime of the third-party codelet through the inserted checkpoint; and In response to the runtime exceeding a threshold, the third-party codelet is terminated.

2. The method of claim 1, wherein the application is a virtualized radio access network (vRAN) network function and the small code is configured to control the vRAN function or derive network metrics.

3. The method according to claim 1 or 2, wherein inserting the checkpoint into the eBPF bytecode comprises: At the beginning of the eBPF bytecode, adding initialization code including a function call to a first function, the first function storing a time when execution of the codelet starts; as well as At various locations of the eBPF bytecode, checkpoint code is added including a function call to a second function, the second function comparing a time of execution of the second function with a time when execution of the stored codelet starts.

4. The method according to claim 3, wherein the various locations of the eBPF bytecode include the following locations: After a function call to a function marked as long-lasting by the application; For each logic path through the codelet, at every N instructions along the logic path from the last checkpoint; and Within each loop of the codelet, regardless of the distance from the last checkpoint.

5. The method according to claim 3 or 4, wherein the initialization code or the checkpoint code is configured to: Store register state to the stack; calling the first function or the second function; and The register state is restored from the stack.

6. The method of any one of claims 3 to 5, wherein the checkpoint code is configured to skip the function call to the second function based on a specified frequency of execution of the checkpoint code.

7. The method of any one of claims 3 to 6, wherein inserting the checkpoint into the eBPF bytecode further comprises updating jump offsets to account for the inserted checkpoint code.

8. The method according to any one of claims 1 to 7, further comprising statically verifying, by a verifier, that the eBPF bytecode including the checkpoint is safe.

9. The method according to any one of claims 1 to 8 further includes modifying the application to include multiple hook points, wherein the multiple hook points include the hook point for the third-party small code.

10. The method according to any one of claims 1 to 9, wherein the application is executed in a container of a container orchestration system, further comprising: Mounting code including functions associated with the hook point of the application to the container; During execution of the application, detecting an annotation for the container, the annotation identifying the third-party codelet; as well as The third-party codelet indicated by the annotation is symbolically linked to the hook point of the application.

11. The method of claim 10, wherein mounting the code including the function comprises: The entry point of the container is modified to execute a script to dynamically load a library capable of executing the codelet at the hook point.

12. The method of claim 10 or 11, further comprising making available to the container a location containing the compiled and verified codelet.

13. The method according to any one of claims 10 to 12 further comprises adding a component to the container, wherein the component is configured to: monitor whether a small code is executed at the hook point, and select which small code is executed at the hook point.

14. A system for injecting telemetry collection capabilities in an application running on a container orchestration system, comprising: The Application Programming Interface (API) server is configured to: receiving a user configuration for a container image for an application, the user configuration comprising a header indicating dynamic loading of a codelet, and receiving a configuration map for a codelet when the application is executed in a main container; A variant webhook, configured to modify the user configuration in response to the header: Add a sidecar container to the container image to run alongside the main container. Mounting one or more libraries into the main container, wherein the one or more libraries define functions associated with hook points within the application, modifying an entry point for the main container to run a script that loads the one or more libraries, and Mounting a directory for compiled and verified codelets into the main container; The node agent is configured as: monitoring the API server for the configuration map, Compile and verify the codelet, and placing the codelet in the directory; and The controller is configured as: An annotation to the user configuration is received via the API server, wherein the sidecar container is configured to symbolically link the codelet to a hook point within the application based on the annotation.

15. An apparatus comprising: a memory storing computer executable instructions; as well as At least one processor, the processor being configured to execute the instructions to implement the method of any one of claims 1 to 13.

Images