RIC SDK

Abstract

To provide a low-latency near-RT RIC, some embodiments separate the functionality of the RIC into several different components that run on different machines (e.g., running on VMs or pods) that run on the same or different host computers. Some embodiments also provide high-speed interfaces between these machines. Some or all of these interfaces operate in a non-blocking, lockless manner to ensure that critical operations of the near-RT RIC (e.g., data path processing) are not delayed due to multiple requests that stall one or more components. In addition, each of these RIC components also has an internal architecture designed to operate in a non-blocking manner so that one process of the component cannot block the operation of another process of the component. All of these low-latency features enable the near-RT RIC to act as a high-speed IO between E2 nodes and xApps.

Description

[Background technology]

[0001] In telecommunication networks, the Radio Access Network (RAN) performs more functions with each iteration of telecommunication standards, i.e., the 5G RAN performs various additional functions to enable the benefits of 5G compared to previous standards.

[0002] These RAN functions are often performed in base stations (e.g., cell towers), which may have limited computing power because they are located between user devices and the core network. Summary of the Invention

[0003] Some embodiments provide a new RAN Intelligent Controller (RIC) for telecommunications networks. For example, to provide a low-latency near-RT RIC, some embodiments separate the RIC functionality into several different components that run on different machines (e.g., on VMs or pods) that run on the same or different host computers. Some embodiments also provide high-speed interfaces between these machines. Some or all of these interfaces operate in a non-blocking, lockless manner to ensure that critical operations (e.g., data path processing) of the near-RT RIC are not delayed due to multiple requests that stall one or more components. Furthermore, each of these RIC components also has an internal architecture designed to operate in a non-blocking manner such that one process of the component cannot block the operation of another process of the component. All of these low-latency features enable the near-RT RIC to service high-speed IO between base station nodes (i.e., E2 nodes) and control plane applications (e.g., xApps).

[0004] In some embodiments, the near RT RIC includes datapath Pods, service Pods, and shared data layer (SDL) Pods. Part of the RIC's low latency architecture results from using different Pods to perform data IO, service, and SDL operations. As such, different resource allocation and management operations can be provided to each of these Pods based on the respective needs of the operations they perform. Also, in some embodiments, the RIC provides low latency messaging between its various Pods.

[0005] The Service Pod, in some embodiments, performs application (e.g., xApp) onboarding, registration, FCAPS (Fault, Configuration, Accounting, Performance, Security), and other services, and also provides services (e.g., metric collection, policy provisioning and configuration) to other RIC components. The SDL Pod implements the shared data layer of the near RT RIC. The SDL Pod of some embodiments also runs one or more service containers to perform one or more pre-processing or post-processing services on data stored in the SDL.

[0006] The data path pod performs data message transfer between base station components of the communications network and executes the control and edge applications of the network. In some embodiments, some or all of the data path services of the data path pod are incorporated into the data path thread and the control thread of the data path pod. In other embodiments, the data path services are incorporated into a data IO thread, multiple data processing threads (DPTs) and a control thread.

[0007] The control thread in some embodiments is the interface to the service pods and SDL pods for the datapath threads, while in other embodiments it is the interface to the service pods for the datapath threads (because the datapath threads can communicate directly with the SDL pods). The control thread in either of these approaches performs the slower control related operations of the datapath, while one or more datapath threads perform the faster IO operations of the datapath. The control thread in some embodiments interfaces with the service pods to receive configuration data for configuring its own operation as well as the operation of the datapath threads.

[0008] The embodiment of separating the data path thread into a data IO thread and multiple data processing threads further optimizes data IO by pushing the more computationally intensive operations of the data path thread to multiple data path processing threads, thereby allowing less computationally intensive operations to run on the data IO thread. Both of these optimizations are intended to ensure fast data path IO (without unnecessary delays) so that the near RT RIC can act as a high-speed interface between base station nodes and control and edge applications.

[0009] The foregoing summary of the invention is intended to serve as a brief introduction to some embodiments of the present invention. It is not intended to be an introduction or summary of all inventive subject matter disclosed in this document. The following detailed description and the drawings referenced in the detailed description further explain the embodiments described in the summary of the invention as well as other embodiments. Therefore, a thorough review of the summary of the invention, the detailed description and the drawings is necessary to understand all embodiments described by this document. Furthermore, since the subject matter described in the claims can be embodied in other specific forms without departing from the concept of the subject matter, the subject matter is not limited by the details illustrated in the summary of the invention, the detailed description and the drawings, but rather is defined by the appended claims. [Brief description of the drawings]

[0010] The novel features of the invention are set forth in the appended claims, however, for purposes of illustration, certain embodiments of the invention are set forth in the following drawings.

[0011] [Figure 1] 1 illustrates an example of an O-RAN architecture according to some embodiments.

[0012] [Diagram 2] 1 illustrates details of both non-real-time and near-real-time RIC components according to some embodiments.

[0013] [Diagram 3] 2 shows a more detailed diagram of a MAC Control Assist of some embodiments.

[0014] [Figure 4] 1 illustrates a more detailed diagram of a user-level tracer of some embodiments.

[0015] [Diagram 5] FIG. 1 illustrates another diagram of the O-RAN architecture of some embodiments, showing a more detailed view of a near real-time RIC.

[0016] [Figure 6] 1 illustrates the deployment of a RIC SDK on a machine that runs a control plane application, in some embodiments.

[0017] [Figure 7] Some embodiments are shown deploying several RICs to execute on several host computers to implement a distributed near-RT RIC that includes the RIC components shown in FIGS.

[0018] [Figure 8]It shows a RIC running on one host computer along with two machines running two control plane applications.

[0019] [Figure 9] It shows two RICs running on two host computers, along with two control plane applications and two machines running the RIC SDK.

[0020] [Figure 10] 1 shows a RIC running on a first host computer and connecting two control plane applications running on two machines that run on two other host computers.

[0021] [Figure 11] Indicates a RIC running on a first host computer and connecting two control plane applications running on two machines, one running on the first host computer and the other running on another host computer.

[0022] [Figure 12] 1 illustrates examples of APIs of different standards supported by the distributed near-RT RIC platform of some embodiments.

[0023] [Figure 13] An embodiment is shown in which the SDL cache is part of each RIC SDK running on the same machine as its control plane.

[0024] [Figure 14] Examples of control or edge applications are shown that have pass-through access to a hardware accelerator in a host computer to perform some or all of their operations.

[0025] [Figure 15]1 illustrates a process performed in some embodiments in response to an O-RAN component instructing a CP or edge application to perform an operation that requires the application to use a hardware accelerator of a host computer.

[0026] [Figure 16] Illustrates an application that performs actions based on data from the E2 node.

[0027] [Figure 17] Another example of a control or edge application having pass-through access to a hardware accelerator in the host computer to perform some (or all) of their operations is shown.

[0028] [Figure 18] Further examples of CP or edge applications that perform some or all of their operations with pass-through access to a hardware accelerator in the host computer are shown.

[0029] [Figure 19] Some embodiments illustrate a process used to deploy O-RAN applications with direct pass-through access to a hardware accelerator on a host computer.

[0030] [Figure 20] 1 illustrates an example of a CP or edge application having pass-through access to a virtual hardware accelerator defined by a hypervisor running on a host computer.

[0031] [Figure 21] 1 shows an example of a near-RT RIC, with several components running on several different machines.

[0032] [Figure 22], [Diagram 23] FIG. 21 shows different examples for deploying components near a near RT RIC.

[0033] [Figure 24] , [Diagram 25] 2 shows another example of a near-RT RIC.

[0034] [Figure 26] 1 shows an example of a RIC data pass pod.

[0035] [Figure 27] 1 illustrates the processing that a datapath thread performs to process a subscription request from an xApp in some embodiments.

[0036] [Figure 28] 1 illustrates the processing that the Data IO Thread and DPT perform in some embodiments to process data messages from an E2 node that are to be received by one or more xApps.

[0037] [Figure 29] 1 illustrates the processing that the Data IO Thread and DPT perform in some embodiments to process a data message from an xApp to be sent to an E2 node.

[0038] [Diagram 30] 13 illustrates an example process that a Data IO thread uses to assign E2 nodes to a DPT in some embodiments.

[0039] [Diagram 31] 1 shows a distributed near-RT RIC implemented by an active RIC and a standby RIC.

[0040] [Diagram 32]In some embodiments, interfaces between a near RT RIC and an E2 node and between a near RT RIC and an xApp pod are shown.

[0041] [Diagram 33] 13 shows E2AP message processing of a data path pod of a near RT RIC.

[0042] [Diagram 34] 1 illustrates a RIC instance of some embodiments with an SDL pod.

[0043] [Diagram 35] 1 conceptually illustrates an electronic system in which some embodiments of the present invention may be implemented. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0044] In the following detailed description of the invention, numerous details, examples and embodiments of the invention are set forth and explained. However, it will be apparent and clear to one skilled in the art that the invention is not limited to the described embodiments, and that the invention may be practiced without some of the specific details and examples described.

[0045] Today, there is a drive to push the Radio Access Network (RAN) of telecommunication networks (e.g., cellular networks) implemented as O-RAN to standards that enable interoperability of RAN elements and interfaces. FIG. 1 illustrates an example of an O-RAN architecture 100, according to some embodiments. The O-RAN architecture 100 includes a service management and orchestration framework 110 having a non-real-time RIC 105, a near-real-time RAN intelligent controller 115, an open control plane centralized unit (O-CU-CP) 120, an open user plane centralized unit (O-CU-UP) 125, an open distributed unit (O-DU) 130, an open radio unit (O-RU) 135, and an O-Cloud 140. The O-CU-CP 120, the O-CU-UP 125, and the O-DU 130 may hereinafter be collectively referred to as managed functions 120-130.

[0046] As defined in the standard, the SMO 110 of some embodiments includes a unified fabric that allows the SMO to connect to and manage the RIC 115, the managed functions 120-130, and the O-Cloud 140 via open interfaces 150. Unlike these elements, the O-RU 135, in some embodiments, is not managed by the SMO 110, but instead is managed by the O-DU 130, as indicated by dashed line 160. In some embodiments, the O-RU 135 processes and transmits radio frequencies to the O-DU 130.

[0047] In some embodiments, the managed functions 120-130 are logical nodes that each host a set of protocols. In accordance with the O-RAN standard, for example, the O-CU-CP 120 in some embodiments includes protocols such as Radio Resource Control (RRC) and the control plane portion of the Packet Data Convergence Protocol (PDCP), and the O-CU-UP 125 includes protocols such as the Service Data Adaptation Protocol (SDAP) and the user plane portion of the Packet Data Convergence Protocol (PDCP).

[0048] Each of the two RICs is adapted for a specific control loop and latency requirements. The near real-time RIC 115 provides programmatic control of the Open Centralized Unit (O-CU) and the Open Distributed Unit (O-DU) with time cycles of 10 ms to 1 second. On the other hand, the non-real-time RIC (non-RT RIC) 105 provides higher layer policies that can be enforced in the RAN via a near-RT RIC or a direct connection to a RAN node. The non-RT RIC is used for control loops of 1 second or longer. Each RIC 105 or 115 serves as a platform to run RAN control applications. These applications can be developed by third-party suppliers different from the RIC vendor. These applications are called "xApps" (for the near-RT RIC 115) and "rApps" (for the non-RT RIC).

[0049] The near real-time RIC 115, in some embodiments, is a logical aggregation of several functions that utilize data collection and communication via interface 155 to control the managed functions 120-130. In some embodiments, the non-real-time RIC 105 uses machine learning and model training to manage and optimize the managed functions 120-130. The near-real-time RIC in some of these embodiments also uses machine learning.

[0050] In some embodiments, the O-Cloud 140 is responsible for creating and hosting the Virtual Network Functions (VNFs) used by the RIC 115 and managed functions 120-130. In some embodiments, the DU is responsible for slot-by-slot decisions of user scheduling and includes a RAN scheduler that performs MAC control assist and user-level tracing. To increase the computing power available in the cloud (i.e., compared to base stations that typically perform RAN functions), the RIC is implemented in one or more public and / or private cloud data centers and implements an improved cloud-defined RAN scheduler in the cloud, thereby offloading these MAC control assist and user-level tracing functions from the DU to the RIC. An interface 155 in some embodiments allows the RAN to provide inputs to functions in the RIC and, in at least some embodiments, receive outputs computed by these functions in the RIC.

[0051] 2 shows a detailed diagram of the components of both the non-real-time RIC 201 and the near-real-time RIC 202. Each of the RICs 201 and 202 includes a respective set of analysis functions 210 and 212 and a respective set of optimization functions 214 and 216, each of which are shown with dashed lines to indicate that they are existing components. In addition to these existing components, the near-real-time optimization function 216 includes two new components, a MAC control assister 220 and a user level tracer 222, which are shown with solid lines to visually distinguish them from the existing components. In some embodiments, these components are part of a larger MIMO component (e.g., along with the pairer and precoder for a MU-MIMO UE).

[0052] In some embodiments, the MAC control assister 220 can include various functions such as: (1) user equipment (UE) specific beamforming weight calculation based on UL SRS channel signal reception, (2) UE radio frequency (RF) condition prediction, and (3) multi-user, multiple input, multiple output (MU-MIMO) pairing proposals to the MAC scheduler based on UE specific beams. For each of these functions, some embodiments expose a report interface (providing input data for the function from the DU to the RIC) and a control interface (providing output data for the function from the RIC to the DU).

[0053] The user level tracer 222, in some embodiments, generates L1 / L2 / L3 level information related to user configuration and traffic performance. This trace data can be used as input to various control algorithms such as MAC scheduler, parameter settings, etc. The user level tracer 222 can include trace operations that can include (i) tracking user behavior within a cell, (ii) tracking user RF conditions, (iii) tracking user data traffic performance within different layers (MAC, Radio Link Control (RLC), Packet Data Convergence Protocol (PDCP)), and (iv) user RF resource consumption.

[0054] 3 shows a more detailed diagram of some embodiment MAC control assistor 300. As illustrated, MAC control assistor 300 includes a UE-specific beamforming weight calculator (BFWC) 310, a UE RF condition predictor 320, and a MU-MIMO pairing suggester 330. The UE-specific BFWC 310 in some embodiments is based on UL SRS channel signal reception. In some embodiments, the MU-MIMO pairing suggester 330 is for a UE-specific beam-based MAC scheduler.

[0055] Each of the components 310-330 of the MAC control assister 300 includes an uplink and a downlink as described. For the UE specific BWC function, some embodiments expose a report interface for an uplink sounding reference signal (UL SRS) channel response matrix, which is an input to a weight calculation function, and a control interface for a UE specific beamforming weight matrix. For the UE RF condition predictor function, some embodiments expose a report interface for a downlink (DL) channel state report, which is an input to the RF condition predictor, and a control interface for predicted DL channel conditions (e.g., including DL SINR, PMI, and rank) for the next scheduling window. For the MU-MIMO pairing proposal function, some embodiments expose a report interface for a UE specific beamforming weight matrix, which is an input to the UE pairing proposal function, and a control interface for UE pairing proposal and SINR impact evaluation.

[0056] FIG. 4 shows a more detailed diagram of a user level tracer 400 of some embodiments. The tracer 400, in some embodiments, includes multiple uplinks 410 and multiple downlinks 415 for performing tracing operations. These operations generate L1 / L2 / L3 level information related to user settings and traffic performance. This trace data can be used as input to various control algorithms such as MAC scheduler, parameter settings, etc. These tracing operations can 1) trace user behavior within a cell, 2) track user RF conditions, 3) track user data traffic performance at different layers (MAC, RLC, PDCP), and 4) track user RF resource consumption.

[0057] For these tracing operations, some embodiments expose reporting interfaces for the DU and / or CU to provide various metrics for user-level tracing operations. These metrics may include selected RRC messages, MAC / RLC / PDCP traffic volume and performance, RF conditions, and RF resource consumption. In some embodiments, messages over these interfaces to the RIC are triggered based on user behavior and / or periodic reports (e.g., for traffic performance and RF conditions / resource consumption).

[0058] The tracing operations may track the various user data noted above and provide this information back to the RAN or to other control algorithms (e.g., other algorithms running in the RIC). For example, these algorithms may perform analysis on user data performance from user-level tracing operations, determine that certain performance is insufficient, and modify how the RAN is handling the user traffic. Examples of control algorithms that may benefit from user-level tracing in some embodiments include (1) traffic steering, (2) quality of service (QoS) scheduling optimization, (3) user configuration adjustment, and (4) user behavior anomaly detection.

[0059] For all of the operations described in Figures 3-4 (i.e., the MAC scheduler functions and user-level tracking operations), the increased computational power available at the RIC in the cloud allows for more complex computations without excessive latency. For example, some or all of these operations can be performed at the RIC using machine learning (e.g., using machine-trained networks).

[0060] FIG. 5 illustrates another diagram of an O-RAN architecture of some embodiments, showing a more detailed view of a near-real-time RIC. The architecture 500 includes a SMO 505 having a non-real-time RIC 510, a distributed near-real-time RIC 515, and an E2 node 520 (e.g., an O-DU and / or an O-CU node). The distributed near-real-time RIC 515 includes a messaging infrastructure 540, a set of services (e.g., 550, 552, 554, and 556), a shared data layer 560, a database 570, and a set of termination interfaces (e.g., 580, 582, and 584). As shown, a set of embedded apps (e.g., 530, 532, 534, etc.) use this distributed near-RT RIC. As described further below, in some embodiments, the distributed near-RT RIC 515 is realized by multiple RICs running on multiple host computers.

[0061] As shown, the set of services includes conflict migration services 550, app subscription management services 552, management services 554, and security services 556. Additionally, the set of termination interfaces includes an O1 termination interface 580 that connects the SMO to the near real-time RIC, an A1 termination interface 582 that connects the non-real-time RIC to the near real-time RIC, and an E2 termination interface 584 that connects the E2 node to the near real-time RIC. Each of the apps, in some embodiments, represents various functions of the RIC that use data sent from the E2 node 520. For example, app 530 corresponds to the UE specific BFWC 310 of the MAC control assistor 300, app 532 corresponds to the UE RF condition predictor 320 of the MAC control assistor 300, etc.

[0062] In some embodiments, the purpose of the framework 500 is to offload computationally intensive near real-time functions and provide the results back to the O-DU (e.g., via the E2 interface to the E2 node 520). In some embodiments, the results can be used to assist or enhance real-time decisions at the MAC layer. Three example use cases of the MAC control assist framework are described below, each example specific to a different component of the MAC control assister (e.g., UE specific BFWC, UE RF condition predictor, MU-MIMO pairing suggester), and one example use case of a user-level tracer.

[0063] A first use case example is specific to UE specific beamforming weight calculation based on a UL SRS signal reception component of the MAC control assist framework (e.g., component 310 of MAC control assistor 300). In some embodiments of this use case, the input metrics can include multiple options based on the UL SRS, such as raw SRS received data, and the SRS channel response matrix from channel estimation.

[0064] In some embodiments, the algorithm for generating the output metric evaluates the optimal beamforming weights to reach the user. Some embodiments use traditional signal processing algorithms based on channel models. Alternatively, or in combination, machine learning based algorithms are used that utilize raw data inputs, which require feedback from the DUs in the E2 node 520.

[0065] In some embodiments, the output metric resulting from the algorithm includes a beamforming weight matrix for the user. In some embodiments, the BFW may also be mapped to a beam index from a pre-designed beam set. The DU in some embodiments uses the matrix to control the MIMO antenna array gain / phasing in the RU (e.g., O-RU 135 in architecture 100) for transmitting and receiving user data.

[0066] A second use case example is specific to the UE RF condition predictor component of the MAC control assist framework (e.g., component 320 of MAC control assistor 300). In this second use case, according to some embodiments, the input metrics include at least channel reports from the UE, such as wideband or sub-band CQI / PMI / RI in the DL case, or SRS in the UL case. The input metrics of some embodiments may also choose to include assistance information such as UE distance, UE positioning, etc.

[0067] In some embodiments, the app algorithms for this second use case are intended to predict the RF conditions of the UE based on observations. Some embodiments utilize traditional signal processing algorithms based on channel and mobility models. Alternatively, or in combination, some embodiments also use machine learning based algorithms that use data inputs and potentially other factors such as site layout (requiring feedback from the DU).

[0068] Power metrics for this use case, in some embodiments, include the user's predicted channel condition for the next scheduling window, as well as the predicted downlink and uplink SINR, precoding matrices (e.g., if applicable), and SU-MIMO layers. In some embodiments, these power metrics are used by the DU for user link adaptation of PDCCH / PDSCH / PUSCH transmissions.

[0069] The third use case is specific to the MU-MIMO pairing proposal or MAC scheduler component (e.g., component 330 of MAC control assistor 300). Input metrics for this example use case include, in some embodiments, at least a UE-specific BFW matrix and a UE RF condition estimate. In addition to the UE-specific BFW matrix and the UE RF condition estimate, some embodiments may also include assisting metrics, such as user data demand, as input metrics.

[0070] The app algorithm for this use case, in some embodiments, is intended to identify users that can be paired for MU-MIMO operation. For example, some embodiments of the third use case use traditional signal processing algorithms based on information theory and cross-channel covariance estimation. Alternatively, or in combination, some embodiments use machine learning based algorithms that use data inputs, again requiring feedback from the DU.

[0071] In some embodiments, the power metric for this third use case may include UE pairing proposals and impact assessments for SINR and SU-MIMO layers. Additionally, the DU of some embodiments uses the power metric to select users for RF scheduling and to determine transmission efficiency.

[0072] An example use case for a user-level tracer can include QoS scheduling optimization with the goal of adjusting a user's scheduling priorities for RF resources to optimize quality of service. Inputs for some embodiments of this use case can include quality of service targets from user subscriptions. In some embodiments, user-level tracing includes (1) tracking user RF conditions, (2) tracking user data traffic performance at different layers (e.g., MAC / RLC / PDCP), and (3) tracking user RF resource consumption.

[0073] In some embodiments, the app algorithm is based on QoS targets and observed user traffic performance and can be used to determine that a user's resource allocation is insufficient. The algorithm format can be logic-based or machine learning-based in some embodiments. In some embodiments, the output includes recommendations issued to the MAC scheduler to adjust traffic priorities or link adaptation to improve performance.

[0074] On each machine (e.g., each VM or pod) that runs a control plane application, some embodiments configure a RIC SDK to act as an interface between the control plane application on the machine and a set of one or more elements of the RAN. In some embodiments, the RIC SDK provides a set of APIs (e.g., a framework) through which applications can communicate with distributed near real-time (RT) RICs implemented by two or more near real-time RICs. Examples of such applications, in some embodiments, include xApps and other control plane and edge applications. In O-RAN, xApps perform control plane, monitoring, and data processing operations. The following discussion of Figures 6 and 8-20 refers to control plane applications (e.g., 615, 815, 820, 915, 920, etc.). These control plane applications, in some embodiments, are xApps in the O-RAN system.

[0075] FIG. 6 illustrates the deployment of a RIC SDK 605 on a machine 610 that executes a control plane application 615 in some embodiments. As described, one or more machines 610 operate on each of several host computers 607 in one or more data centers. In some embodiments, the RIC SDK 605 on each machine 610 includes a set of network connectivity processes that establish network connectivity to a set of RAN elements (e.g., E2 node 520, shared data layer 560, management services 554, SMO 505, etc.) for control plane applications. The use of the RIC SDK processes avoids the need to perform network connectivity operations in the control plane applications on the machine. In some embodiments, the set of network connectivity processes in each RIC SDK on each machine establishes and maintains network connectivity between the machine and a set of RAN elements used by the machine's control plane applications and handles data packet transmissions between the set of RAN elements for the control plane applications.

[0076] The control plane applications on each machine communicate with a set of RAN elements via high-level APIs 620 that the RIC SDK translates to low-level APIs 625. In some embodiments, at least a subset of the low-level API calls 625 are defined by a standards development body. Also, in some embodiments, the high-level APIs 620 are written in a high-level programming language (e.g., C++), and the low-level API calls include low-level calls to establish and maintain network connections and pass data packets over those connections.

[0077] The set of RAN elements that the RIC SDK connects to the control plane applications on the machine includes RAN elements produced or developed by different RAN vendors and / or developers. These RAN elements include the CU 630 and DU 635 of the RAN in some embodiments. The SDK also communicates with the CU and DU via an E2 interface defined in a low-level standard, while the control plane applications on the machine communicate with the CU and DU via the RIC SDK using high-level API calls. In some embodiments, the high-level API calls define the E2 interface operations at a high-level application layer that does not include low-level transport or network operations.

[0078] In combination or alternatively, the set of RAN elements that the RIC SDK interfaces with the control plane application 615 on its machine 610 includes the network elements of the RIC. Again, these network elements in some embodiments include RAN elements produced and / or developed by different RAN vendors and / or developers. These RIC elements in some embodiments include a shared data layer 560, data path input / output (I / O) elements, and in some embodiments application and management services 552 and 554. FIG. 7 illustrates that some embodiments deploy several near-RT RICs 705 to operate on several host computers to implement a distributed near-RT RIC 700 that includes the RIC components shown in FIGS. 5 and 6. In some embodiments, one RIC 705 operates on each host computer that also runs the control plane application 615. In other embodiments, the control plane application 615 can operate on a host computer that does not run a RIC. For example, in some embodiments, one or more control plane applications run on one or more host computers that have a graphics processing unit (GPU), but the RIC does not run on such host computers because it does not require the processing power of the GPU.

[0079] The RIC SDK also connects its control plane applications to other control plane applications that do not run on other machines via the distributed near-RT RIC. In other words, the RIC SDK and the distributed near-RT RIC in some embodiments act as a communication interface between the control plane applications. In some embodiments, different control plane applications are developed by different application developers that use a common set of RIC APIs to communicate with each other via the distributed near-RT RIC. In some of these embodiments, the distributed near-RT RIC adds one or more parameters to the API calls when forwarding the API calls from one control application to another.

[0080] 8 through 11 show examples of RIC architectures in which the RIC SDK and distributed near-RT RICs establish a communication interface between control plane applications. These architectures are mutually exclusive in some embodiments, while other embodiments use a combination of two or more of these architectures. FIG. 8 shows a RIC 800 running on one host computer 805 along with two machines 810 and 812 on which two control plane applications 815 and 820 run. The RIC 800 receives API calls from a CP application 815 via RIC SDKs 802 and 804 running on machines 810 and 812, forwards the API calls to the CP application 820, and passes responses to these API calls from the second CP application 820 to the first CP application 815. It also passes API calls from the second CP application 820 to the first CP application 815, and responses from the first CP application 815 to the second CP application 820.

[0081] 9 shows two RICs 900 and 901 running on two host computers 905 and 907, and two machines 910 and 912 on which two control plane applications 915 and 920 and two RIC SDKs 902 and 904 run. As shown in the figure, API calls from the first CP application 915 to the second CP application 920 are routed through the first RIC SDK 902, the first RIC 900, the second RIC 901, and the second RIC SDK 904. The second CP application's responses to these API calls to the first CP application 915 traverse the reverse path from the second RIC SDK 904, the second RIC 901, the first RIC 900, and the first RIC SDK 902.

[0082] API calls from the second CP application 920 to the first CP application 915 are forwarded via the second RIC SDK 904, the second RIC 901, the first RIC 900, and the first RIC SDK 902, while responses to these API calls from the first CP application 915 to the second CP application 920 are forwarded via the first RIC SDK 902, the first RIC 900, the second RIC 901, and the second RIC SDK 904.

[0083] 10 shows a RIC 1000 running on a first host computer 1005 to connect two control plane applications 1015 and 1020 running on two machines 1010 and 1012 running on two other host computers 1006 and 1007. The RIC 1000 receives API calls from a CP application 1015 via RIC SDKs 1002 and 1004 running on machines 1010 and 1012, forwards the API calls to the CP application 1020, and passes responses to these API calls from the second CP application 1020 to the first CP application 1015. It also passes API calls from the second CP application 1020 to the first CP application 1015, and responses from the first CP application 1015 to the second CP application 1020.

[0084] 11 shows a RIC 1100 running on a first host computer 1105 to connect two control plane applications 1115 and 1120 running on two machines 1110 and 1112, one of which runs on host computer 1105 and the other on host computer 1106. Through RIC SDKs 1102 and 1104 running on machines 1110 and 1112, RIC 1100 receives API calls from CP application 1115, forwards the API calls to CP application 1120, and passes responses to these API calls from second CP application 1120 to first CP application 1115. Through these SDKs 1102 and 1104, RIC 1100 also passes API calls from second CP application 1120 to first CP application 1115, and responds from first CP application 1115 to second CP application 1120.

[0085] 12 illustrates examples of different standard APIs supported by the distributed near-RT RIC platform of some embodiments. As illustrated, the distributed near-RT RIC platform 1200 of some embodiments uses E2, O1, and A1 interfaces defined by the O-RAN standards body. It uses the E2 API to communicate with E2 O-RAN nodes such as O-CU-CP 1202, O-CU-UP 1204, and O-DU 1206. It also uses the A1 API to communicate with the non-real-time RIC platform 1208 and the O1 API to communicate with the SMO 1210.

[0086] For each of the E2, A1, and O1 time APIs, the RIC SDK 1215 provides a high-level corresponding API for the control plane application 1220 to communicate with the E2 nodes 1202-1206, the non-real-time RIC platform 1208, and the SMO 1210 using the RIC SDK and the distributed near-RT RIC platform. In FIG. 12, the higher level corresponding APIs for the E2, O1, and A1 interfaces are designated using prime signs as E2' API calls, O1' API calls, and AE API calls. These high level corresponding APIs are APIs that are not specified by a standards body, but that the RIC SDK and / or the distributed near-RT RIC translate into standard API calls.

[0087] FIG. 12 also shows several internal RIC APIs to enable control plane applications 1220 to communicate with each other via the RIC SDK and the distributed near-RT RIC, and with one or more elements of the distributed near-RT RIC (e.g., shared data layer 560, data path input / output (I / O) elements, and application and management services 552 and 554).

[0088] Enabling APIs are APIs that are used in some embodiments to allow control plane applications 1220 to communicate with each other. As discussed above with reference to Figures 8-11, in some embodiments, these APIs pass through the distributed near-RT RIC. In other embodiments, these APIs allow the RIC SDKs of the control plane applications to communicate directly with each other without passing through other components of the distributed near-RT RIC. Thus, Figure 12 includes a dashed, double-headed arrow between the RIC SDKs 1215 of the two control plane applications 1220 to indicate that, in some embodiments, the RIC SDKs 1215 of these applications communicate directly with each other.

[0089] The enablement APIs in some embodiments include a registration API, a service discovery API, and an inter-app communication API. The registration API is used by applications 1220 (e.g., xApps) to introduce themselves to other applications 1220 by providing their network identifiers (e.g., their network addresses and available L4 ports) and providing their capabilities (e.g., performing channel prediction). The service discovery API allows control plane applications 1220 (e.g., xApps) to query a service directory (e.g., distributed near-RT RIC) for other control plane applications (e.g., other xApps) that offer specific services. The inter-app communication API allows control plane applications to communicate with each other to pass data or request specific actions.

[0090] Some embodiments deploy an SDL cache on the same host computer as the control plane applications and use the cache to process at least a subset of the SDL storage access requests of the control plane applications. In some embodiments, the control plane applications and the SDL cache run on a machine that runs on the host computer. In other embodiments, the SDL cache runs on the same host computer, but external to the machine on which the control plane applications run. In some of these embodiments, multiple control plane applications running on the same host computer use a common SDL cache on that host computer.

[0091] The SDL cache, in some embodiments, is part of the RIC running on the same host computer as the control plane. In other embodiments, the SDL cache is part of the RIC SDK running on the same machine as the control plane application. In any of these embodiments, a synchronization process in the RIC or RIC SDK synchronizes the data stored in the SDL cache with the data stored in the SDL storage.

[0092] In some embodiments, the SDL storage runs on a different host computer than the host computer on which the control plane application runs, while in other embodiments, at least a portion of the SDL storage runs on the same host computer on which the control plane application runs. Also, in some embodiments, if the RIC SDK cannot process the SDL access request through the SDL cache, the RIC or RIC SDK forwards the SDL access request from the control plane application to the SDL storage. For example, if the SDL cache does not store data requested by the control plane application, the RIC or RIC SDK cannot process the SDL access request through the SDL cache.

[0093] 13 illustrates an embodiment in which the SDL cache 1302 is part of each RIC SDK 1300 that runs on the same machine 1305 as its control plane application 1310. As illustrated, the RIC SDK 1300 includes a query manager 132 that processes SDL requests from the CP application 1310 and a synchronization service 1327 that synchronizes data stored in the SDL cache with data stored in the SDL storage 1350 of the SDL 1355 of the distributed near-RT RIC 1330. In this example, the SDL storage 1350 runs on a different host computer than the host computer on which the control plane application 1310 runs. However, in other embodiments, at least a portion of the SDL storage 1350 runs on the same host computer on which the control plane application 1310 runs.

[0094] When the control plane application 1310 uses a high-level API call to read or write data to the SDL storage, the query manager 1325 of the RIC SDK 1300 first determines whether the data record being read or written is stored in the SDL cache 1302. If so, the query manager 1325 reads or writes from this record. If the operation is a write operation, the synchronization service 1327 writes the new data to the SDL storage 1350 in real time or on a batch basis. On the other hand, if the query manager 1325 of the RIC SDK 1300 determines that the data record being read or written is not stored in the SDL cache 1302, it passes the API call to the SDL layer of the distributed near-RT RIC to perform the requested read or write operation. When passing this API call, the RIC SDK 1300 modifies the format of the call and / or the parameters supplied with the call in some embodiments.

[0095] Some embodiments provide various methods for offloading operations in an Open Radio Access Network (RAN) onto a control plane (CP) or an edge application running on a host computer with a hardware accelerator in a software-defined data center (SDDC). For example, in a CP or edge application running on a machine running on a host computer with a hardware accelerator, the method of some embodiments receives data from an O-RAN E2 unit that must perform an operation. The method uses a driver of the machine to directly communicate with the hardware accelerator and instruct the hardware accelerator to perform a set of calculations related to the operation. This driver allows communication with the hardware accelerator to bypass an intermediate set of drivers running on the host computer between the machine driver and the hardware accelerator. Through this driver, the application of some embodiments receives the computation results, which are then provided to one or more O-RAN components (e.g., the E2 unit that provided the data, another E2 unit, or another xApp).

[0096] 14-20 show several different embodiments for offloading O-RAN operations to a CP or edge application with pass-through access to a hardware accelerator in the host computer. Examples of such hardware accelerators include graphical processing units (GPUs), field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), and structured ASICs.

[0097] 14 illustrates examples of CP or edge applications 1402 that have pass-through access to a hardware accelerator 1450 of a host computer 1410 to perform some or all of their computations. As illustrated, each application 1402 runs on a pod 1404 that has an accelerator driver 1412 that has direct pass-through access to the accelerator 1450 of the host computer 1410. Each pod 1404 runs within (i.e., runs on) a VM 1406 that runs on a hypervisor 1408 of the host computer.

[0098] In some embodiments, a pod is a small, deployable computing unit that can be created and managed in Kubernetes. A pod contains a group of one or more containers with shared storage and network resources, and a specification of how the containers should run. In some embodiments, the contents of a pod are always co-located, scheduled together, and run in a shared context. A pod models an application-specific logical host computer. A pod contains one or more application containers that communicate with each other. In some embodiments, a pod's shared context is a collection of operating system namespaces (e.g., Linux cgroups). Within the context of a pod, further sub-isolation may be applied to individual applications.

[0099] Each pod's accelerator driver 1412 has direct access to the hardware accelerator 1450, which bypasses the hardware accelerator drivers 1414 and 1416 of the VM 1406 and the hypervisor 1408. In some embodiments, the hypervisor 1408 runs on an operating system (not shown) of the host computer 1410. In these embodiments, each pod's accelerator driver 1412's direct access to the hardware accelerator 1450 also bypasses the operating system's hardware accelerator driver.

[0100] To communicate with the hardware accelerator, each application 1402 of some embodiments communicates through a RIC SDK 1430 running on its pod. For example, in some embodiments, each application 1402 communicates with the hardware accelerator 1450 using a high-level API of the RIC SDK 1430. The RIC SDK 1430 then translates the high-level API to a low-level API required to communicate with the machine's driver 1412, which relays the communication to the hardware accelerator 1450. The low-level API is provided by a first company associated with selling the hardware accelerator 1450, while the RIC SDK 1430 is provided by a second company associated with selling the RIC SDK 1430. In some embodiments, the low-level API used by the RIC SDK 1430 is an API specified in an API library 1432 associated with the hardware accelerator 1450.

[0101] 15 illustrates a process 1500 for implementing the method of some embodiments. The process 1500 is performed in response to an O-RAN component instructing a CP or edge application to perform an operation that the application requires using a hardware accelerator of its host computer. This process 1500 is described below with reference to FIG. 16, which illustrates an application 1402 performing an operation based on data received from an E2 node 1650.

[0102] As shown in FIG. 15, process 1500 begins when application 1402 receives (at 1505) data from O-RAN E2 unit 1650 running on host computer 1610. In some embodiments, application 1402 subscribes to data from E2 unit 1650, and the data received at 1505 is responsive to the subscription. The subscription is made via a distributed near-RT RIC in some embodiments. Host computers 1410 and 1610 of application 1402 and E2 unit 1650 run in one SDDC in some embodiments. In other embodiments, these two host computers 1410 and 1610 run in two different physical locations. For example, host computer 1410 runs in a first location while host computer 1610 runs in a second location closer to an O-RAN cell site. In some embodiments, the second location does not include a computer with a hardware accelerator to perform complex operations including the received operation.

[0103] The application 1402 (1) receives data (at 1505) from the E2 unit 1650 via a distributed near-RT RIC 1680 formed of near-RT RICs 1640 and 1645 executing on the host computers 1410 and 1610, and (2) a RIC SDK 1430 executing on its pod 1404. The application 1402 then uses the hardware accelerator 1450 to perform (at 1510) a set of calculations related to the operation.

[0104] To communicate with the hardware accelerator 1450, the application 1402 uses high-level APIs provided by the RIC SDK 1430. The RIC SDK 1430 then translates the high-level APIs into low-level APIs defined in an API library 1432 associated with the hardware accelerator 1450. These low-level APIs are then communicated to the hardware accelerator 1450 by the pod's driver 1412 via direct pass-through access to the accelerator 1450 that bypasses the drivers 1414 and 1416 of the VM 1406 and hypervisor 1408. Through this driver 1412, the APIs defined in the API library 1432, and the RIC SDK 1430, the application 1402 also receives results of operations (e.g., calculations) performed by the hardware accelerator 1450.

[0105] The application 1402 provides (at 1515) the results of its operations to one or more O-RAN components, such as the E2 unit 1650 that provided the data or SDL storage that initiated the process 1500. The results are provided via the RIC SDK 1430 and the distributed near-RT RIC 1680. In other embodiments, the application 1402 provides (via the RIC SDK 1430) the results of its operations to one or more other applications (other than the E2 unit that provided the data on which the application performed its operations) running on another O-RAN E2 unit or machine running on the same host computer as the application that performed the operations using the hardware accelerator 1450 or on a different host computer. The process 1500 ends after 1515.

[0106] In other embodiments, pass-through access of O-RAN control or edge applications is used in other deployment settings. For example, FIG. 17 shows another example of CP or edge applications 1702 having pass-through access to a hardware accelerator 1750 of a host computer 1710 to perform some (or all) of their operations. In this example, each application 1702 (1) runs on a pod 1704 running on a VM 1706 and (2) uses an accelerator driver 1712 of this VM 1706 that has direct pass-through access to the accelerator 1750 of the host computer 1710. The VM 1706 runs on a hypervisor 1708 running on the host computer 1710. The virtual machine's accelerator driver 1712 bypasses the hardware accelerator driver 1716 of the hypervisor 1708. In some embodiments, the hypervisor 1708 runs on an operating system (not shown) of the host computer 1710. In these embodiments, the virtual machine's accelerator driver 1712 direct access to the hardware accelerator 1750 bypasses the operating system's hardware accelerator driver.

[0107] To use the hardware accelerator 1750, each application 1702 of some embodiments (running on its pod 1704) uses the high-level APIs of the RIC SDK 1730 to communicate with the hardware accelerator 1750. The RIC SDK 1730 translates the high-level APIs to the low-level APIs needed to communicate with the VM's driver 1712, and then relays the communication to the hardware accelerator 1750. In some embodiments, the low-level APIs used by the RIC SDK 1730 are the APIs defined in an API library 1732 associated with the hardware accelerator 1750. This API library 1732 is part of the driver interface of the VM 1706.

[0108] 18 illustrates yet another example of a CP or edge application 1802 that has pass-through access to a hardware accelerator 1850 of a host computer 1810 to perform some or all of its computations. In this example, each application 1802 (1) runs on a VM 1804 running on a hypervisor 1806 running on the host computer 1810, and (2) uses an accelerator driver 1812 of that VM 1804 that has direct pass-through access to the accelerator 1850 of the host computer 1810.

[0109] The virtual machine accelerator driver 1812 bypasses the hardware accelerator driver 1816 of the hypervisor 1806. In some embodiments, the hypervisor 1806 runs on an operating system (not shown) of the host computer 1810. In these embodiments, the virtual machine accelerator driver 1812's direct access to the hardware accelerator 1850 bypasses the operating system's hardware accelerator driver.

[0110] To use the hardware accelerator 1850, each application 1802 of some embodiments (running on its pod 1804) uses the high-level APIs of the RIC SDK 1730 to communicate with the hardware accelerator 1850. The RIC SDK 1830 translates the high-level APIs to the low-level APIs needed to communicate with the VM's driver 1812 and then relays the communication to the hardware accelerator 1850. In some embodiments, the low-level APIs used by the RIC SDK 1830 are those specified in an API library 1832 associated with the hardware accelerator 1850. This API library 1832 is part of the driver interface of the VM 1806.

[0111] Those of ordinary skill in the art will recognize that pass-through access for O-RAN control or edge applications is used in other deployment configurations of other embodiments. For example, instead of running on a pod, the applications of other embodiments run on a container. These embodiments then have pass-through access to the hardware accelerator for the control or edge application using the hardware accelerator drivers of those pods or VMs. In some of these embodiments, the control or edge application communicates with the hardware accelerator through its associated RIC SDK, communicates with other O-RAN components, and communicates with other O-RAN components (to receive data and provide results of its data processing) through its associated RIC SDK and a distributed near-RT RIC that connects the O-RAN components and the application. In some embodiments, the control or edge application of these embodiments performs a process similar to process 1500 of FIG. 15.

[0112] The direct pass-through access to hardware accelerators mentioned above is extremely beneficial for O-RAN. The RIC is all about decoupling the intelligence that was previously embedded in the RAN software (CU and DU) and moving it to the cloud. One benefit of this is that the more advanced computing in the cloud is used for xApps and edge operations (ML, deep learning, reinforcement learning for control algorithms, etc.). DUs close to cell sites typically cannot perform pre-computation as it is not economically feasible to place GPUs at each cell site due to the very high network cap X.

[0113] By using hardware accelerators (GPUs, FPGAs, eASICs, ASICs) in the SDDC, some embodiments execute complex control algorithms in the cloud. Examples of such xApps include massive MIMO beamforming and multi-user (MU) MIMO user pairing, as discussed above. In general, xApps that can benefit from massive parallelism can benefit from GPUs or other accelerators. The use of ASICs is beneficial for channel decoding / encoding (Turbo coding, LDPC coding, etc.). In some embodiments, the RIC is typically on the same worker VM as the xApps. However, in other embodiments, the RIC runs on a different host computer so that more xApps that require GPUs and other hardware accelerators can run on hosts that have GPUs and / or other hardware accelerators.

[0114] 19 illustrates a process that some embodiments use to deploy an O-RAN application with direct pass-through access to a hardware accelerator of a host computer. To install the application on the host computer, a process 1900 selects (at 1905) a set of one or more installation files that include descriptions for configuring the application's pass-through access to the host computer's hardware accelerator. In some embodiments, the set of files includes one description file that specifies direct pass-through access for the application to the computer's hardware accelerator.

[0115] Based on the description of pass-through access, the process 1900 employs (at 1910) a set of installation files that configure a program running on a host computer to pass calls from a particular hardware accelerator driver associated with an application to a hardware accelerator without an intervening set of one or more drivers running on the host computer between the particular hardware accelerator driver and the hardware accelerator. This configuration allows an application to bypass the intervening set of drivers when instructing the hardware accelerator to perform an application operation and receiving the results of the operation from the hardware accelerator.

[0116] In some embodiments, the program configured at 1910 is a host's operating system, and in other embodiments, it is a hypervisor running on the host computer. In yet another embodiment, the program is a virtual machine (VM) and the application runs on a pod or a container running on the VM. Process 1900 processes the remaining set of installation files selected at 1905 to complete (at 1915) the installation of the application, and terminates. In other embodiments, process 1900 performs the configuration of the program as its final operation, rather than as its first operation at 1910. In yet another embodiment, it performs this configuration as one of its intervening installation operations.

[0117] Prior to performing the selection and configuration, the deployment process of some embodiments identifies a host computer from a number of host computers as the computer on which the application should be installed. The process of some embodiments identifies the host computer by determining that the application requires a hardware accelerator, identifying a set of host computers each of which configures a hardware accelerator, and selecting a host computer from the set of host computers. The process selects the host computer by (1) determining that the application needs to communicate with a set of one or more other applications that run on the selected host computer, and (2) selecting the host computer as the set of other applications that will run simultaneously on the host computer. In this manner, installing the application together with the set of other applications on the selected host computer reduces communication delays between the application and the set of other applications.

[0118] Some embodiments have a hardware accelerator driver for an O-RAN control or edge application that communicates with a virtualized hardware accelerator provided by an intervening virtualization application (e.g., a hypervisor) running on the same host computer as the application. For example, a method of some embodiments deploys a virtualized application on a host computer to share the host computer's resources among several machines running on the host computer, the host computer having a first set of one or more physical hardware accelerators.

[0119] The method deploys a number of applications on a number of machines to perform a number of O-RAN related operations on a set of O-RAN components. Through the virtualization applications, the method defines a second set of two or more virtual hardware accelerators that are mapped by the virtualization application to the first set of physical hardware accelerators. The method assigns different virtual hardware accelerators to different applications. The method also includes configuring the applications to perform operations using the assigned virtual hardware accelerators.

[0120] In some embodiments, the deployed machines are pods and applications are deployed to run on the pods. At least two pods run on one VM running on top of a virtualized application. The VM includes a hardware accelerator driver configured to communicate with two different virtual hardware accelerators of the two applications running on the two pods. In other embodiments, multiple pods run on one VM running on top of a virtualized application, with each pod having a hardware accelerator driver configured to communicate with a virtual hardware accelerator assigned to that driver.

[0121] 20 illustrates an example of CP or edge applications 2002 that have pass-through access to virtual hardware accelerators 2052 and 2054 defined by a hypervisor 2008 executing on a host computer 2010 to perform some or all of their operations. As illustrated, each application 2002 runs on a pod 2004 that has an accelerator driver 2012 that has direct pass-through access to the virtual accelerator 2052 or 2054. Each pod 2004 runs within (i.e., runs on) a VM 2006 that runs on the hypervisor 2008 of the host computer.

[0122] Each pod's accelerator driver 2012 has direct access to the virtual accelerator 2052 or 2054, which bypasses the accelerator drivers 2014 and 2016 of the VM 2006 and the hypervisor 2008. In some embodiments, the hypervisor 2008 runs on top of an operating system (not shown) of the host computer 2010. In these embodiments, each pod's accelerator driver 2012's direct access to the virtual accelerator 2052 or 2054 also bypasses the operating system's hardware accelerator driver.

[0123] As illustrated, the virtual accelerators 2052 and 2054 communicate with the hardware accelerator 2050 through an accelerator manager 2060 of the hypervisor 2008. The accelerator manager 2060 allows the virtual accelerators 2052 and 2054 (and their associated applications 2002) to share a single hardware accelerator 2050 and operate with this accelerator 2050 as if it were dedicated to the respective applications and pods 2002 and 2004. Examples of such hardware accelerators 2050 include graphical processing units (GPUs), field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), and structured ASICs.

[0124] To communicate with the virtual accelerator 2052 or 2054, each application 2002 of some embodiments communicates through a RIC SDK 2030 running on the pod 2004. For example, in some embodiments, each application 2002 uses a high-level API of the RIC SDK 2030 to communicate with its virtual accelerator 2052 or 2054. The RIC SDK 2030 then translates the high-level API into a low-level API required to communicate with each machine's driver 2012, which relays the communication to the virtual accelerator 2052 or 2054. The virtual accelerator 2052 or 2054 then relays the communication to the hardware accelerator 2050 through the accelerator manager 2060.

[0125] 14, in some embodiments, the low-level APIs are provided by a first company associated with selling the hardware accelerator 2050, while the RIC SDK 2030 is provided by a second company associated with selling the RIC SDK 2030. In some embodiments, the low-level APIs used by the RIC SDK 2030 are APIs defined in an API library 2032 associated with the hardware accelerator 2050. Each application 2002 receives the results of the operation of the hardware accelerator 2050 via the accelerator manager 2060, its virtual accelerator 2052 or 2054, its driver 2012, and its RIC SDK 2030.

[0126] To provide a low-latency near-RT RIC, some embodiments separate the RIC functionality into several different components that run on different machines (e.g., running on VMs or pods) that run on the same or different host computers. Some embodiments also provide high-speed interfaces between these machines. Some or all of these interfaces operate in a non-blocking, lockless manner to ensure that critical operations (e.g., data path processing) of the near-RT RIC are not delayed due to multiple requests that stall one or more components. Furthermore, each of these RIC components also has an internal architecture designed to operate in a non-blocking manner so that a process of one component cannot block the operation of another process of the component. All of these low-latency features enable the near-RT RIC to act as a high-speed IO between E2 nodes and xApps.

[0127] FIG. 21 shows an example of a near-RT RIC 2100 with several components running on several different machines. In this example, the near-RT RIC is split into three pods: a datapath pod 2105, a services pod 2110, and an SDL pod 2115. In some embodiments, this RIC (1) processes E2AP messages between the E2 node 2118 and the xApps 2120, (2) manages connections with the E2 node 2118, (3) processes xApp subscriptions to the E2 node, and (4) processes xApp liveness operations. The RIC 2100 provides reliable, low-latency messaging between the various components, between the E2 node and the xApps, and between the E2 node / xApps and the RIC components. Part of the low-latency architecture of the RIC results from the use of different pods to perform data IO, services, and SDL operations. Therefore, different resource allocation and management operations can be provided to each of these pods based on the needs of each of the operations they perform in the different pods.

[0128] Each of the three RIC pods 2105, 2110, and 2115 communicates with one or more xApp pods 2120. In some embodiments, each pod (2105, 2110, 2115, or 2120) is assigned hardware resources (e.g., CPU, memory, disk storage, network IO, etc.) according to the pod's unique needs (i.e., the data paths, services, and storage operations performed by each pod). Also, in some embodiments, each pod has its own high availability and lifecycle update settings that match the unique needs of each pod.

[0129] Service pod 2110, in some embodiments, performs xApp onboarding, registration, FCAPS (Fault, Configuration, Accounting, Performance, Security), and other services. For example, in some embodiments, service pod 2110 provides management services 554 for near-RT RICs and performs O1 termination 580 and A1 termination 582 for SMOs and their associated non-RT RICs. In some embodiments, each of these components 554, 580, and 582 runs on a separate container within service pod 2110, and in other embodiments, two or more of these components run on one container within service pod 2110.

[0130] As mentioned above, in some embodiments, the A1 interface is between the near-RT RIC and the non-RT RIC. Through this interface, the near-RT RIC relays relevant network information reported by the E2 nodes (e.g., CU and DU) and the non-RT RIC provides control commands for the E2 nodes (e.g., for control use case operations at non-RT granularity). The O1 interface is between the near-RT RIC and the SMO and, in some embodiments, is used for discovery, configuration, resource management, auto-scaling, lifecycle management, and fault tolerance.

[0131] RIC management services 554 in some embodiments include services that the near-RT RIC provides to xApps and other RIC components. Examples of services provided to xApps include an xApp service registry / directory (which xApps can use to identify other xApps associated with the distributed near-RT RIC and the operations performed by these other xApps) and FCAP operations (metrics collection, policy provisioning, configuration, etc.). In some embodiments, xApps can query the service registry / directory to identify other xApps or other xApps that perform specific services, and can register to receive notifications about xApps and their capabilities when xApps are added to the directory.

[0132] Examples of FCAP operations performed by service pod 2110 for xApps include metrics to collect CPU and memory utilization for analysis to generate alerts, configuration operations to configure or reconfigure xApps, and accounting operations to collect data required for accounting and performance operations to collect metrics from xApps and analyze them to quantify the performance of the xApps.

[0133] For other RIC components (e.g., datapath pod 2105 and SDL pod 2115), the service pod 2110 also performs services such as metric collection, policy provisioning, configuration, etc. The service pod 2110 can be seen as a centralized controller, i.e., a local controller that performs operations at the direction of the SMO. Through the SMO, the service pod 2110 receives configurations and policies that it distributes to the xApps and other RIC components. To the SMO, the service pod 2110 also provides metrics, logs, and trace data collected from the xApps and / or RIC components (e.g., datapath pod and SDL pod). In some embodiments, the service pod can be scaled (e.g., replicated) and backed up independently of other pods. In some embodiments, the service pod has a data cache, which is a cache for the SMO's time series database. In this cache, the service pod stores statistics, logs, trace data, and other metrics collected from the xApps and one or more RIC components before uploading this data to the SMO database.

[0134] The SDL pod 2115 implements the SDL 560 and its associated database 570. As described further below, the SDL pod 2115 of some embodiments also runs one or more service containers to perform one or more pre-processing or post-processing services on data stored in the SDL. Like service pods, the SDL pods of some embodiments can be scaled (e.g., replicated) and backed up independently of other pods.

[0135] The datapath pod 2105 contains several important near-RT RIC components: E2 termination 584, conflict mitigation 550, application subscription management 552, and the RIC SDK interface 2150. As described further below, some or all of these datapath services in some embodiments are incorporated into the datapath thread and control thread of the datapath pod. In other embodiments, the datapath services are incorporated into a data IO thread, multiple data processing threads, and a control thread.

[0136] A thread is a component of a process running on a computer. A process can be an application or a part of a larger application. A thread is a sequence of programmed instructions that can be managed independently of other threads of the process. Multiple threads of a particular process can run simultaneously (such as using the multithreading capabilities of multi-core processors) while sharing the memory allocated to the process. Multithreading is a programming and execution model that allows multiple threads to exist within the context of a single process. These threads share the resources of the process but can run independently.

[0137] The control thread in some embodiments is the interface to the service pods and SDL pods for the datapath threads, while in other embodiments it is the interface to the service pods for the datapath threads (because the datapath threads can communicate directly with the SDL pods). The control thread in either of these approaches performs the slower control related operations of the datapath, while one or more datapath threads perform the faster IO operations of the datapath. The control thread in some embodiments interfaces with the service pods to receive configuration data for configuring its own operation as well as the operation of the datapath threads.

[0138] The embodiment of separating the data path thread into a data IO thread and multiple data processing threads further optimizes data IO by pushing the more computationally intensive operations of the data path thread to multiple data path processing threads, thereby allowing less computationally intensive operations to run on the data IO thread. Both of these optimizations are intended to ensure fast data path IO (one that does not introduce unnecessary delays), allowing the near RT RIC to act as a high-speed interface between the E2 node 2118 and the xApps 2120.

[0139] As described above, pods 2105, 2110, and 2115 communicate through high-speed pod interfaces. In some embodiments, pod-to-pod connections are established through SCTP (Streaming Control Transmission Protocol) or even faster shared memory (shmem) connections. In some embodiments, shared memory connections are used only between pairs of pods that run on the same host computer. Examples of such pod pairs include (1) datapath pod and SDL pod, (2) datapath pod and service pod, (3) service pod and SDL pod, (4) xApp pod and datapath pod, (5) xApp pod and SDL pod, etc. The shared memory is lockless, and in some embodiments access to it is non-blocking. In other embodiments, a slower interface (e.g., gRPC) between service pod 2110 and other pods 2105, #115, and 2120 is used for service pods, but not as critical as other pods.

[0140] Different pods (e.g., 2105, 2110, and 2115) of a near RT RIC in some embodiments can run on the same host computer, or on more than one host computer. In other embodiments, one or more pods (e.g., service pod 2110) always run on a separate host computer than the other pods (e.g., datapath pod 2105 and SDL pod 2115). Also, in some embodiments, pods 2105, 2110, and 2115 run on one host computer 2205 along with one or more xApp pods 2220a, and other xApp pods 2220b run on other host computers 2210, as shown in FIG. 22. In other embodiments, two pods 2105, 2110, and 2115 run on one host computer along with one or more xApp pods, and one pod 2105, 2110, and 2115 runs on another host computer along with one or more xApp pods.

[0141] For example, FIG. 23 shows a datapath pod 2105 and a service pod 2110 running on a host computer 2300 with two xApp pods 2320a, while an SDL pod 2115 runs on a host computer 2310 with two xApp pods 2320b. In some embodiments, pods that require hardware accelerators are placed on host computers that have such hardware resources, and pods that do not require these accelerators are placed on host computers that do not have these accelerators, as described above. In some embodiments, the SDL pod and the xApp pods use hardware accelerators, and the datapath pod and the service pods do not. Various examples of pods that can benefit from hardware accelerators, and bypasses to these accelerators, are described further above and below.

[0142] Also, while some near-RT RICs are described above and below as being implemented using pods, other embodiments of the near-RT RIC use VMs to implement the RIC components. Further, even in embodiments that implement different RIC components in pods, some or all of the pods run on VMs, such as lightweight VMs (e.g., Photon VMs provided by VMware, Inc.).

[0143] In addition to using high-speed communication interfaces between pods, in some embodiments, some or all of the pods use non-blocking, lockless communication protocols and architectures. For example, the datapath pod 2105 uses non-blocking, lockless communication between the threads and processes that make up the pod. The datapath pod 2105 also uses non-blocking, lockless communication when communicating with the service pod 2110, the SDL pod 2115, and the xApp pod 2120. In non-blocking communication, a first component sending a request to a second component does not stall the operation of the second component if the second component has too many requests to process. In such a case, the second component instructs the first component to resend the request at a later time. The datapath pods use lockless communication because they use single-threaded processing with no thread handoffs. As such, there is no need to lock a portion of memory to prevent another process thread from modifying the memory for a temporary period of time.

[0144] The communication interface between the RIC SDK interface 2150 of the datapath pod 2105 and the RIC SDK 2112 of the xApp pod 2120 is also novel in some embodiments. In some embodiments, this interface parses the headers of E2AP messages received from the E2 node and stores some or all of the parsed components in a new encapsulation header that encapsulates the E2SM payload of the E2AP message along with some or all of the original E2AP headers. In performing this encapsulation, the SDK interface 2150 of some embodiments performs certain optimizations, such as efficiently performing data packing, to reduce message size overhead for communication from one pod to another (e.g., reducing the size of E2 global ID values). These optimizations improve the efficiency of near RT RIC datapath pod and xApp pod communication.

[0145] In other embodiments, the near-RT RIC has one or more other pods. For example, FIG. 24 shows a near-RT RIC 2400 that includes a lifecycle management (LCM) pod 2405 in addition to pods 2105, 2110, and 2115. The LCM pod is a special service pod that upgrades each of the other pods 2105, 2110, and 2115 of the near-RT RIC 2400. Separating the lifecycle management from the service pod 2110 allows the service pod 2110 to be upgraded more easily.

[0146] In some embodiments, the LCM pod 2405 uses different upgrade methods to upgrade different pods. For example, the LCM pod of some embodiments replicates the SDL data storage and seamlessly transitions from an active data store to a separate standby data store to perform a hitless upgrade of the SDL. On the other hand, to upgrade a datapath pod, the procedure for the LCM pod is more complicated. This is because the active datapath pod and the standby datapath pod are configured to dual-home each xApp with the E2 node, and the active datapath pod is configured to replicate state with the standby datapath.

[0147] 25 shows a more detailed view of a near-RT RIC 2500 for some embodiments. As illustrated, the near-RT RIC 2500 includes a datapath pod 2505, a service pod 2510, and an SDL pod 2515. The SDL pod 2515 includes an SDL agent 2526 and shared SDL storage 2528 for the RIC 2500, while the service pod 2510 includes a service agent 2524 along with O1 and A1 termination interfaces 2535 and 2537. The datapath pod 2505 includes a datapath thread 2507 and a control thread 2509.

[0148] Datapath thread 2507 provides high speed datapath IO for the near-RT RIC between E2 node 2518 and xApps 2520. The dataplane capacity of the RIC in some embodiments can be scaled up by implementing the RIC datapath IO with one control thread and multiple datapath threads that share the load for datapath processing for the datapath pods. Some such implementations are further described below. Control thread 2509 performs some control operations related to the datapath of the RIC. The near-RT RIC 2500 separates the control and datapath threads because data IO operations need to be fast and should not be slowed down by control operations that can operate at slower speeds. In some embodiments, the control and datapath threads are two threads within a single process (i.e., they run in the same shared memory address space).

[0149] In some embodiments, each of these threads uses a non-blocking, lockless interface to communicate with other components of the architecture (e.g., the RIC SDK, the service pod agent, the SDL agent, and / or the E2 node), to the extent that it communicates with these other components. Also, in some embodiments, both threads use minimal OS system calls and run as an infinite loop. As described further below, the datapath thread and the control thread exchange data via two circular rings 2522 (called cbufs), with one ring handling messages from the datapath thread to the control thread and the other handling messages from the control thread to the datapath thread.

[0150] The control thread 2509 serves as the control interface to the E2 node 2518, the SMO 2530 (through the service pod agent 2524), xApps (e.g., through SCTP), and the SDL layer (through the SDL agent 2526). In some embodiments, the control thread is the main thread for communicating with these external entities, although the datapath thread in some embodiments communicates with the SDL 2515 through the SDL agent 2526, as described further below.

[0151] The control thread 2509 in some embodiments handles all control functions. This thread sends various control parameters to other functions and in some embodiments performs admission control. In other embodiments, the datapath thread 2507 performs admission control and the SMO via the service pod specifies admission control. In some embodiments, the control thread 2509 has a control channel communication with the xApp pod's RIC SDK via SCTP. In other embodiments, the control thread communicates with the xApp pod's RIC SDK via gRPC. Also, in some embodiments, the control thread communicates with the RIC SDK via shared memory (shmem) when the xApp pod and the datapath pod run on the same host computer.

[0152] The control thread 2509 also provides a transport mechanism for transporting statistics, logs, and trace data generated by the datapath thread 2507. In some embodiments, some or all of this data is forwarded to the SDL pod 2515 via the SDL agent 2526 and / or to the SMO 2530 via the service agent 2524. The control thread 2509 in some embodiments negotiates security keys with E2 node peers and passes these keys to the datapath thread, which uses the keys to perform encryption / decryption operations.

[0153] The datapath thread 2507 provides high speed IO between E2 nodes and xApps. This thread handles the RIC SDK interface and E2 termination operations, and in some embodiments conflict resolution and xApp subscription operations. This thread performs ASN.1 decoding of E2AP messages to extract message data. In some embodiments, the datapath thread does not decode the E2SM payload of these messages. The datapath thread 2507 validates E2 and xApp messages and sequences. In some embodiments, message types include E2 node setup and service updates, E2 node indication reports, xApp initiated subscriptions to E2 node data, and xApp initiated control requests.

[0154] The datapath thread 2507 in some embodiments executes the E2 state machine to create and maintain state on behalf of xApps (e.g., state of the E2 node, subscriptions to the E2 node, etc.). In some embodiments, the datapath thread also performs table lookups to send messages to xApps requesting data. This thread also handles control requests from xApps to the E2 node and forwards responses to these requests from the E2 node to the xApps.

[0155] The data path thread communicates with xApps via SCTP if they are on another host computer, or via shared memory if they are on the same host computer. In some embodiments, xApp messages have CRC bits to detect corruption. These messages also carry a timestamp, and in some embodiments can be compressed. The data path thread 2507 performs data replication for multiple subscriptions. The data path thread 2507 also performs data path security operations, such as by signing, encrypting, and decrypting data messages.

[0156] As mentioned above and further described below, the datapath thread 2507 communicates with the control thread 2509 in some embodiments via a pair of rings 2522. In some embodiments, the frequency of messages between the two threads is adjustable (e.g., configurable) from sub-milliseconds to seconds per ring pair. Through the control thread, the datapath thread 2507 receives configuration data updates and state changes. The datapath thread 2507 generates statistics, logs, and traces and provides the generated statistics, logs, and trace data to the control thread for storage in the SDL and / or for providing to the SMO.

[0157] The datapath thread 2507 also performs conflict management operations when multiple xApps attempt to set the same parameters on the same E2 node at the same time. For example, conflict management operations prevent two xApps from trying to change mobile network settings (such as antenna direction) in different ways within a short period of time. In some embodiments, the datapath thread conflict management employs different methodologies to deal with different types of conflicts. For example, (1) for a set of requests, reject a second request to change a parameter after receiving a conflicting first request for a certain period of time; (2) for another set of requests, reject a request for a parameter from one xApp when another higher priority xApp makes a conflicting request for the same parameter; and (3) for another set of requests for another set of parameters, accept only requests made by xApps that are allowed to make such requests during a certain period of time. Policies for handling these conflicts are provided by the SMO 2530 via the service pod's agent 2524.

[0158] In FIG. 25, each xApp Pod 2520 can run one or more xApps 2532 and interfaces with the Datapath Pod 2505, SDL Pod 2515, and Service Pod 2510 through a RIC SDK 2534 running on the xApp Pod. Each RIC SDK provides a high-level interface for the xApps to communicate with the RIC and E2 nodes. This high-level interface hides the underlying implementation details. The RIC SDK communicates with the RIC instance through a high-speed data IO communication channel (such as shared memory or SCTP).

[0159] The RIC SDK also uses a control communication channel with the service pods 2510 and the control thread 2509 for xApp control operations such as xApp onboarding, registration, features, subscriptions, FCAPS, etc. In some embodiments, the control channel communication between the SDK and the control thread 2509 is via shared memory when the xApp pod (and its SDK) and the datapath pod run on the same host computer, and via SCTP when they run on different computers. In some embodiments, the control channel communication between the xApp pod (and its SDK) and the service pod is via shared memory when the SDK and the service pod run on the same host computer, and via gRPC when they run on different computers. In other embodiments, the communication between the SDK and the service pod uses SCTP when the xApp pod (and its SDK) and the service pod run on different host computers.

[0160] In some embodiments, the RIC SDK uses proto bufs when communicating with service pods over gRPC. Also, in some embodiments where the RIC SDK communicates with data path pods over shared memory, the shared memory communication uses protocol bufs. The RIC SDK has APIs for data functions, such as E2 messages to and from E2 nodes. These APIs also include control function messages, such as onboarding xApps (name, version, function), message subscriptions, keep-alive messaging, and A1 and O1 interface communication with SMO via service pods (e.g., communication to store statistics, logs, and trace data in a time series database on SMO or on the service pod (e.g., Prometheus or ELK)).

[0161] In some embodiments, the datapath thread and control thread are assigned to one processor core, the SDL (to separate from the data and control threads) is assigned to another processor core, and the service pod is assigned to yet another processor core. If one or more xApps run on the same host computer as the RIC, the xApps are assigned to different cores than the RIC pod. In a RIC pod, multiple xApps can be assigned to the same core, or individual cores can be assigned to separate xApps as needed.

[0162] To further improve the performance of RIC and xApp, other embodiments perform other hardware allocation optimizations, such as specific memory allocation (e.g., larger RAM allocation) and specific IO allocation. Examples of dedicated IO allocation for some pods include (1) SRIOV allocation for an xApp pod on one host computer to communicate with a datapath pod on another host computer, (2) SRIOV allocation for a datapath pod to communicate with an E2 node, (3) SRIOV allocation for an xApp pod on one host computer to communicate with a service pod on another host computer, and (4) gRPC or SCTP communication using SRIOV allocation with gRPC communication where gRPC communication has a lower bandwidth allocation and a lower priority than SCTP communication.

[0163] In some embodiments, a RIC and several xApps are bundled together to run on different pods that run on one VM. In some embodiments, multiple instances of a RIC can also be deployed with different sets of xApps. Also, in some embodiments, xApps that need to communicate with each other are bundled together on the same VM.

[0164] As mentioned above, some embodiments implement the RIC datapath as one data IO thread with multiple datapath processing threads rather than one datapath thread. In some embodiments, each datapath processing thread (DPT) is responsible for performing datapath processing for a set of different E2 nodes, where each E2 node is assigned to only one datapath processing thread. In some embodiments, the data IO thread identifies the DPT associated with an E2 or xApp message by hashing the E2 node identifier included in the message and using the hashed value (obtained by the hash) as an index into a lookup table that provides the DPT identifier of the DPT that needs to process the data message.

[0165] FIG. 26 illustrates an example of a RIC datapath pod 2600 having one data IO thread 2605, one control thread 2610, and multiple DPTs 2615. The DPTs share the datapath processing load of the datapath pod 2600. As illustrated, between each DPT 2615 and the data IO thread 2605, each DPT 2615 and the control thread 2610, and between the data IO thread 2605 and the control thread 2610, there is a pair of cbuf rings 2620. Each ring of a cbuf pair 2620 passes data messages in one direction from one of the two threads associated with the ring to the other thread, with one ring handling communications in one direction (e.g., from a first thread to a second thread) and the other (e.g., from a second thread to a first thread).

[0166] Separating the data IO thread 2605 from the multiple DPTs 2615 optimizes the data IO of the data path pod 2600 by pushing the more computationally intensive operations to the DPT, thereby allowing the less computationally intensive IO operations to run within the data IO thread 2605. This optimization ensures fast data path IO (without unnecessary delays) so that the RIC can act as a fast interface between the E2 nodes and the xApps. Also, each E2 node is the responsibility of only one DPT thread 2615, which typically serves several E2 nodes. Since each E2 node is handled by one specific DPT, two DPTs will not attempt to modify one or more records associated with one E2 node. Thus, the data path pod 2600 does not need to lock the records of the E2 nodes because there is a clear separation of responsibilities for communicating with the E2 nodes.

[0167] The data IO thread 2605 performs the following functions: (1) manages connections to E2 nodes and xApp pods, (2) sends data messages over these connections between E2 nodes and xApp pods, (3) performs security operations, (4) controls ring communication with the control thread 2610 and DPT 2615, and (5) generates statistics, log, and trace data about the messages it processes.

[0168] Each DPT thread 2615 performs the following operations: (1) message decoding and encoding operations (e.g., message encryption and decryption operations); (2) message validation operations; (3) sequence validation operations; (4) maintaining state machines to track the state of E2 nodes and xApp requests and subscriptions; (5) performing conflict management; (6) control ring communication with control thread 2610 and DPT 2615; and (7) generating statistics, log, and trace data about the messages it processes.

[0169] Figure 27 illustrates a process 2700 that a data path thread performs to process a subscription request from an xApp in some embodiments. As shown, the process begins when the DPT receives (at 2705) an xApp subscription request from the data IO thread. The subscription request is directed to a particular E2 node for a particular set of data tuples (e.g., a particular set of operational parameters or other parameters) that the particular E2 node maintains.

[0170] At 2710, the process 2700 determines whether the DPT has already subscribed to the particular E2 node to receive the particular set of data tuples, either because the DPT previously sent one or more subscription requests to the particular E2 node, requesting the particular set of data tuples, or a larger set of data tuples that includes the particular set of data tuples, either individually or collectively.

[0171] If the process 27100 determines (at 2710) that it has already subscribed to a particular E2 node to receive a particular set of data tuples, it adds (at 2715) a new record, or updates a previously specified record, for the xApp in the subscription list of this E2 node, specifying in this record the particular set of data tuples that the xApp should receive. After 2715, the process ends.

[0172] On the other hand, if the process 27100 determines (at 2710) that it is not already subscribed to the particular E2 node to receive the particular set of data tuples, it either sends a first subscription to the particular E2 node if it does not have an active subscription with this node, or sends an updated subscription to the node if it does not include all data tuples of the particular data tuple specified in the request received at 2705.

[0173] Thus, in such a case, the process 2700 (at 2720) adds a new record or updates a previously specified record for the xApp in the subscription list of this E2 node, specifying in this record the specific set of data tuples that the xApp should receive. It then sends an updated subscription request to the specific E2 node using the previously assigned RIC request ID. This updated subscription specifies all data tuples in the specific set of requested specific data tuples, if none of these tuples were previously requested by a previous subscription to the specific E2 node, or specifies some of these data tuples, if other data tuples in the specific set were previously requested by one or more previous subscriptions to the specific E2 node. After 2725, the process 2700 ends.

[0174] 28 shows a process 2800 that a data IO thread 2605 and a DPT 2615 execute to process a data message from an E2 node that, in some embodiments, one or more xApps are to receive. As illustrated, the process 2800 begins (at 2805) when a data message is received by a data path pod over an SCTP connection with the E2 node. At 2810, the data IO thread 2605 generates a hash value from the ID of the E2 node. It then uses (at 2815) the hash value as an index into a lookup table to identify a DPT that is assigned to process the message associated with the E2 node.

[0175] At 2820, the data IO thread passes the received data message to the identified DPT (i.e., the DPT identified at 2815) along a cbuf ring 2620 for passing messages from the data IO thread to the identified DPT. The DPT then uses its data structure record (e.g., a record maintained by its state machine) to identify a set of one or more xApps that should get the E2 message, at 2825. In some embodiments, the identified set of xApps are xApps that have subscribed to receive data (e.g., all data or a subset of data) from the E2 node.

[0176] At 2830, the DPT identifies a data message for the Data IO Thread 2605 to send to the set of identified xApps. The data message is in an encapsulated format with reference to Table 1 below. The DPT then passes the data message to the Data IO Thread 2605 (at 2835) along the cbuf ring 2620 which passes the message from the DPT 2615 to the Data IO Thread 2605. The Data IO Thread 2605 then retrieves the data message from the cbuf ring 2620, identifies the xApps that need to receive the data message, and then sends the data message to each identified xApp. After 2840, processing ends.

[0177] 29 shows a process 2900 in which a data IO thread 2605 and a DPT 2615 execute in some embodiments to process a data message from an xApp to be sent to an E2 node. As illustrated, the process 2900 begins (at 2905) when a data message is received by the data path pod via an SCTP connection or shared memory communication with the xApp RIC SDK. The message is in an encapsulated format described below with reference to Table 1. The message includes an E2 node identifier that identifies the E2 node that receives the message.

[0178] At 2910, the data IO thread 2605 generates a hash value from the ID of the E2 node. It then uses (at 2915) the hash value as an index into a lookup table to identify a DPT assigned to process the message associated with the E2 node. At 2920, the data IO thread passes the received data message to the identified DPT (i.e., the DPT identified at 2915) along a cbuf ring 2620 for passing messages from the data IO thread to the identified DPT.

[0179] Next, at 2925, the DPT uses its data structure records (e.g., records maintained by its state machine) to identify E2 nodes that should receive the message. In some embodiments, the data message is a subscription request, and the identified E2 nodes are the E2 nodes to which the xApp wants to subscribe. At 2930, the DPT specifies a data message for the data IO thread 2605 to send to the identified E2 nodes. This data message is in the E2AP message format required by the standard. The DPT then passes the data message to the data IO thread 2605 (at 2935) along the cbuf ring 2620, which passes the message from the DPT 2615 to the data IO thread 2605. Next, at 2940, the data IO thread 2605 retrieves the data message from the cbuf ring 2620, identifies the E2 nodes that need to receive the data message, and sends the data message to each identified E2 node. After 2940, processing ends.

[0180] In some embodiments, the DPT 2615 may determine that it does not need to send a new subscription message to the E2 node identified at 2925. For example, prior to receiving (at 2905) a subscription request from the first xApp for a set of data tuples from the E2 node, a data path pod previously sent to the second xApp sends a subscription request to the same E2 node for the same set of data tuples or a larger set of data tuples that includes the data tuples requested by the first xApp. In such a case, the DPT 2615 need only add the first xApp to the subscription list of the E2 node so that it can provide the first xApp with values ​​received from the E2 node thereafter. In some embodiments, the DPT 2615 also provides the first xApp with values ​​previously received from the E2 node that were stored in the SDL, or instructs the xApp to obtain these values ​​from the SDL.

[0181] In some cases, the first xApp requests additional data tuples from the E2 node that the second xApp did not previously request. In such cases, the DPT 2615 prepares an updated subscription message for the data IO thread to send to the E2 node to request the data tuples newly requested by the first xApp. The DPT also prepares such a message when the second xApp requests additional data tuples from the E2 node after the first subscription.

[0182] In some embodiments, the service pod 2510 configures the data path pod 2600 to instantiate N DPTs when it starts up, where N is an integer greater than 1. For a data path pod 2600 in a near RT RIC, the number N is calculated in some embodiments based on the expected number of E2 nodes and xApps that will communicate with the E2 nodes through the near RT RIC. In some embodiments, the data IO thread 2605 of the data path pod 2600 then assigns E2 nodes to DPTs based on the order of incoming subscription requests and the load on the DPTs at the time of those requests.

[0183] 30 illustrates an example process that a data IO thread uses to assign an E2 node to a DPT in some embodiments. As illustrated, process 3000 begins when a data IO thread 2605 receives 3005 a first subscription request for a particular E2 node from an xApp associated with the data IO thread's near RT RIC. The first subscription request for a particular E2 node means that no other subscription requests have been previously received by the data IO thread for this particular E2 node.

[0184] Next, at 3010, the data IO thread 2605 generates an N-bit hash value from the global E2 node ID of the particular E2 node, where N is an integer (e.g., 6 or 8). This N-bit value is used to identify the particular E2 node in a hash LUT (lookup table), as described below. At 3015, the process 3000 selects a particular DPT for the particular E2 node based on the current load on each DPT of the data path pod 2600 (e.g., by selecting the DPT with the least amount of load). In some embodiments, the current load is based only on the number of E2 nodes assigned to each DPT, while in other embodiments, the current load is based on the number of E2 nodes and the number of xApp subscriptions to those nodes. In still other embodiments, the current load is calculated in other ways.

[0185] Next, at 3020, the process 3000 creates a record in the LUT that associates the N-bit hash value with an identifier of the particular DPT selected at 3015 for the particular E2 node. In some embodiments, the N-bit hash value is an index into the LUT that identifies a record that specifies the ID of the particular E2 node. At 3020, the process 3000 also designates the state of this record as active.

[0186] At a later point in time, if the data IO thread encounters a situation where all xApps have cancelled their subscriptions to a particular E2 node, process 3000 retains the LUT record created in 3020, but changes the status of this record to inactive. The data IO thread maintains this inactive state until the next time an xApp sends a subscription request for a particular E2 node, at which point the state of this record is changed back to active. This status value is used as a mechanism to ensure that the data IO thread does not have to continually reassign E2 nodes to DPTs.

[0187] 31 illustrates a distributed near-RT RIC implemented with an active RIC 3102 and a standby RIC 3104. As shown, the E2 node 3118 and the xApp pod 3120 communicate with the active RIC 3102 until one or more components of this RIC fail. If the active RIC fails, the standby RIC 3104 becomes the active RIC and the E2 node and the xApp pod continue to communicate through the now active RIC 3104.

[0188] Both of these RICs have the same components: datapath pod 3105, service pod 3110, and SDL pod 3115. The datapath pod is shown to include a control thread 3109 and a datapath thread 3107. Instead of one datapath thread 3107, some embodiments use one data IO thread and multiple DPTs, as described above. In some embodiments, the active RIC 3102 is implemented by a first set of one or more computers, while the standby RIC 3104 is implemented by a different second set of one or more computers.

[0189] As shown, each E2 node 3118 has a dual-homed SCTP connection with the data path threads 3107 of the active and standby RICs 3102 and 3104. Similarly, each xApp pod 3120 has a dual-homed SCTP connection with the data path threads 3107 of the active and standby RICs 3102 and 3104. Dual-homing connections are a feature that SCTP provides. When a first component connects to an active / standby pair of components through a dual-homed connection, the first component can automatically switch to use the standby component when the active component fails. Thus, the use of dual-homed SCTP connections allows each E2 node or xApp pod to switch to the data path thread 3107 of the standby RIC 3104 when the active RIC 3102 or its data path pod fails.

[0190] As described, the RIC SDK interface 3122 of the data path thread 3107 of the active RIC 3102 forwards messages it receives from and sends to the xApp RIC SDK to the RIC SDK interface 3122 of the data path 3107 of the standby RIC 3104. This, in some embodiments, allows the data path thread 3107 of the standby RIC to update its state machine to match the state of the data path thread 3107 of the active RIC. Also, as described, the synchronization agent 3127 of the active and standby RICs 3102 and 3104 synchronizes the SDL storage 3126 of the standby RIC 3104 with the SDL storage 3126 of the active RIC 3102. All components of the active and standby RICs 3102 and 3104 are consistently managed by the SMO 3130.

[0191] FIG. 32 illustrates an interface between a near RT RIC 3200 and an E2 node 3205, and between a near RT RIC 3200 and an xApp pod 3210, in some embodiments. In some embodiments, the near RT RIC is one of the near RT RICs described above. As described above and illustrated in FIG. 32, the near RT RIC in some embodiments uses SCTP interfaces 3220 and 3222 with both the E2 node 3205 and the xApp pod 3210. If the xApp pod and the near RT RIC run on the same host computer, in some embodiments, a shared memory is used as the interface between the near RT RIC and the xApp pod, as illustrated. These interfaces keep message exchanges fast and minimize latency with minimal encoding and decoding overhead across all paths (e.g., one ASN decode and one ASN encode). In some embodiments, the interface 3220 between the E2 node and the near RT RIC follows the E2AP specification, and all message exchanges conform to the E2AP specification.

[0192] Also, in some embodiments, the interface 3222 between the near RT RIC and the xApp pod uses a novel encapsulation header, described below with reference to Table 1. The interface 3222 handles a mix of different types of messages. Examples of such messages in some embodiments include: (1) an entire E2AP message from the E2 node (e.g., an E2 setup request), (2) some fields of the E2AP header along with the entire E2SM content (i.e., the entire E2AP message payload), (3) an internal message between the near RT RIC and the xApp pod (e.g., a message from the near RT RIC that a previous message for the xApp has erroneous), and (4) a message from the xApp to the near RT RIC or to the E2 node. In some embodiments, the E2 content may not be ASN1 encoded (e.g., parts of the subscription request may not be encoded).

[0193] In some embodiments, the near RT RIC 3200 can be configured on a case-by-case basis to decode only the E2AP message or the entire E2AP header along with the E2SM payload before sending the message to the xApp. In some cases, the near RT RIC sends the entire E2AP header, and in other cases, it sends only a portion of this header. In some embodiment RIC E2AP message processing, all fields are in network byte order, and the near RT RIC 3200 operates in that order as much as possible. To display the fields, some embodiments can convert the data to host order. In some embodiments, the near RT RIC 3200 never sees the E2SM payload, but in other embodiments, it does (e.g., to avoid duplicate subscription errors).

[0194] In some embodiments, the RAN capability ID is E2 node specific. Since each subscription belongs to an individual E2 node, xApps do not subscribe to RAN capabilities across E2 nodes. Also, in some embodiments, the RIC request ID space is local to an E2 node. In some embodiments, the RIC request ID number space is a temporary component as well as a persistent component. For example, a RIC request ID used for a display report will persist even if a RIC request ID used from a subscription is reused.

[0195] Table 1 below shows an exemplary message format used in some embodiments for communication between the RIC and the RIC SDK. This is the format of the encapsulation header used to encapsulate all messages sent to and from the RIC SDK. In some embodiments, the encapsulation header stores data required by the RIC SDK and the RIC for efficient processing of the data message. In the example shown in Table 1, the first 16 bytes associated with msg_type, msg_serial_num, msg_len, msg_flags, and ctrl_len are part of the encapsulation header along with the ctrl_info field. The payload of the encapsulation packet can contain any data. In the example shown in Table 1, the payload contains the original E2AP packet and the E2SM payload.

[0196] All messages between the RIC and the RIC SDK are encapsulated with the headers shown in Table 1. Control information and payload are optional. Some messages may have control information but no payload fields, some have payloads with no control information, and some have both control and payload fields. In some embodiments, the RIC SDK can be configured to trap these messages and reformat them for presentation to xApps. The message format is a raw byte stream. In some embodiments, the message CRC field is not used and in other embodiments it is. [Table 1]

[0197] The near RT RIC 3200 handles E2 node and xApp connections, disconnections, resets, and crashes as follows: For E2 nodes, the RIC of some embodiments handles connections, disconnections, and crashes in the same way. Specifically, when a connection to an E2 node is dropped and comes back for any of these reasons, the E2 node sends a connection setup again as if it had first started up, and the near RT RIC cleans up and starts over with all state related to the E2 node. In some embodiments, the near RT RIC notifies all xApps when an E2 node connection drops and comes back, regardless of whether they were previously subscribed to a particular E2 node. This is because the E2 node may advertise new capabilities that an xApp that was not previously subscribed to may have been interested in. When an xApp connects, disconnects, or crashes, the near RT RIC performs the same operations again: it resets all state of the xApp in the near RT RIC and removes its subscriptions from all E2 nodes.

[0198] FIG. 33 illustrates E2AP message processing of the data path pod of the near-RT RIC 3200. In the following description of E2AP message processing and other message processing, for brevity, this data path pod is referred to simply as the near-RT RIC 3200 or RIC 3200. As shown, the near-RT RIC 3200 first decodes the E2AP message received from the E2 node. In some embodiments, this decoding includes decoding only the E2AP header, while in other embodiments, this decoding includes decoding the E2AP header and the E2SM payload. For some or all of this decoding (e.g., the E2SM), the near-RT RIC of some embodiments uses a hardware accelerator (e.g., a GPU accelerator) that is accessed via the bypass path described above.

[0199] After decoding the E2AP message, the near RT RIC creates or updates an internal data structure taking into account the received data message, and then creates a flat encapsulated message to the xApp in the format described above with reference to Table 1. Because the near RT RIC and the RIC SDK run on different containers, and in some embodiments on different pods, they do not pass any data structures to each other, and in some embodiments, format their data exchange into an encapsulated message having a specific byte sequence. After encapsulating the data message, the near RT RIC forwards the data message to the xApp pod, which in turn forwards it to the appropriate xApp.

[0200] Internal data structures created or updated in the near RT RIC during processing of an E2AP message are used to process response messages from xApps to the E2AP message and to process subsequent E2AP messages. In some embodiments, examples of data stored in the near RT RIC's internal data structures include: (1) subscription lists of xApps interested in data from particular E2 nodes, (2) the specific data tuples in which each xApp is interested from each E2 node, (3) records identifying network addresses and other location data associated with the E2 nodes and xApps, and (4) reserved and assigned identifiers (e.g., RIC request IDs).

[0201] When an xApp sends a message, its RIC SDK processes the message and forwards it to the RIC along the shared memory or SCTP interface as described above. The near-RT RIC then parses the message and stores the parsed components. Based on these components and one or more data tuples stored in the internal data structure of the associated E2 node message, the RIC creates an E2AP response, encodes this response, and forwards it to the destination E2 node.

[0202] For example, after a first xApp sends a subscription request to receive M data tuples from the E2 node, it creates state in a data path pod in the near-RT RIC to record the first xApp's intended subscription, request a subscription with the E2 node for the M data tuples, forward these M data tuples to the xApp the first time they are received, and thereafter forward these M data tuples as they are received. In some embodiments, the data path pod in the near-RT RIC can be configured to forward M data tuples to an associated SDL each time they are received from the E2 node.

[0203] After a first xApp subscribes to receive M data tuples from the E2 node, a second xApp can subscribe to receive N different data tuples from the E2 node, where N is greater than M. The near RT RIC sends an updated subscription request to the E2 node. This update now requests N data tuples. Each time the near RT RIC receives N data tuples, it sends M data tuples to the first xApp, and all N data tuples are sent to the second xApp.

[0204] Another example is to delete a near RT RIC and cache the RIC request ID from the E2AP message from the E2 node in response to the subscription request. When this ID is deleted, the RIC provides part of the E2AP message and its E2SM payload (if applicable) to the xApp. If the xApp subsequently deletes the subscription, the RIC retrieves the RIC request ID from its state and inserts it into an E2AP message to the E2 node requesting the subscription to be deleted.

[0205] In some embodiments, the near RT RIC's processing of E2 setup, response, and failure messages is as follows: The near RT RIC first receives a setup request message from the E2 node. In response, the near RT RIC decodes the message and builds internal data structures. The RIC also caches the raw ASN1 payload. In some embodiments, the near RT RIC accepts any RAN function identifiers that have been added. In some embodiments, after decoding the E2AP header, the near RT RIC sends the setup message to the xApps without decoding anything else (i.e., as a message with an ASN1 encoded E2SM payload). In some embodiments, the setup message that the near RT RIC sends to the xApps has a control length (ctrl_len) of 0 along with its ASN1 encoded payload.

[0206] When the xApp later connects, the near RT RIC sends all setup requests from the E2 nodes to the xApp and maintains an inventory of connected E2 nodes. In some embodiments, the near RT RIC sends these messages one at a time. Also, as described above, the near RT RIC in some embodiments constructs an E2Setup response and sends it to the E2 node. In some embodiments, the near RT RIC sends a failure message when a setup request is malformed (e.g., removes records that are duplicates of the RAN function list or that were not added to the list).

[0207] After receiving a reset from the E2 node, the near RT RIC decodes the message and then performs the following actions: Sends a message about this reset to all xApps that have subscriptions to this E2 node. In some embodiments, this is an internal message with no ASN 1 content. The near RT RIC finishes sending subscription removal messages to the E2 node for all previous subscriptions it sent. It also sends control, insert, and policy removal to this E2 node. It cleans up any outstanding requests and sends a reset response to the E2 node.

[0208] The near RT RIC also employs service update, confirmation, and failure messages in some embodiments, which update the supported RAN function list with additions, modifications, and deletions. The near RT RIC notifies all xApps about the new service configuration of the E2 node. In some embodiments, the near RT RIC sends messages to the xApps after applying the configuration, so that it reflects the final state of the configuration. In other embodiments, the near RT RIC sends messages directly to the xApps to calculate the delta between the previous and new states of the supported RAN functions. In the latter approach, the near RT RIC does not need to ASN1 encode the resulting delta.

[0209] The process for E2AP subscription is as follows in some embodiments: xApp formats the E2SM portion of the subscription and ASN1 encodes it. Table 2 below details the control portion of the subscription message (i.e., the portion stored in the control field of the message from xApp to near RT RIC). The payload will be the ASN1 encoded E2SM content. In some embodiments, multiple subscription message types are defined to clarify optional information. Also, in some embodiments, message flags are used to specify the exact format. In some embodiments, each join message specifies one E2 node global ID and one RAN function ID.

[0210] In some embodiments, the E2 node sends a 113-byte identifier, which the RIC compresses into a 40-byte ID. When sending a subscription message to the E2 node, the RIC converts the 40 bytes into a 113-byte ID. The subscription message control field is in a fixed format as much as possible. In some embodiments, the RIC caches all subscription requests and compares requests from multiple xApps to avoid sending duplicate subscription messages. However, if after the first initial xApp, a second subsequent xApp requests additional information from the same E2 node, the RIC resends the subscription (using the same RIC request ID in some embodiments), but this time requesting additional information. When sending a subscription request to the E2 node, the RIC in some embodiments sends the entire payload received from the xApp as the E2AP message payload. [Table 2]

[0211] The near-RT RIC processes the E2AP RIC subscription response by storing the E2 node's global ID and the RIC request ID (generated by the RIC) as control information and sending the correctly ASN1 encoded message from the E2 node back to the xApps.

[0212] An E2AP RIC subscription deletion request, response, or failure message is sent from the xApp to the near RT RIC with message fields sent as control information (i.e., as part of ctrl_info). The near RT RIC creates an encoded ASN1 message and sends it to the E2 node. The deletion request does not specify the E2 node global ID; therefore, this information is provided by the xApp's RIC. In some embodiments, the response message is sent from the near RT RIC to the xApp as packed bytes (not ASN1 encoded).

[0213] The E2 AP Notification Report for an E2 node is processed as follows: The message is decoded by the near RT RIC and the RIC Request ID field is determined. This helps to determine the xApp that has subscribed to the indication. The near RT RIC in some embodiments sends the message as ASN1 encoded to the xApp. In some embodiments, the near RT RIC also sends the reduced E2 Global ID as control information with the message.

[0214] Near RT RIC processing of an E2AP control request is as follows in some embodiments: the xApp sends this request as packed bytes of information. In the near RT RIC, no E2 Global ID is specified in the message since this information is specified by the xApp. The near RT RIC format of this message is shown in Table 3. [Table 3]

[0215] The near-RT RIC processes the E2AP control response or failure message as follows: The near-RT RIC decodes the message and obtains the RIC request ID. It then sends an ASN1 encoded message, with the global E2 node ID prepended as control information, to the xApp.

[0216] In some embodiments, the SDL data store is an in memory database that runs on a unique set of one or more pods. It has its own computing and memory resources allocated to it. As previously mentioned, multiple near-RT RIC instances define a distributed near-RT RIC. In some embodiments, each near-RT RIC instance has its own instance of the SDL that stores system-wide information for the RIC instance. Examples of such information include connected E2 nodes (i.e., base station nodes), xApps, subscriptions from each xApp, and a list of critical cell data returned by the E2 nodes. Additionally, each SDL instance of some embodiments runs custom algorithms internally as the data arrives, and provides services to pre-process incoming data by interfacing to hardware accelerators (e.g., GPUs) or by post-processing data retrieved from its storage.

[0217] The data IO pods and xApp pods of a RIC instance are connected to the SDL pods of the RIC instance. In some embodiments, each SDL instance runs only on the data IO pods and service pods of its own RIC instance. Also, in some embodiments, the SDL pods are managed by SMO and configured through service pods. Data flows to and from the SDL include (1) data IO to SDL data storage, (2) xApps from SDL data storage, (3) xApps to SDL data storage, (4) data IO from SDL data access (e.g., retrieving E2 node information, subscription information), and (5) service pods to and from SDL communication to provide and retrieve configuration information.

[0218] 34 illustrates a RIC instance 3400 of some embodiments having an SDL pod 3410. As illustrated, the SDL 3410 includes a configuration agent 3412, an SDL datapath agent 3414, a RIC SDK agent 3417, SDL pre- and post-processors 3416 and 3418, and one or more SDL data stores 3420. The SDL configuration agent 3412 interfaces with a service pod agent 3430, which configures and manages the other components of the SDL pod. The SDL pod is managed by the SMO and configured via the service agent 3430 of service pod #090.

[0219] The SDL datapath agent 3414 is the datapath interface that the SDL pod exposes to the control and datapath threads of the datapath pod 3450. The SDL datapath agent 3414 handles communications from these entities on behalf of SDL and performs reads and writes to the SDL data store 3420 for these datapath entities. In some embodiments, the SDL datapath agent can be configured to communicate with the datapath pod 3450 using either SCTP or a shared memory library.

[0220] In some embodiments, the RIC SDK agent 3417 is an agent that the SDL pod exposes to the RIC SDK of the xApp pod. The RIC SDK agent 3417 handles communication from the RIC SDK to the SDL and performs reads and writes to the SDL data store 3420 for the RIC SDK. In some embodiments, the RIC SDK agent 3417 can be configured to communicate with the RIC SDK using either SCTP or a shared memory library. This agent 3417 also performs cache synchronization operations to synchronize the SDL cache of the RIC SDK with the SDL data store 3420. Also, when scaling to tens of connections from xApps is required, the connection manager of some embodiments utilizes the epoll connection handling of the RIC instance (e.g., the epoll connection handling used by the data IO pod of some embodiments).

[0221] The SDL agents 3414 and 3417 handle event subscriptions and notifications from the RIC SDK and Data Pass pods. This is separate from the E2AP subscription management, but conceptually similar. For example, through this subscription service in SDL, an xApp specifies interest in some data via a key or report frequency. The RIC SDK agent 3417 then provides periodic updates to this xApp based on the subscription. It also provides security services in some embodiments by encrypting and decrypting data.

[0222] Data pre- and post-processors 3416 and 3418 are part of the SDL 3410 in some embodiments to flexibly run specific value-add algorithms on the data. In some embodiments, both of these processors 3416 and 3418 run in one container, while in other embodiments, each of them runs in a separate container within the SDL pod 3410. Each of these processors also interfaces with an external accelerator (e.g., GPU 3480) to perform their operations in some embodiments. The data pre-processor 3416 runs inline as the data is stored in the SDL data store 3420.

[0223] In some embodiments, the data post-processor 3418 runs inline as data is read from the SDL data store 3420. Alternatively, the post-processor 3418 in some embodiments can be configured to run in the background on data stored in the SDL data storage 3420 (e.g., retrieve data from this data storage, perform some operation on this data, and store the result in the data storage). The data processor of some embodiments encodes and / or decodes the E2SM payload of the E2AP message. This is advantageous because the data pass pod can pass the ASN1 string to SDL for decoding and storage. As mentioned above, the RIC SDK in some embodiments can also be configured to provide E2SM encoding / decoding services.

[0224] Another example of a post-processor operation that the data processor 3418 performs in some embodiments is a machine trained operation. In some embodiments, the post-processor 3418 collects various data tuples stored by various xApps and / or various pods (e.g., datapath pods) and passes these data tuples through a machine learning network (e.g., a neural network trained by machine learning). In some embodiments, the post-processor 3418 performs its operations using one or more hardware accelerators (e.g., one or more GPUs) of the SDL host computer to run its machine training network. The post-processor 3418 accesses the hardware accelerators through the bypass approach described above with reference to Figures 14-20.

[0225] The post-processor 3418 can pass the results of its operations to one or more xApps or store the results in the SDL data store 3420 for retrieval by one or more xApps. Examples of results obtained by post-processing the SDL data in a machine-trained network include anomaly detection (e.g., identifying E2 nodes that are behaving abnormally, e.g., a cell site that suddenly receives too many connections).

[0226] Using a machine trained network in the SDL pod 3410 to process the different xApp outputs stored in the SDL data store 3420 is advantageous because this data store 3420 stores the outputs of several xApps as well as the many data tuples that the E2 node provides based on the xApp subscription. In many cases, each individual xApp only knows the data tuples to which it subscribes, as well as its own computation results and the outputs of several other xApps. Meanwhile, the SDL pod 3410 has access to a larger set of E2 node input data and xApp output data. Instead of using a machine trained network to perform such post-processing, the post-processor 3418 uses an algorithm (e.g., a constrained optimization solver) to post-process the data stored in the SDL data storage 3420. In other words, the post-processor 3418 of some embodiments does not use a machine trained network and still uses its host computer's hardware accelerators (e.g., via a bypass path) to perform its operations.

[0227] Some embodiments also use the post-processor 3418 to provide the current state of the E2 node when the first xApp begins subscribing to the state of the E2 node. After previously subscribing to the second xApp to receive the state of the E2 node, the near RT RIC stores multiple data tuples related to the state of this node for a period of time. When the first xApp subsequently subscribes to the state of the E2 node and either this xApp or a datapath pod attempts to access this state of the first xApp, the post-processor 3418 retrieves all data tuples previously stored for this E2 node in the SDL storage and uses these data tuples to calculate the current state of the E2 node, which it then provides to the first xApp directly or via a datapath pod.

[0228] The SDL data store 3420 is an in-memory database. In some embodiments, the memory in memory database is a database that is loaded into and run from computer system memory (e.g., the host computer non-volatile memory, including its RAM). One example of such a database is Redis. In some embodiments, the size of the data storage is selected to minimize search and storage latency. When an existing data storage reaches its desired maximum size, some embodiments create an additional instance of this data storage within the same or a different instance of SDL.

[0229] Also as shown in FIG. 34, the SDL 3410 of some embodiments has an active data storage 3420 and a standby data storage 3421 for HA reasons. Additionally, some embodiments allow different SDL instances in different RIC instances to sync some or all of their data in the background for HA and / or data availability reasons to the data storage 3420. In some embodiments, xApps and RIC components read and write data to the active SDL storage which is synced with the standby SDL storage in the background. In the event of a failure of the active SDL storage, the RIC of these embodiments can seamlessly switch to the standby SDL storage. Also, the RIC can switch to the standby SDL storage when the active SDL storage is upgraded. This allows the SDL storage to be hitless.

[0230] As previously described with reference to FIG. 13, the RIC SDK includes an SDL cache that provides local SDL storage for xApps, and synchronizes its data with the SDL's data store. In some embodiments, the RIC SDK cache pulls data from the main SDL data store in bulk and periodically, which can reduce the number of requests made to the main SDL pod. The RIC SDK cache also provides some of this data locally, reducing the latency for reading SDL data. The RIC SDK cache also reduces the time to write data to the SDL by allowing data to be written locally at the xApp pod and synced in the background. The size of the SDK cache in some embodiments is smaller than the SDL's data storage 3420. In some embodiments, the size is based on the requirements of the set of one or more xApps that run on the pod along with the RIC SDK.

[0231] In some embodiments, sources of data to the SDL 3410 include: (1) control threads and datapath threads of the datapath pod 3450, (2) xApps via the RIC SDK, (3) ML model and policy storage and servers in non-RT RICs accessed via the A1 interface, (4) xApp pod configuration storage, and (5) configuration servers in the SMO. In some embodiments, examples of system data in the SDL (e.g., written by the control thread) include: (1) E2 node information, (2) cell information, (3) UE information, (4) E2 node reports, (5) KPM (key performance metrics) reports, and (6) topology information (from the EMS adapter).

[0232] Examples of SDL transactions in some embodiments include: (1) a Data IO Pod (control or Data IO thread) that writes data to SDL that is read by an xApp; (2) an xApp that reads data from another xApp that is written to SDL, or that writes data to SDL for another xApp; (3) an xApp that writes to SDL so that a service container (e.g., a post processor) running in an SDL pod performs an operation on data previously written by the same xApp or another xApp, and retrieves the results of this operation from SDL (e.g., by using GPU services or simply using a general CPU); (4) a non-RT RIC that reads data from and writes data to SDL as part of an A1 subscription; (5) an SMO that stores 01 configuration data in SDL; and (6) a non-RT RIC that stores ML data in SDL.

[0233] 35 conceptually illustrates an electronic system in which some embodiments of the present invention may be implemented. The electronic system 3500 may be a computer (e.g., a desktop computer, a personal computer, a tablet computer, a server computer, a mainframe, a blade computer, etc.), or any other type of electronic device. Such an electronic system 3500 includes various types of computer readable media and interfaces with various other types of computer readable media. The electronic system 3500 includes a bus 3505, a processing unit 3510, a system memory 3525, a read-only memory 3530, a persistent storage device 3535, input devices 3540, and output devices 3545.

[0234] The bus 3505 collectively represents all system buses, peripheral buses, and chipset buses that communicatively connect the various internal devices of the electronic system 3500. For example, the bus 3505 communicatively connects the processing unit 3510 with the read-only memory 3530, the system memory 3525, and the persistent storage device 3535.

[0235] From these various memory units, the processing unit 3510 retrieves instructions to execute and data to process in order to perform the processes of the present invention. The processing unit 3510 may, in various embodiments, be a single processor or a multi-core processor.

[0236] The read-only memory 3530 stores static data and instructions required by the processing unit 3510 and other modules of the electronic system 3500. The persistent storage device 3535, on the other hand, is a readable and writable memory device. This device 3535 is a non-volatile memory unit that stores instructions and data even when the electronic system 3500 is turned off. Some embodiments of the invention use mass storage devices (such as magnetic or optical disks and their corresponding disk drives) as the persistent storage device 3535.

[0237] In other embodiments, a removable storage device (such as a floppy disk, flash drive, etc.) is used as the persistent storage device 3535. Like the persistent storage device 3535, the system memory 3525 is a readable and writable memory device. However, unlike the storage device 3535, the system memory 3525 is a volatile readable and writable memory, such as a random access memory. The system memory 3525 stores some of the instructions and data that the processor needs during execution. In some embodiments, the processes of the present invention are stored in the system memory 3525, the persistent storage device 3535, and / or the read-only memory 3530. From these various memory units, the processing unit 3510 retrieves instructions to execute and data to process in order to perform the processes of some embodiments.

[0238] The bus 3505 also connects to input / output devices 3540 and 3545. The input devices 3540 allow a user to communicate information and select commands to the electronic system 3500. The input devices 3540 include alphanumeric keyboards and pointing devices (also called "cursor control devices"). The output devices 3545 display images generated by the electronic system 3500. The output devices 3545 include printers and display devices such as cathode ray tubes (CRT) or liquid crystal displays (LCD). Some embodiments include devices such as touch screens that function as both input and output devices.

[0239] 35, the bus 3505 also couples the electronic system 3500 to a network 3565 via a network adapter (not shown). In this manner, the computer may be part of a network of computers, such as a local area network ("LAN"), a wide area network ("WAN"), an intranet, or the Internet. Any or all of the components of the electronic system 3500 may be used with the present invention.

[0240] Some embodiments include electronic components, such as microprocessors, storage and memory that store computer program instructions in a computer-readable or computer-readable medium (alternatively referred to as a computer-readable storage medium, machine-readable medium, or machine-readable storage medium). Some examples of such computer-readable media include RAM, ROM, read-only compact disks (CD-ROMs), recordable compact disks (CD-Rs), rewritable compact disks (CD-RWs), read-only digital versatile disks (e.g., DVD-ROMs, dual-layer DVD-ROMs), various recordable / rewritable DVDs (e.g., DVD-RAMs, DVD-RWs, DVD+RWs, etc.), flash memories (e.g., SD cards, mini-SD cards, micro-SD cards, etc.), and / or solid-state hard drives, read-only and recordable Blu-Ray® disks, ultra-density optical disks, other optical or magnetic media, and floppy disks. The computer-readable medium can store a computer program that is executable by at least one processing unit and includes a set of instructions for performing various operations. Examples of computer programs or computer code include machine code, such as produced by a compiler, and files containing higher level code that are executed by a computer, electronic component, or microprocessor using an interpreter.

[0241] While the above description primarily refers to microprocessors or multi-core processors executing software, some embodiments are performed by one or more integrated circuits, such as application specific integrated circuits (ASICs) or field programmable gate arrays (FPGAs). In some embodiments, such integrated circuits execute instructions stored on the circuit itself.

[0242] As used herein, the terms "computer," "server," "processor," and "memory" all refer to electronic or other technological devices. These terms exclude a person or group of people. In the specification, the terms display or display refer to a display on an electronic device. As used herein, the terms "computer-readable medium"(s), "computer-readable medium," and "machine-readable medium"(s) are generally limited to tangible physical objects that store information in a form readable by a computer. These terms do not include any wireless signals, wired downloaded signals, and any other ad-hoc signals.

[0243] Although the present invention has been described with reference to numerous specific details, those skilled in the art will recognize that the present invention may be embodied in other specific forms without departing from the inventive concept. For example, some of the figures conceptually illustrate processes. The specific operations of these processes may not be performed in the exact order shown and described. The specific operations may not be performed in one continuous series of operations, and different specific operations may be performed in various embodiments. Furthermore, the processes may be implemented using several sub-processes or as part of a larger macro process.

[0244] Also, some of the embodiments described above show only one hardware accelerator per host computer. However, one of ordinary skill in the art will recognize that the methodology and architecture of some embodiments can be used to provide direct pass-through access to multiple hardware accelerators on one host computer. Furthermore, some of the embodiments described above relate to xApp operations and near-RT RIC communication with xApps. One of ordinary skill in the art will recognize that these embodiments are equally applicable to edge applications in a telecommunications network and near-RT RIC communication with edge applications. Thus, one of ordinary skill in the art will recognize that the present invention is not limited by the foregoing exemplary details, but rather is defined by the appended claims.

Claims

1. 1. A method for performing control plane operations in a radio access network (RAN) including a host computer on which a plurality of machines operate, the method comprising: On each machine Deploying a control plane application that performs control plane operations; and configuring a RAN Intelligent Controller (RIC) software development kit (SDK) to forward messages between the control plane application and a RAN Intelligent Controller (RIC) on a same machine, the RIC forwarding messages between the control plane application on the deployed machine and a set of one or more base station components of the RAN.

2. 2. The method of claim 1, wherein on each machine, the RIC SDK includes a set of network connectivity processes that establish network connectivity to the set of base station components of the RAN for the control plane application, and enables the control plane application on the machine to forego having the set of network connectivity processes.

3. 3. The method of claim 2, wherein the set of network connectivity processes in each RIC SDK of each machine establishes and maintains network connectivity between the machine and the set of base station components of the RAN used by the control plane application of the machine, and controls data packet transmissions to and from the set of base station components of the RAN for the control plane application.

4. 2. The method of claim 1, wherein the control plane application on each machine communicates with the set of base station components of the RAN via high-level Application Programming Interface (API) calls that the RIC SDK translates to low-level API calls.

5. The method of claim 4 , wherein at least a subset of the low-level API calls are defined by a standards development body.

6. 5. The method of claim 4, wherein the high-level API calls are made in a high-level programming language, while the low-level API calls include low-level calls for establishing and maintaining network connections and passing data packets through those connections.

7. 5. The method of claim 4, wherein the set of base station components of the RAN includes a centralized unit (CU) and a distributed unit (DU) of the RAN.

8. 8. The method of claim 7, wherein the RIC SDK on each machine communicates with the CU and the DU via the low-level E2 interface, while the control plane application on the machine communicates with the CU and the DU via the RIC SDK using high-level API calls that define E2 interface operations at a high-level application layer that do not include low-level transmission or network operations.

9. The method of claim 4 , wherein the set of base station components of the RAN includes network elements of the RIC.

10. 10. The method of claim 9, wherein the RIC element includes at least one Shared Data Layer (SDL) element.

11. 10. The method of claim 9, wherein the RIC element includes at least one data path input / output (I / O) element.

12. 10. The method of claim 9, wherein the RIC element includes at least one service management element.

13. A computer program which, when executed on at least one processing unit, performs the method according to any one of claims 1 to 12.

14. 1. A computing device, comprising: A set of processing units; and a machine-readable medium storing a program which, when executed by at least one of said processing units, performs the method of any one of claims 1 to 12.

Images