RoCE multi-tenant management system based on SDN

Abstract

The invention relates to the technical field of cloud service systems, in particular to an SDN-based RoCE multi-tenant management system, which comprises the steps that a control node interacts with a working node, determines network configuration for a newly added tenant container, translates according to a logic network object pointed by the network configuration to obtain a logic data path flow table, and sends the logic data path flow table to the working node; the DPU node generates a flow table rule in accordance with the logical data path flow table. In order to solve the problem that the RoCE performance is reduced due to the fact that an existing tenant management system needs to consume long time to perform rule searching and routing when data writing is performed among multiple containers, the three nodes are introduced, and the RoCE performance is improved by determining a network model of a current pod network for a newly-added tenant container. Therefore, the network resources of the current network are accurately described, the related resources allocated to the tenant container are determined, the DPU node is controlled to route the data through the logic path data flow table, the more efficient inter-container communication efficiency is achieved, and the related network requirements of the RoCE process are met.

Description

Technical Field

[0001] This invention relates to the field of cloud service system technology, and more specifically to a RoCE multi-tenant management system based on SDN. Background Technology

[0002] A tenant management system is a core component used by cloud service providers (CSPs) to manage customers (tenants) and the resources they use on the cloud platform. Based on a specific system architecture, the tenant management system implements functions such as tenant lifecycle management, identity and access management, resource quota allocation and billing, and security isolation. Taking a typical scenario as an example, when a user creates an instance on a cloud service platform, the tenant management system needs to generate a virtualized container around that user and allocate certain resources to the container, including processors, bandwidth, and GPU computing resources, to run the instance within the container. Simultaneously, considering different user needs, it should also provide functions such as sandbox data isolation, external access control, and data interaction between multiple containers. Billing is based on the various resources used by the user, and unauthorized related behaviors are isolated.

[0003] There are already many tenant management systems in the existing technology.

[0004] For example, patent application CN202010017855.0 discloses a multi-tenant management system and its implementation method, relating to the field of computer technology. One specific implementation of this system includes: the system comprises an OpenTSDB cluster, a management module, and an authentication module; the OpenTSDB cluster is used to deploy at least one application; the management module, located on the OpenTSDB cluster, is used to receive tenant creation requests and create tenants and their corresponding users, user groups, and roles; the authentication module is used to receive user login requests and authenticate the user to be logged in based on the login request. This implementation optimizes data storage, significantly improves data query efficiency, reduces storage space usage, achieves complete resource isolation of the OpenTSDB cluster, and enhances user experience.

[0005] For example, patent application CN202110797687.6 discloses a tenant resource management method and a tenant management system applicable to the cloud computing field. The method includes: statistically analyzing one or more request messages from tenants within a given period; identifying one or more target request messages based on preset information types (such as connection, query, write, and subscription); determining the tenant's target activity level based on the target request messages; and adjusting the tenant's resources on the resource sharing platform if the tenant's current activity level differs from the target activity level. The method calculates the tenant's activity level within the statistical period based on their request information and implements dynamic, differentiated management of reserved resources for different tenants based on their activity level. This avoids inactive tenants (i.e., zombie tenants) wasting reserved resources, enabling the resource sharing platform to support more tenants simultaneously with the same system resources, thus minimizing the operating costs of zombie tenants.

[0006] However, in actual implementation, the inventors found that when the above-mentioned tenant management is applied to the RoCE system, since RoCE users usually have high concurrency and large amount of data exchange needs between multiple containers, the traditional tenant management system will cause a large delay in internal auditing and data routing, resulting in the problem that the service performance does not meet the requirements. Summary of the Invention

[0007] To address the aforementioned problems in existing technologies, a RoCE multi-tenant management system based on SDN is provided.

[0008] The specific technical solution is as follows: A RoCE multi-tenant management system based on SDN includes worker nodes, control nodes, and DPU nodes built on a Kubernetes cluster; The DPU node runs multiple tenant containers, which are used to perform different data processing tasks. The control node interacts with the worker node to determine the network configuration for the newly added tenant container; The control node translates the logical data path flow table according to the logical network object pointed to by the network configuration; The DPU node listens to the control node and generates flow table rules according to the logical data path flow table when the logical data path flow table appears. The DPU node assigns network addresses to the newly added tenant container according to the flow table rules and controls packet forwarding to complete the addition of the tenant container.

[0009] On the other hand, the working node includes: The proxy component module is connected to the Kubernetes control plane and network infrastructure of the Kubernetes cluster, respectively. The proxy component module collects data from the network infrastructure to confirm the network model within the network infrastructure. The working node negotiates with the control node according to the network model to determine the network configuration corresponding to the newly added tenant container.

[0010] On the other hand, the Kubernetes control plane obtains new container information based on the kube-apiserver to detect the tenant container.

[0011] On the other hand, the Kubernetes control plane invokes the proxy component module to create a sandbox for the tenant container on the DPU node.

[0012] On the other hand, the working node writes the network configuration to the control node based on the invoked proxy component module.

[0013] On the other hand, the control node includes: an SDN controller, an OVN northbound interface, an OVN conversion module, and an OVN southbound interface; The SDN controller writes the network configuration to the OVN northbound interface; The OVN conversion module monitors the database of the OVN northbound interface to determine whether the network configuration is written to the OVN northbound interface. When the network configuration is written to the OVN northbound interface, the OVN conversion module calls the OVN conversion module to generate the logical path data flow table and stores it in the OVN southbound interface to send it to the DPU node.

[0014] On the other hand, the OVN conversion module extracts logical network objects from the network configuration and generates the logical path data flow table according to the data flow direction between the logical network objects.

[0015] On the other hand, the DPU node includes: OVN controller, the OVN controller obtains the newly generated logical path data flow table in the control node; A virtual switch, which forwards data to the tenant container according to the logical path data flow table.

[0016] The above technical solution has the following advantages or beneficial effects: To address the issue that existing tenant management systems require significant time for rule lookup and routing when writing data across multiple containers, leading to RoCE performance degradation, the aforementioned three nodes were introduced. For newly added tenant containers, the network model of the current pod network is determined, thereby accurately describing the network resources of the current network and deciding on the relevant resources to allocate to the tenant container. Then, the DPU node is controlled by the logical path data flow table to route the data, achieving more efficient inter-container communication and meeting the relevant network requirements of the RoCE process. Attached Figure Description

[0017] Embodiments of the invention will be described more fully with reference to the accompanying drawings. However, the drawings are for illustration and explanation only and do not constitute a limitation on the scope of the invention.

[0018] Figure 1 This is an overall schematic diagram of an embodiment of the present invention; Detailed Implementation

[0019] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0020] It should be noted that, unless otherwise specified, the embodiments and features described in the present invention can be combined with each other.

[0021] The present invention will be further described below with reference to the accompanying drawings and specific embodiments, but this is not intended to limit the scope of the invention.

[0022] This invention includes: A RoCE multi-tenant management system based on SDN includes worker node 1, control node 2 and DPU node 3 built on a Kubernetes cluster; DPU node 3 runs multiple tenant containers, which are used to perform different data processing tasks. Control node 2 interacts with worker node 1 to determine the network configuration for the newly added tenant container; Control node 2 translates the logical data path flow table based on the logical network object pointed to by the network configuration; DPU node 3 listens to control node 2 and generates flow table rules according to the logical data path flow table when a logical data path flow table appears. DPU node 3 assigns network addresses to newly added tenant containers according to flow table rules and controls packet forwarding to complete the addition of tenant containers.

[0023] Specifically, to address the issue that existing tenant management systems require significant time for rule lookup and routing when writing data across multiple containers, leading to a decrease in RoCE performance, the aforementioned three nodes were introduced. For newly added tenant containers, the network model of the current pod network is determined, thereby accurately describing the network resources of the current network and deciding on the relevant resources to allocate to the tenant container. Then, the DPU node is controlled by the logical path data flow table to route the data, achieving more efficient inter-container communication and meeting the relevant network requirements of the RoCE process.

[0024] Specifically, the aforementioned tenant management system is configured in a cloud service system with RoCE functionality during actual application.

[0025] RoCE (RDMA over Converged Ethernet) is a network protocol that allows Remote Direct Memory Access (RDMA) over standard Ethernet networks. Cloud service systems based on this technology enable users to exchange data between multiple virtualized containers via RDMA, thereby significantly reducing data exchange latency.

[0026] To achieve unification between the application and hardware layers, this cloud service system adopts the Kubernetes network stack, comprising three parts: "application-driver-hardware," controlled by the OVN control plane, Kubernetes control plane, and CNI control plane, respectively. The Kubernetes control plane is the highest-level control plane, oriented towards applications and resources. The OVN control plane is the control plane that implements advanced virtual network functions, responsible for translating the network intents of the Kubernetes control plane into underlying forwarding rules. The CNI control plane provides the interface standards and execution framework, acting as an adapter between the Kubernetes control plane and the specific network hardware.

[0027] The relationship between the OVN control plane, Kubernetes control plane, and CNI control plane is a precise collaborative one of "decision-translation-execution." The Kubernetes control plane drives the CNI, actively calling CNI plugins through the kubelet. Their relationship follows a "command-execution" model. The CNI's processing logic is reactive, defining a standard interface protocol that awaits kubelet calls. Its role is adaptation, "translating" Kubernetes' general network requirements into requests that OVN can understand. The OVN control plane processes API requests from the CNI, performs specific network resource allocation and logical configuration (creating ports, assigning IPs), and returns the results. Through its built-in controller, it continuously monitors and processes changes to network resources in the Kubernetes API Server.

[0028] The data transmission between the OVN control plane, the K8s control plane, and the CNI control plane is not the transmission of user data, but the exchange of control information (instructions, configurations, and states). It presents a top-down, layer-by-layer unidirectional instruction flow, with the direction being K8s→CNI→OVN. That is, the higher layer (K8s) issues abstract instructions to the lower layer (CNI), the lower layer (CNI) converts them into more specific instructions and passes them to the implementation layer (OVN), and OVN finally turns them into executable flow table rules.

[0029] The flow of state information is bottom-up. OVN and kubelet continuously feed back their own state to the K8sAPI Server to ensure that the state of the entire cluster is consistent with the expected value.

[0030] Based on the three control planes mentioned above, the master node, DPU card, and host can all be added as nodes to the Kubernetes cluster network to achieve the corresponding functions.

[0031] Taking the creation of a tenant container in a pod network as an example, a tenant container is essentially a virtualized machine running in a computer cluster based on a sandbox mechanism. The tenant container needs to connect to the virtual network to exchange data with other containers, including RoCE communication and other types of communication processes.

[0032] In a Kubernetes cluster, each tenant container has a unique IP address, and all tenant containers can communicate directly through this IP address. This network model is called a pod network.

[0033] When a new tenant container is added, sandbox virtualization, container creation, and adding the tenant container to the pod network need to be performed simultaneously to form a new network model. Considering that the tenant container will subsequently perform RoCE communication, routing rules for the tenant container to point to other containers need to be configured. To this end, control node 2 first interacts with worker node 1 to determine the network configuration for the newly added tenant container. Then, control node 2 translates the logical data path flow table based on the logical network object pointed to by the network configuration. Finally, DPU node 3 listens to control node 2 and generates flow table rules according to the logical data path flow table when it appears. In the actual communication process, DPU node 3 allocates network addresses to the newly added tenant container and controls packet forwarding according to the flow table rules. The addition of a tenant container is completed through the above process.

[0034] In one embodiment, working node 1 includes: Proxy component module 11 connects to the Kubernetes control plane and network infrastructure of the Kubernetes cluster. The proxy component module 11 collects data from the network infrastructure A1 to confirm the network model within the network infrastructure; Worker node 2 negotiates with the control node to determine the network configuration corresponding to the newly added tenant container according to the network model.

[0035] Specifically, to implement the above process, working node 1 includes a proxy component module 11, which is implemented using Node-CNI.

[0036] In the network model of this application, Node-CNI is a proxy component running as a DaemonSet on each worker node (Host) of the cluster. It is a concrete implementer of the Kubernetes CNI (Container Network Interface) standard and a key bridge connecting the Kubernetes control plane and the node's local network infrastructure (such as the Linux kernel, bridges, OVS, etc.). Its core mission is to respond to the lifecycle events of Pods on the local node and configure, manage, and clean up network resources for them.

[0037] Network infrastructure A1 corresponds to the network model of the aforementioned pod network, including the IP address and port of each container.

[0038] Based on the aforementioned working node 1, the network configuration generation process includes: The Kubernetes control plane obtains information about newly added containers based on the kube-apiserver in order to become aware of tenant containers.

[0039] First, the kube-scheduler in the Kubernetes control plane schedules a container to a specific worker node 1. This scheduling process corresponds to the creation of a tenant container.

[0040] Subsequently, the Kubernetes control plane on worker node 1 becomes aware of the newly added tenant container from the kube-apiserver. The Kubernetes control plane then prepares to create a sandbox for the tenant container on DPU node 3, and then calls CNI plugins according to the configuration, such as / opt / cni / bin / ovn-k8s-cni-overlay, etc. These CNI plugins are integrated into the call proxy component module 11.

[0041] Once created, the CNI plugin actually invoked in the proxy component module 11 will act as an OVN client and send a request to control node 1 to write the network configuration.

[0042] In one embodiment, the control node 2 includes: an SDN controller 21, an OVN northbound interface 22, an OVN conversion module 23, and an OVN southbound interface 24; SDN controller 21 writes network configuration to OVN northbound interface 22; The OVN conversion module 23 monitors the database of the OVN northbound interface 22 to determine whether the network configuration is written to the OVN northbound interface 22. When the network configuration is written to the OVN northbound interface 22, the OVN conversion module 23 calls the OVN conversion module 23 to generate a logical path data flow table and stores it to the OVN southbound interface 24 to send it to the DPU node 3.

[0043] Specifically, in order to realize the conversion between network configuration and logical flow table, this embodiment adds an SDN controller 21, an OVN northbound interface 22, an OVN conversion module 23, and an OVN southbound interface 24 to the control node.

[0044] Specifically, the SDN controller 21 is an application program in Software-Defined Networking (SDN). In an SDN network, it is responsible for managing and controlling the behavior of the entire network hardware through programmatic management, thereby achieving flexible and centralized network management. The controller communicates with network devices to obtain real-time network topology and link status information, distributes flow table rules to network devices, controls data flow and forwarding logic, and can quickly detect network device faults, enabling rapid failover and recovery.

[0045] In an SDN network, the SDN controller 21 is responsible for issuing flow table rules to the control plane, rather than directly processing data packets. This separation of the control and data planes makes the network more flexible and programmable. The controller can dynamically adjust rules according to network needs without modifying network devices. Secondly, through a flexible flow table management mechanism, the controller can perform fine-grained control of network traffic, enabling targeted traffic guidance and policy restrictions. It can also dynamically adjust based on network load to ensure efficient network operation. The controller also plays a crucial role in load balancing and failover, ensuring network stability and reliability. Through reasonable load balancing strategies and failover mechanisms, the controller can effectively handle device failures and anomalies in the network. The controller provides rich interfaces and communication mechanisms, allowing applications to interact and communicate with it to provide various network functions.

[0046] The OVN Northbound Interface 22 refers to the northbound interface of OVN resources, that is, the communication interface between the SDN controller 21 and the upper-layer application. It is the starting point of the SDN controller 21 and the entry point for user interaction, allowing applications to interact with network devices through APIs. The database of the OVN Northbound Interface 22 stores the configuration information of all logical network objects, mainly including logical switches, logical routers, logical ports, access control lists, etc.

[0047] The core function of OVN Northbound Interface 22 is as the interface between OVN and the CMS cloud management system. It provides a set of abstract logical network objects, allowing users to define virtual network topologies using familiar concepts (such as switches and routers) without needing to concern themselves with the specific implementation of the underlying physical or virtual infrastructure. This allows users to define rich network services and policies for the aforementioned logical networks, enabling advanced network functions. OVN Northbound Interface 22 provides a programmatic interface for external systems to call, serving as the primary means of interaction between the CMS cloud management platform and OVN.

[0048] OVN conversion module 23 is a crucial core background process in the OVN architecture. It is mainly responsible for converting the high-level configuration of OVN into logical configurations for use by the background processes of OVN controller 21, that is, converting them into flow tables in the logical path of the database of OVN southbound interface 24.

[0049] OVN translation module 23 understands the high-level abstract concepts in OVN Northbound Interface 22 and converts them into logical data path flow tables in OVN-Sb. It performs "translation" and "synchronization" between the OVN northbound and southbound databases, serving as a crucial bridge connecting user intent and network implementation. OVN translation module 23 continuously monitors changes in OVN Northbound Interface 22. When a user creates or modifies logical network objects (such as switches, routers, and ACL rules) in OVN Northbound Interface 22, OVN translation module 23 "compiles" these high-level, declarative configurations into lower-level, data flow-oriented logical network rules and writes the compiled rules to OVN Southbound Interface 24.

[0050] OVN Southbound Interface 24 refers to the southbound database in the OVN architecture. It is the core hub connecting the control logic (OVN transformation module 23) and the distributed data plane (SDN controller 21), and contains the logical and physical configuration and status of OVN.

[0051] The OVN southbound interface 24 acts as the core hub connecting the abstract logical network and the concrete physical infrastructure in OVN. By storing and distributing the logical flow tables translated by the OVN translation module 23, maintaining the dynamic binding relationship between all virtual ports and physical computing nodes, and recording the tunnel endpoint information of the entire network, it provides the OVN controller 31 distributed on each DPU node 3 with the global network view required for decision-making, thereby ensuring that the logical network intent can be consistently and correctly executed in a distributed environment.

[0052] Based on the above settings, after the network configuration is generated, the CNI plugin that is actually called in the proxy component module 11 will act as an OVN client and send a request to the SDN controller 21 in the control node 1 to write the network configuration.

[0053] SDN controller 21 writes the network configuration to OVN northbound interface 22.

[0054] Subsequently, the core daemon of the OVN conversion module 23 continuously monitors changes in OVN-Nb, translates the high-level logical network objects in the OVN northbound interface 22 into logical data path flow tables, and writes these flow tables into the OVN southbound interface 24.

[0055] In one embodiment, DPU node 3 includes: OVN controller 31, OVN controller 31 acquires the newly generated logical path data flow table in the control node; Virtual switch 32 forwards tenant containers according to the logical path data flow table; Acceleration processing module 33 manages the hardware acceleration function of the DPU node.

[0056] Specifically, the OVN controller 31 is a local agent daemon running on each DPU node in the OVN architecture, responsible for managing the network devices on that node. The OVN controller is the cornerstone of each DPU node and the key to OVN's distributed control capabilities, perfectly combining the simplicity of centralized management with the high performance and reliability of distributed execution.

[0057] The OVN controller 31 obtains global logical flow tables and network status by connecting to the OVN southbound interface 24, and translates and compiles these abstract rules into specific OpenFlow flow tables and configurations, which are then distributed to the local virtual switch 32. Simultaneously, it is responsible for managing local ports (such as registering virtual machine port binding information with the central control), handling tunnel encapsulation and decapsulation, and responding to network changes in real time to ensure that logical network policies are consistently and efficiently implemented across all nodes in a distributed environment.

[0058] Virtual Switch 32 is an open-source, high-performance virtual switch designed specifically for virtualized environments. It implements standard switch functions (such as VLAN, STP, QoS) and supports management protocols (such as OpenFlow and OVSDB).

[0059] As the core execution engine of the data plane, the virtual switch 32 provides local switching functions such as virtual port connection, VLAN isolation, and traffic policy enforcement for virtual machines on the compute nodes, and handles cross-node network traffic through tunnel encapsulation (such as Geneve and VXLAN). On the gateway node, it plays the role of a bridge between the virtual network and the physical network, responsible for routing and forwarding and NAT translation of north-south traffic. All its forwarding behavior is centrally controlled by the OpenFlow flow table issued by the OVN controller 31, thereby efficiently fulfilling the data forwarding requirements of the OVN logical network.

[0060] The acceleration processing module 33 runs the DPU node operating system and a complete OVN data plane (OVS) and control plane agent (ovn-controller), which performs centralized, hardware-accelerated processing of all network functions, thus making the host operating system extremely streamlined, only needing to host the service container itself.

[0061] The accelerated processing module 33 treats DPU node 3 as an independent network node to uniformly handle the virtualization, policy enforcement, and traffic forwarding of all containers. By listening to the api-server, it configures complete virtual network configurations for containers located within the DPU, and configures OVS networks for containers located on its host. This completely frees up host computing resources, enhances network security isolation, and greatly simplifies the operational complexity on the host side.

[0062] Based on the above settings, the OVN controller 31 on each DPU node 3 listens to the changes related to itself in the OVN southbound interface 24 and obtains these logical flow tables. It then compiles them into specific OpenFlow flow table rules and sends them to the local virtual switch 32. The virtual switch 32 then performs the forwarding of the data packets, thereby completing the routing management of the newly added tenant containers.

[0063] The above are merely preferred embodiments of the present invention and are not intended to limit the implementation methods and protection scope of the present invention. Those skilled in the art should recognize that any equivalent substitutions and obvious changes made based on the description and illustrations of the present invention should be included within the protection scope of the present invention.

Claims

1. A RoCE multi-tenant management system based on SDN, characterized in that, This includes worker nodes, control nodes, and DPU nodes built on a Kubernetes cluster; The DPU node runs multiple tenant containers, which are used to perform different data processing tasks. The control node interacts with the worker node to determine the network configuration for the newly added tenant container; The control node translates the logical data path flow table according to the logical network object pointed to by the network configuration; The DPU node listens to the control node and generates flow table rules according to the logical data path flow table when the logical data path flow table appears. The DPU node assigns network addresses to the newly added tenant container according to the flow table rules and controls packet forwarding to complete the addition of the tenant container.

2. The RoCE multi-tenant management system according to claim 1, characterized in that, The working nodes include: The proxy component module is connected to the Kubernetes control plane and network infrastructure of the Kubernetes cluster, respectively. The proxy component module collects data from the network infrastructure to confirm the network model within the network infrastructure. The working node negotiates with the control node according to the network model to determine the network configuration corresponding to the newly added tenant container.

3. The RoCE multi-tenant management system according to claim 2, characterized in that, The Kubernetes control plane obtains new container information based on the kube-apiserver to detect the tenant container.

4. The RoCE multi-tenant management system according to claim 2, characterized in that, The Kubernetes control plane invokes the proxy component module to create a sandbox for the tenant container on the DPU node.

5. The RoCE multi-tenant management system according to claim 4, characterized in that, The worker node writes the network configuration to the control node based on the invoked proxy component module.

6. The RoCE multi-tenant management system according to claim 1, characterized in that, The control node includes: an SDN controller, an OVN northbound interface, an OVN conversion module, and an OVN southbound interface; The SDN controller writes the network configuration to the OVN northbound interface; The OVN conversion module monitors the database of the OVN northbound interface to determine whether the network configuration is written to the OVN northbound interface. When the network configuration is written to the OVN northbound interface, the OVN conversion module calls the OVN conversion module to generate the logical path data flow table and stores it in the OVN southbound interface to send it to the DPU node.

7. The RoCE multi-tenant management system according to claim 6, characterized in that, The OVN conversion module extracts logical network objects from the network configuration and generates the logical path data flow table based on the data flow direction between the logical network objects.

8. The RoCE multi-tenant management system according to claim 1, characterized in that, The DPU node includes: OVN controller, the OVN controller obtains the newly generated logical path data flow table in the control node; A virtual switch, which forwards data to the tenant container according to the logical path data flow table.

Images