An edge computing platform architecture
Through embedded K3S cluster and configuration resource structure, the coupling problem caused by the complexity of edge computing applications is solved, isolation between applications and reasonable allocation of resources is achieved, and system stability and troubleshooting efficiency are improved.
Patent Information
- Application Number
- CN202410422523.9
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-04-09
- Publication Date
- 2024-11-05
- Estimated Expiration
- 2044-04-09
AI Technical Summary
The increasing application coupling caused by the continuous improvement of the complexity and scale of edge computing applications, leading to problems such as competing for resource occupation and dependence, affecting the stability of the system and the difficulty of troubleshooting.
Using the embedded K3S cluster method, users can provide complete K8S cluster functions, so that associated applications or components can exclusively occupy cluster resources, and by configuring the IP address and resource structure of internal cluster nodes, isolation between applications is achieved and failures are reduced. Troubleshooting difficulty.
It realizes a strong isolation edge computing platform architecture, avoids mutual influence between applications, improves system stability and resource allocation efficiency, and simplifies the troubleshooting process.
Smart Images

Figure CN118170497B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of edge computing system architecture, and in particular to an edge computing platform architecture. Background Art
[0002] Edge computing is a product of network evolution and cloud computing technology development. It provides computing, storage, network and other infrastructure close to users, and provides edge cloud services to users by deploying and running applications on this infrastructure. The development of edge computing has received widespread attention from all parties. Representative companies of operation and management service providers include Alibaba Cloud, Tencent Cloud, Tianyi Cloud, Baidu Smart Cloud, etc. The growth rate of the edge computing market size continues to grow. With the surge in applications and data volume, network bandwidth and computing throughput have become performance bottlenecks for computing. At the same time, the real-time processing demand for massive "small data" generated by terminal devices has increased rapidly, driving edge computing to become an important computing platform for the implementation of technology in the data age, and a key support for agile connection, real-time business, privacy protection, etc. in the digital transformation of the industry.
[0003] The construction of edge computing platforms involves multiple technical fields, including computing, storage, networking, security, etc. Kubernetes (K8S for short) is an open source container orchestration system that can automatically deploy, scale, and manage containerized applications. In the edge computing architecture, K8S can be used to manage containerized applications on edge nodes to better manage and schedule edge computing resources. The application of K8S in the edge computing architecture mainly includes edge node management, edge node automatic scaling, edge node high availability, edge node security, and edge computing application management and control. The application of K8S in the edge computing architecture can improve the efficiency, stability, and reliability of the edge computing platform.
[0004] K3S is a highly available, CNCF-certified Kubernetes distribution designed for production workloads in unattended, resource-constrained, remote areas, or within IoT devices. It is easy to install, small, has few dependencies, is highly scalable, and supports heterogeneous resources. Based on the above features, K3S is very suitable for building edge computing infrastructure. Combined with independently deployed 5G private networks, a complete edge computing ecosystem from network to platform can be built for enterprises.
[0005] With the continuous development of edge computing with K8S technology as the platform base, more and more edge computing applications are constantly improving, and the scale and complexity of applications are constantly increasing. Therefore, the isolation between applications becomes more and more important. Isolation refers to the isolation of different computing resources, data, applications, etc. in an edge computing environment to ensure the security and stability of the edge computing system. The important properties of edge computing system isolation include: security, stability, reliability, flexibility, and scalability. By isolating different applications, interference between different applications can be effectively avoided, thereby ensuring the stability of the system. At the same time, in a system with high isolation, it helps application developers to troubleshoot problems and speed up application development. Therefore, the present invention proposes an edge computing platform architecture. Summary of the invention
[0006] The embodiment of the present invention provides a configuration method to solve the above-mentioned technical problems in the prior art.
[0007] In order to have a basic understanding of some aspects of the disclosed embodiments, a brief summary is given below. This summary is not a general review, nor is it intended to identify key / important components or to describe the scope of protection of these embodiments. Its only purpose is to present some concepts in a simple form as a prelude to the detailed description that follows.
[0008] According to the first aspect of an embodiment of the present invention, an edge computing platform architecture is provided, which mainly solves the problem of increased application coupling caused by the increasing complexity and scale of edge computing applications. In the same edge computing platform, various applications may compete for resource occupation (such as CPU, memory, storage, port number, domain name, IP address, etc.) and resource dependencies (such as files, sockets, etc.). By adopting an embedded K3S cluster method, a complete K8S cluster function is provided to users, so that a group of related applications or components can monopolize a cluster resource, thereby improving the isolation between applications, thereby ensuring the stability of the application and reducing the difficulty of troubleshooting application failures.
[0009] In one embodiment, the edge computing platform architecture includes a management node located in a data center and a plurality of edge computing nodes connected to the management node, the edge computing node has a plurality of external cluster nodes for managing containerized applications, each external cluster node has a plurality of internal cluster nodes; the external cluster nodes are connected to the internal cluster nodes via a cluster gateway configured with an IP address of the internal cluster node; the internal cluster nodes are composed of a data storage unit, a service node, and an agent node;
[0010] The data storage unit is used to provide data persistence to ensure that the data storage volume is remounted after the container is stopped or restarted to ensure that data is not lost;
[0011] The service node is used to store the internal cluster ETCD data and the persistent data of the application scheduled on the internal cluster node; the proxy node is used to store the persistent data of the application scheduled on the internal cluster node.
[0012] In one embodiment, the external cluster node is a K8S cluster node, the internal cluster node is a K3S cluster node, and the K8S cluster node and the K3S cluster node are connected via a cluster gateway configured with an IP address of the K3S cluster node.
[0013] In one embodiment, when selecting an external cluster node, the process includes:
[0014] According to the resource level of the external cluster node and the instance requirements of the newly created internal cluster resources, the external cluster node where each internal cluster node container will be created is selected. If the resource requirements of the internal cluster instance cannot be met, the resource creation status is in the waiting state.
[0015] In one embodiment, each internal cluster node is configured with a corresponding data resource structure, which includes the cluster name, cluster domain name information, the number of service nodes, the number of proxy nodes, the node IP pool, the container IP address segment, the service IP address segment and resource limitation information. The data resource structure of the internal cluster node is defined in the form of a K8S cluster node custom resource CRD, and a CRD controller is deployed to listen to the creation and modification task events of the internal cluster CR.
[0016] In one embodiment, before creating the internal cluster node, the process includes:
[0017] Verify whether the cluster name already exists, whether the IP pool is occupied or overlaps with the existing internal cluster, whether the cluster domain name complies with the specifications, and whether the required computing resources and storage resources are available in the current main cluster.
[0018] In one embodiment, the creation of the data store includes:
[0019] When the external cluster nodes meet the requirements, the information of the internal cluster to be created is pushed to the queue to be created, and the internal cluster name and storage resource request are created through the controller. After the storage resource request is created, the corresponding storage resources are automatically created.
[0020] In one embodiment, creation of a service node container includes:
[0021] After all service nodes are started normally, service resources are created and an access domain name is set for the service resources for the proxy node to register a fixed address with the service node; the number of internal cluster service nodes and the number of external cluster schedulable nodes are set, and the service node containers are distributed to different external cluster nodes;
[0022] The creation of the agent node container includes:
[0023] After the service node is started normally and there are proxy nodes in the internal cluster resource definition, the controller starts the creation process of all proxy nodes in parallel.
[0024] In one embodiment, in each external cluster node, multiple internal cluster nodes are prioritized by preconfigured cluster weights, and an internal cluster with a higher priority is created first.
[0025] In one embodiment, the container application data in the internal cluster node is mounted through the hostPath method or the local-path Provider method.
[0026] The technical solution provided by the embodiment of the present invention may have the following beneficial effects:
[0027] 1) The present invention implements a highly isolated nested edge computing platform architecture, which allows some edge applications to monopolize logical cluster resources and avoid mutual influence between applications.
[0028] 2) By adding CRD "internal cluster" and its controller to the existing K8S cluster, you can easily create a k3s cluster running on the K8S cluster. The internal cluster has the characteristics of strong isolation, fast startup, second-level expansion and contraction, persistent data storage, and internal cluster network isolation.
[0029] 3) Internal clusters can set different weights for different internal clusters according to the usage scenarios and the nature of the application. At the same time, weights can be set separately according to different resource weights to maximize the needs of internal clusters, making the resource allocation and scheduling of the edge computing platform more reasonable and efficient.
[0030] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the invention. BRIEF DESCRIPTION OF THE DRAWINGS
[0031] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the invention and, together with the description, serve to explain the principles of the invention.
[0032] Figure 1It is a structural diagram of an edge computing platform architecture according to an exemplary embodiment. DETAILED DESCRIPTION
[0033] The following description and accompanying drawings fully illustrate the specific embodiments of this article so that those skilled in the art can practice them. Parts and features of some embodiments may be included in or replace parts and features of other embodiments. The scope of the embodiments of this article includes the entire scope of the claims, as well as all available equivalents of the claims. Herein, the terms "first", "second", etc. are only used to distinguish one element from another, without requiring or implying any actual relationship or order between these elements. In fact, the first element can also be called the second element, and vice versa. Moreover, the terms "include", "comprise" or any other variant thereof are intended to cover non-exclusive inclusion, so that the structure, device or equipment including a series of elements includes not only those elements, but also other elements that are not explicitly listed, or also include elements inherent to such structure, device or equipment. In the absence of more restrictions, the elements defined by the sentence "including one..." do not exclude the existence of other identical elements in the structure, device or equipment including the elements. Each embodiment is described in a progressive manner herein, and each embodiment focuses on the differences from other embodiments, and the same and similar parts between the embodiments can be referred to each other.
[0034] The terms "longitudinal", "lateral", "upper", "lower", "front", "back", "left", "right", "vertical", "horizontal", "top", "bottom", "inside", "outside", etc. in this document indicate the orientation or position relationship based on the orientation or position relationship shown in the drawings, and are only for the convenience of describing this document and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, be constructed and operate in a specific orientation, and therefore cannot be understood as a limitation on the present invention. In the description of this document, unless otherwise specified and limited, the terms "installed", "connected", and "connected" should be understood in a broad sense, for example, it can be a mechanical connection or an electrical connection, it can also be the internal communication of two elements, it can be a direct connection, or it can be an indirect connection through an intermediate medium. For ordinary technicians in this field, the specific meanings of the above terms can be understood according to specific circumstances.
[0035] As used herein, the term "plurality" means two or more than two, unless otherwise specified.
[0036] In this document, the character " / " indicates that the preceding and following objects are in an "or" relationship. For example, A / B means: A or B.
[0037] In this article, the term "and / or" is a description of the association relationship between objects, indicating that three relationships may exist. For example, A and / or B means: A or B, or, A and B.
[0038] It should be understood that, although the various steps in the flow chart are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear description in this article, the execution of these steps is not strictly limited in order, and these steps can be executed in other orders. Moreover, at least a portion of the steps in the figure may include multiple sub-steps or multiple stages, and these sub-steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these sub-steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a portion of the sub-steps or stages of other steps.
[0039] Each module in the device or system of the present application can be implemented in whole or in part by software, hardware, or a combination thereof. The above modules can be embedded in or independent of a processor in a computer device in the form of hardware, or can be stored in a memory in a computer device in the form of software, so that the processor can call and execute operations corresponding to the above modules.
[0040] In the absence of conflict, the embodiments of the present invention and the features of the embodiments may be combined with each other.
[0041] Figure 1 An embodiment of an edge computing platform architecture of the present invention is shown.
[0042] The abbreviations and key terms in this embodiment are defined as follows:
[0043] K8S Kubernetes (a container management engine);
[0044] K3S is a simplified version of k8s, retaining only the core functions;
[0045] Pod is the smallest unit of scheduling in k8s, which contains one or more containers;
[0046] Deployment k8s is a type of resource used to maintain a group of Pods;
[0047] Persistent storage volumes in PV k8s;
[0048] Application statement for persistent storage volume in PVC k8s;
[0049] CRD "User Defined Resource" definition;
[0050] CR An instance of a "User Defined Resource";
[0051] Minio is a distributed object store.
[0052] In this optional embodiment, the edge computing platform architecture includes a management node located in a data center and a plurality of edge computing nodes connected to the management node, the edge computing node has a plurality of external cluster nodes for managing containerized applications, each external cluster node has a plurality of internal cluster nodes; the external cluster nodes are connected to the internal cluster nodes via a cluster gateway configured with an IP address of the internal cluster node; the internal cluster nodes are composed of a data storage unit, a service node, and a proxy node;
[0053] The data storage unit is used to provide data persistence, that is, to ensure that the data storage volume can be remounted after the container is stopped or restarted to ensure that data is not lost. From the perspective of the internal cluster, it means that the internal cluster data is not lost after the node is restarted. From the perspective of the external cluster, it means that the internal cluster data is not lost;
[0054] The service node is used to store the internal cluster ETCD data and the persistent data of the application scheduled on the internal cluster node (the internal cluster node refers to a service node or proxy node of the internal cluster); the proxy node is used to store the persistent data of the application scheduled on the internal cluster node.
[0055] Both service nodes and proxy nodes support running Pods of a certain application. For example, suppose a personnel management application contains two Pods (Pod is the smallest unit for running services on K8S / K3S, containing one or more containers), Pod1 runs on service node 1, and Pod2 runs on proxy node 2. Then the application data of Pod1 is persistently stored in service node 1, and the application data of Pod2 is persistently stored in proxy node 2.
[0056] ETCD is a key-value database based on the distributed consensus algorithm Raft. Internal cluster ETCD data is the internal K3S cluster ETCD data. Like K8S, it mainly stores cluster status information, resource (Pod, deployment, daemonset, etc.) information, configuration information, etc. It is the core database of the K8S / K3S cluster.
[0057] The present invention provides an edge computing platform architecture, the main function of which is to run a complete K3S cluster in an edge computing platform with K8S as the underlying framework, and provide a complete K8S experience to business applications or developers. Users run the application into the built-in K3S cluster without perception, and completely isolate the application's operating environment according to certain rules. Provide complete resources (CPU, memory, storage) to the sub-cluster, and use restriction schemes to accurately limit the occupancy of computing resources. Provide persistent data storage for sub-clusters to prevent data loss due to sub-cluster node drift. In terms of network, the main K8S cluster can use overlay or underlay network plug-ins. Each K3S sub-cluster has a separate IP pool, and the node of each sub-cluster is a Pod of the main K8S cluster. The networks between the sub-clusters are not interconnected. If the sub-cluster network needs to be connected, the relevant routing information can be configured in the cluster gateway, including routing entries, effective time, expiration time, priority, etc.
[0058] Through the edge computing platform architecture of the present invention, it is possible to truly achieve isolation between edge computing applications, avoiding the problem of complex permission management and resources required by using native K8S. Users can divide cluster allocation rules according to different dimensions as needed, and allocate and limit corresponding computing resources, network resources, and storage resources. Improve the isolation and scalability of applications.
[0059] In specific applications, the specific steps performed by the edge computing platform architecture of the present invention are as follows:
[0060] 1. Define the resource structure of the internal cluster. Define the internal cluster in the form of "user-defined resources". The main information includes cluster name (globally unique) clusterName, cluster domain information domain, server node number serverNum, agent node number agentNum, node IP pool ippool, Pod IP address segment podCidr, serviceCidr address segment and resource limit information: resource.server.mem.Limit (maximum service node memory), resource.server.cpu.Limit (maximum service node cpu), resource.server.disk.Limit (maximum service node storage space), resource.agent.mem.Limit (maximum agent node memory), resource. agent.cpu.Limit (maximum agent node cpu), resource.agent.disk.Limit (maximum agent node storage space). Define the above internal cluster data structure in the form of K8S custom resource CRD, deploy CRD controller, and listen to the creation and modification task events of the internal cluster CR.
[0061] 2. Implement the internal cluster "custom resource controller". The main core function is to verify and create corresponding resources when creating an "internal cluster custom resource" instance. The main steps are described in 3-10.
[0062] 3. Resource definition verification, including whether the cluster name already exists, whether the IP pool is occupied or overlaps with the existing internal cluster, whether the cluster domain name complies with the specifications, and whether the required computing resources and storage resources are available in the current main cluster.
[0063] 4. Node selection: According to the resource (CPU, memory, storage, bandwidth) water level of each node in the main cluster and the requirements of the newly created internal cluster resource instance, select the main cluster node where the Pod of each internal cluster node will be created. If the resource requirements of the internal cluster instance cannot be met, the resource creation status will be in Pending status.
[0064] "Node selection" means searching the external K8S cluster for external K8S cluster nodes that meet the node requirements of the internal cluster to run the service nodes or agent nodes of the internal cluster.
[0065] 5. Node storage creation. If the main cluster can meet the requirements, the information of the internal cluster to be created will be pushed into the queue to be created. At the same time, the controller will first create the namespace (name of the internal cluster) and PVC. After the PVC is created, the corresponding PV will be automatically created. For the server node of the internal cluster, it mainly stores two parts of data that need to be persisted: the internal cluster ETCD data and the persistent data of the application scheduled on the node of the internal cluster; for the agent node, the persistent data is the persistent data of the application scheduled on the node.
[0066] 6. The internal cluster server node Pod is created. In particular, to prevent IP drift after the container restarts, the IP addresses of the server node and the agent node are fixed. After all servers are started normally, a service resource named server will be created. The full access domain name of this service is server. <clustername>.svc.cluster.local. This address is used as the fixed address for the agent to register with the server. Let the number of server nodes in the internal cluster be IC, and the number of schedulable nodes in the external main cluster be OC. The Pods of the server nodes will be distributed to different nodes in the external cluster as much as possible. This prevents the situation where the server nodes in the internal cluster are unavailable due to a node failure in the external main cluster.
[0067] 7. Internal cluster agent node Pod creation. After the server node is started, if there is an agent node in the internal cluster resource definition, the controller will start the creation process of all agent nodes in parallel. The agent's registration address to the server is the service address described in step 6.
[0068] 8. Add corresponding access rules in the cluster routing. After all nodes in the internal cluster are successfully started, the "cluster gateway" running in the external main cluster in deployment mode will add corresponding access rules, so that users outside the cluster can access the internal cluster and can only access the specified internal cluster. The main process is to add an IP in the "cluster gateway" that belongs to the internal cluster network segment, and then add a route to forward the traffic accessing the internal cluster node IP to the Pod of the server node of the internal cluster.
[0069] 9. When creating each internal cluster, you can select a weight. The higher the weight, the higher the priority will be when the cluster is created. In addition to setting the overall weight of the internal cluster, you can also set the weight of the required resources separately. For example, if application A has a high demand for CPU and a low demand for storage, you can set the CPU weight to 90 and the storage weight to 20. In this way, when creating internal cluster nodes, nodes with high CPU idleness will be scheduled first, even if the node has a low storage score. The same applies to resources such as bandwidth.
[0070] 10. The container application data in the internal cluster is mounted to the internal cluster through the hostPath method or the PV is mounted using the local-path Provider method. The data volume of the internal cluster is mounted to the minio distributed storage of the external cluster. Therefore, the internal cluster can also be "stopped". After stopping, the currentNum of each node in the internal cluster definition will be set to 0.
[0071] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided by the present invention can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory or optical memory, etc. Volatile memory can include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM).
[0072] The present invention is not limited to the structures which have been described above and shown in the drawings, and various modifications and changes may be made without departing from the scope thereof. The scope of the present invention is limited only by the appended claims.< / clustername>
Claims
1. An edge computing platform architecture, comprising a management node located in a data center and a plurality of edge computing nodes connected to the management node, characterized in that: The edge computing node has multiple external cluster nodes for managing containerized applications, each external cluster node has multiple internal cluster nodes; the external cluster nodes are connected to the internal cluster nodes via a cluster gateway configured with an IP address of the internal cluster node; the internal cluster nodes are composed of a data storage unit, a service node and a proxy node; The data storage unit is used to provide data persistence to ensure that the data storage volume is remounted after the container is stopped or restarted to ensure that data is not lost; The service node is used to store the internal cluster ETCD data and the persistent data of the application scheduled on the internal cluster node; the proxy node is used to store the persistent data of the application scheduled on the internal cluster node.
2. The edge computing platform architecture according to claim 1, characterized in that: The external cluster node is a K8S cluster node, the internal cluster node is a K3S cluster node, and the K8S cluster node and the K3S cluster node are connected via a cluster gateway configured with an IP address of the K3S cluster node.
3. The edge computing platform architecture according to claim 1, characterized in that: When selecting external cluster nodes, include: According to the resource level of the external cluster node and the instance requirements of the newly created internal cluster resources, the external cluster node where each internal cluster node container will be created is selected. If the resource requirements of the internal cluster instance cannot be met, the resource creation status is in the waiting state.
4. The edge computing platform architecture according to claim 1, characterized in that: Each internal cluster node is configured with a corresponding data resource structure, which includes the cluster name, cluster domain name information, number of service nodes, number of proxy nodes, node IP pool, container IP address segment, service IP address segment and resource restriction information. The data resource structure of the internal cluster node is defined in the form of K8S cluster node custom resource CRD, and a CRD controller is deployed to listen to the creation and modification task events of the internal cluster CR.
5. The edge computing platform architecture according to claim 1, characterized in that: Before creating the internal cluster nodes include: Verify whether the cluster name already exists, whether the IP pool is occupied or overlaps with the existing internal cluster, whether the cluster domain name complies with the specifications, and whether the required computing resources and storage resources are available in the current main cluster.
6. The edge computing platform architecture according to claim 1, characterized in that: The creation of the data storage includes: When the external cluster nodes meet the requirements, the information of the internal cluster to be created is pushed to the queue to be created, and the internal cluster name and storage resource request are created through the controller. After the storage resource request is created, the corresponding storage resources are automatically created.
7. The edge computing platform architecture according to claim 1, characterized in that: The creation of a service node container includes: After all service nodes are started normally, create service resources and set an access domain name for the service resources for the proxy node to register a fixed address with the service node; Set the number of internal cluster service nodes and the number of external cluster schedulable nodes, and distribute the service node containers to different external cluster nodes.
8. The edge computing platform architecture according to claim 1, characterized in that: The creation of the agent node container includes: After the service node is started normally and there are proxy nodes in the internal cluster resource definition, the controller starts the creation process of all proxy nodes in parallel.
9. The edge computing platform architecture according to claim 1, characterized in that: In each external cluster node, multiple internal cluster nodes are prioritized by pre-configured cluster weights. The internal cluster with the higher priority is created first.
10. The edge computing platform architecture according to claim 1, characterized in that: The container application data in the internal cluster node is mounted through the hostPath method or the local-path Provider method.
