Resource multi-level scheduling method in edge scenarios

Abstract

Disclosed in the present application is a resource multi-level scheduling method in edge scenarios. The method comprises: a CDN edge server being connected to a k8s cluster by means of a cloud-edge channel kubeedge, and being exposed to a service for use by means of a virtual cluster; on the basis of the resource requirements of the service, applying for resources to a CDN resource scheduling platform, and when a CDN service is prioritized, returning a resource list meeting a condition; a service side mapping the responded resource list to the virtual cluster, evenly distributing resources to all clusters, generating CRD deployment orchestration, and deploying same to a CDN edge cluster by means of an ESK platform; a CRD controller monitoring EdgeInfo resources and creating a pod on the basis of resource content; and a service container reporting the actual usage amount of the resources to a monitor by means of indicators, and the resource scheduling platform performing global blending on the basis of monitoring data and resource thresholds.

Description

A multi-level resource scheduling method in edge scenarios

[0001] Related applications

[0002] This application claims priority to Chinese patent application number 2023117223701, filed on December 14, 2023, entitled “A multi-level resource scheduling method in edge scenarios,” the entire text of which is hereby incorporated by reference. Technical Field

[0003] The present application relates to the field of distributed edge computing technology, and more specifically to a multi-level resource scheduling method in an edge scenario. Background Art

[0004] As the content delivery network (CDN) service market grows, CDN service providers have tens of thousands of edge node machines, and the number of devices continues to grow. CDN services experience significant peaks and troughs in resource usage, primarily utilizing uplink bandwidth, with relatively low utilization of other resources. With the rapid development of edge computing and the increasing diversity of edge services, the rational reuse of CDN edge nodes can improve device resource utilization, thereby reducing service costs and enhancing product competitiveness.

[0005] Because CDN edge nodes are distributed across different regions and in different operator network environments, CDN services are not evenly distributed across edge nodes, resulting in uneven resource availability on edge node machines. While ensuring CDN services, CDN edge nodes can be reused through distributed resource scheduling. Summary of the Invention

[0006] In view of the shortcomings of the existing technology, the purpose of this application is to provide a multi-level resource scheduling method in an edge scenario to solve the problem of idle resource reuse of large-scale CDN edge nodes.

[0007] To achieve the above objectives, this application provides the following technical solutions:

[0008] A multi-level resource scheduling method in an edge scenario includes the following steps:

[0009] The CDN edge server is connected to the Kubernetes cluster through the cloud-edge channel kubeedge and is exposed to business use through the virtual cluster virtual cluster;

[0010] Apply for resources from the CDN resource scheduling platform based on the business's resource needs. With CDN business priority, a list of qualified resources is returned.

[0011] The business side maps the response resource list to the virtual cluster and evenly distributes it to all clusters. It then generates a CRD deployment orchestration and deploys it to the CDN edge cluster through the ESK platform.

[0012] The CRD controller listens to the EdgeInfo resource and creates a pod based on the resource content;

[0013] Business containers report actual resource usage to monitoring through indicators, and the resource scheduling platform performs global coordination based on monitoring data and resource thresholds.

[0014] As a further improvement of the present application, the pod is synchronized to the CDN edge working cluster through syncer.

[0015] As a further improvement to this application, in order to protect the controllable network resources of the edge work nodes, CNI is enabled and the meta plug-in under the containernetworking project is used to limit the flow.

[0016] As a further improvement of the present application, the following one or more dimensions are specified when applying for resources:

[0017] Dimension 1: Total upstream and downstream bandwidth, in Mbps;

[0018] Dimension 2: Application health value requirements, expulsion health value requirements;

[0019] Dimension 3: Single-machine CPU core count requirement, load application requirement, load eviction requirement, and load as a percentage;

[0020] Dimension 4: memory size requirement (unit: M), memory load request requirement, memory load eviction requirement, load as a percentage;

[0021] Dimension 5: Disk size requirement (unit: M), disk load application requirement, disk load eviction requirement (load as a percentage);

[0022] Dimension 6: Link load application requirements, link load eviction requirements, load is the number.

[0023] As a further improvement of this application, the node dimension defines a resource usage template. After the business is applied to the resource pool, the node dimension resource usage is generated within the resource pool. The resource usage of each node is declared in EdgeInfo.

[0024] The EdgeTask is used to define label selection, image version, number of replicas, and EdgeInfo instance information.

[0025] As a further improvement of the present application, the controller schedules the pod to an idle Node node through node affinity, monitors the status of the node in the cluster, and deletes the corresponding Pod when the node status is abnormal and an idle pod is deployed.

[0026] As a further improvement of this application, the K8s operator cooperates with the cloud-edge channel to implement edge node load scheduling, including the following steps:

[0027] The Operator registers custom resources of type EdgeTask and EdgeInfo with the Kubernetes master's API server to specify node usage rules and workloads.

[0028] The Operator starts a custom controller and maintains the number of EdgeTask replicas through reconcile.

[0029] The K8s node is registered with the k8smaster through the cloud-edge channel composed of cloud-manager and edge-agent;

[0030] When users operate businesses through the platform, the system writes information such as the specified machine range into the custom resource service orchestration of the EdgeInfo type, and writes information such as the image version and number of replicas into the custom resource service orchestration of the EdgeTask type.

[0031] When orchestrating deployment to a virtual cluster, the custom controller's reconcile function creates a pod for each node in the EdgeInfo list.

[0032] The custom controller uses reconcile to perform tuning based on the information in the EdgeInfo orchestration, creates a task service pod, and submits it to the API server.

[0033] Use syncer to synchronize the pod to the CDN edge worker cluster.

[0034] As a further improvement of this application, the K8s operator cooperates with the cloud-edge channel to implement edge node load scheduling, further comprising the following steps:

[0035] The cloud-manager in the cloud will monitor the resource changes of the api-server in real time and identify pod creation / updates;

[0036] Cloud-manger distributes pod orchestration modification messages to edge-agents at the edge through the cloud-edge channel;

[0037] The edge-agent at the edge receives the orchestration modification message and calls CRI to manage the pod.

[0038] A computer-readable storage medium stores a computer program, which, when executed by a processor, implements the multi-level resource scheduling method in the edge scenario as described above.

[0039] A device comprising:

[0040] a memory for storing instructions;

[0041] The processor is used to execute the instructions so that the device performs operations to implement the multi-level resource scheduling method in the edge scenario as described above.

[0042] Beneficial effects of this application:

[0043] 1. Because native Kubernetes doesn't provide scheduling for bandwidth resource reuse, existing CDN bandwidth scheduling is business-based. This approach only addresses CDN business scheduling and isn't applicable to other edge computing scenarios. It also fails to address resource reuse issues for large-scale edge nodes. By adopting cloud-native concepts and leveraging CDN resource scheduling, resource dimension grouping, multi-cluster scheduling, and the Kubernetes Operator, we've implemented a distributed resource scheduling method for large-scale edge nodes with automatic coordination capabilities.

[0044] 2. From the perspective of resource scheduling, the scheduling process is split into two steps: central resource pool scheduling and edge Kubernetes node scheduling. Central resource pool scheduling is based on resource dimensions, while edge scheduling is within the scope of resource pool scheduling results. This is defined and implemented through declarative APIs and Kubernetes operators.

[0045] 3. Precise control of large-scale edge node resources is achieved through multi-level scheduling at the center and edge. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the conventional technology, the following briefly introduces the drawings required for use in the embodiments or the conventional technology descriptions. Obviously, the drawings described below are merely embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on the disclosed drawings without any creative work.

[0047] Figure 1 is a diagram of the SaaS platform architecture;

[0048] Figure 2 is a flowchart of the SaaS service delivery orchestration process. DETAILED DESCRIPTION

[0049] The following will be combined with the drawings in the embodiments of this application to clearly and completely describe the technical solutions in the embodiments of this application. Obviously, the embodiments described are only part of the embodiments of this application, not all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0050] Unless otherwise defined, the technical or scientific terms used in this application should have the usual meanings understood by people with ordinary skills in the field to which this application belongs. The "first", "second" and similar words used in this application do not indicate any order, quantity or importance, but are only used to distinguish different components. "Include" or "comprise" and similar words mean that the elements or objects appearing before the word include the elements or objects listed after the word and their equivalents, without excluding other elements or objects. "Connect" or "connected" and similar words are not limited to physical or mechanical connections, but may include electrical connections, whether direct or indirect. "Up", "down", "left", "right" and the like are only used to indicate relative position relationships. When the absolute position of the object being described changes, the relative position relationship may also change accordingly.

[0051] In order to keep the following description of the embodiments of the present application clear and concise, detailed descriptions of known functions and known components are omitted in this application.

[0052] Example 1:

[0053] 1 and 2 , a specific implementation of a multi-level resource scheduling method in an edge scenario of the present application is shown, including the following steps:

[0054] The CDN edge server is connected to the Kubernetes cluster through the cloud-edge channel (kubeedge), avoiding the potential risks of direct external exposure of the CDN Kubernetes cluster. It is exposed to business use through the virtual cluster.

[0055] Based on the resource requirements of the business, the platform applies for resources from the CDN resource scheduling platform. Prioritizing CDN business, the platform returns a list of eligible resources. Resource requirements include, but are not limited to, bandwidth granularity, device health, and distribution area. During this process, the business side needs to report actual usage through monitoring indicators and be able to quickly release resources.

[0056] The business side maps the response resource list to the virtual cluster and evenly distributes it to all clusters. It then generates a CRD deployment orchestration and deploys it to the CDN edge cluster through the ESK platform.

[0057] The CRD controller monitors EdgeInfo resources and creates pods based on the resource content. The pods are then synchronized to the CDN edge cluster through the syncer.

[0058] Business containers report actual resource usage to monitoring through indicators. The resource scheduling platform performs global coordination based on monitoring data and resource thresholds. To ensure that network resources on edge work nodes are controllable, you need to enable CNI and implement rate limiting through the meta plug-in under the containernetworking project.

[0059] When applying for resources, specify one or more of the following dimensions:

[0060] Dimension 1: Total upstream and downstream bandwidth, in Mbps;

[0061] Dimension 2: Application health value requirements, expulsion health value requirements;

[0062] Dimension 3: Single-machine CPU core count requirement, load application requirement, load eviction requirement, and load as a percentage;

[0063] Dimension 4: memory size requirement (unit: M), memory load request requirement, memory load eviction requirement, load as a percentage;

[0064] Dimension 5: Disk size requirement (unit: M), disk load application requirement, disk load eviction requirement (load as a percentage);

[0065] Dimension 6: Link load application requirements, link load eviction requirements, load is the number.

[0066] The node dimension defines a resource usage template. After the service is applied to the resource pool, the node dimension resource usage is generated within the resource pool. The resource usage of each node is declared in EdgeInfo. The arrangement information is as follows:

[0067] 1.apiVersion:esx.ctcdn.cn / v1

[0068] 2.kind:EdgeInfo

[0069] 3.metadata:

[0070] 4.name:myinfo

[0071] 5.spec:

[0072] 6.global:

[0073] 7.sceneName:test

[0074] 8.sceneId:1

[0075] 9.startTime:20220912

[0076] 10.token:abcd

[0077] 11.infos:

[0078] 12.-node:node1

[0079] 13.up:100

[0080] 14.down:20

[0081] 15.-node:node2

[0082] 16.up:300

[0083] 17.down:60.

[0084] The EdgeTask is used to define label selection, image version, number of replicas, and EdgeInfo instance information. Its orchestration information is as follows:

[0085] 1.apiVersion:esx.ctcdn.cn / v1

[0086] 2.kind:EdgeTask

[0087] 3.metadata

[0088] 4.name:sample

[0089] 5.spec:

[0090] 6.confName:test

[0091] 7.replicas:2

[0092] 8.template:

[0093] 9.metadata:

[0094] 10.name:sample

[0095] 11.spec:

[0096] 12.-name:mySample

[0097] 13.imagePullPolicy:IfNotPresent

[0098] 14.image:ct-image:v1.0.1.

[0099] The controller schedules pods to idle nodes based on node affinity and monitors the status of nodes in the cluster. When a node is in an abnormal state and an idle pod is deployed, the corresponding pod will be deleted.

[0100] The K8s operator cooperates with the cloud-edge channel to implement edge node load scheduling, which includes the following steps:

[0101] The Operator registers custom resources of type EdgeTask and EdgeInfo with the Kubernetes master's API server to specify node usage rules and workloads.

[0102] The Operator starts a custom controller and maintains the number of EdgeTask replicas through reconcile.

[0103] The K8s node is registered with the k8smaster through the cloud-edge channel composed of cloud-manager and edge-agent;

[0104] When users operate businesses through the platform, the system writes information such as the specified machine range into the custom resource service orchestration of the EdgeInfo type, and writes information such as the image version and number of replicas into the custom resource service orchestration of the EdgeTask type.

[0105] When orchestrating deployment to a virtual cluster, the custom controller's reconcile function creates a pod for each node in the EdgeInfo list.

[0106] The custom controller uses reconcile to perform tuning based on the information in the EdgeInfo orchestration, creates a task service pod, and submits it to the API server.

[0107] Use syncer to synchronize pods to the CDN edge working cluster;

[0108] The cloud-manager in the cloud will monitor the resource changes of the api-server in real time and identify pod creation / updates;

[0109] Cloud-manger distributes pod orchestration modification messages to edge-agents at the edge through the cloud-edge channel;

[0110] The edge-agent at the edge receives the orchestration modification message and calls CRI to manage the pod.

[0111] Performance Testing Service (PTS) stresses the SaaS platform, simulating real-world business scenarios with massive user volumes to comprehensively verify the performance, capacity, and stability of business sites. This requires a large number of edge nodes to initiate stress testing. Analysis shows that PTS stress testing generally consumes a large amount of total bandwidth, primarily downstream bandwidth and network connections, while CDN acceleration primarily consumes upstream bandwidth.

[0112] When running a business, the PTS applies to the resource scheduling platform for resource requirements in two dimensions: downlink bandwidth and region, generates business CRD resources, and deploys them to edge nodes.

[0113] Since native k8s itself does not provide scheduling for bandwidth resource reuse, existing CDN bandwidth scheduling is business-based. This type of method can only solve CDN business scheduling and is not applicable to other business scenarios of edge computing, nor can it solve the resource reuse problem of large-scale edge nodes. Adopting the cloud native concept, we use CDN resource scheduling, resource dimension grouping, multi-cluster scheduling, and k8s operator to implement a distributed resource scheduling method for large-scale edge nodes with automatic coordination capabilities. From the perspective of resource scheduling, we split the scheduling process into two steps: central resource pool scheduling and edge k8s node scheduling. Central resource pool scheduling is based on resource dimensions, while edge scheduling is within the scope of resource pool scheduling results. It is defined and implemented through declarative APIs and k8s operators. Through multi-level scheduling at the center and edge, we achieve precise control of large-scale edge node resources.

[0114] Example 2:

[0115] In this embodiment, a computer device is provided, including a memory and a processor, the memory is used to store instructions, and the processor is used to execute the instructions, so that the computer device executes the multi-level resource scheduling method in the edge scenario as described above.

[0116] Example 3:

[0117] In this embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed, the operation of the multi-level resource scheduling method in the edge scenario as described above is implemented.

[0118] The computer-readable storage medium includes various media for storing program codes, such as a USB flash drive, a mobile hard disk, a read-only memory, a random access memory, a magnetic disk, or an optical disk.

[0119] Those skilled in the art will appreciate that the embodiments of the present application can be provided as methods, systems, or computer program products. Therefore, the present application can adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application can adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) that contain computer-usable program code.

[0120] The present application is described with reference to the flow chart and / or block diagram of the method, device (system), and computer program product according to the embodiment of the present application. It should be understood that each flow process and / or box in the flow chart and / or block diagram and the combination of the flow process and / or box in the flow chart and / or block diagram can be realized by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processing machine or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device produce a device for realizing the function specified in one flow chart flow or multiple flows and / or one box or multiple boxes of the block diagram.

[0121] These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce a product including an instruction device that implements the functions specified in one or more processes in the flowchart and / or one or more boxes in the block diagram.

[0122] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, so that the instructions executed on the computer or other programmable device provide steps for implementing the functions specified in one or more processes in the flowchart and / or one or more boxes in the block diagram.

[0123] In addition, although exemplary embodiments have been described in this application, the scope includes any and all embodiments based on the present application with equivalent elements, modifications, omissions, combinations (e.g., solutions that intersect various embodiments), adaptations, or changes. The elements in the claims are to be interpreted broadly based on the language used in the claims and are not limited to the examples described in this specification or during the prosecution of this application, which examples are to be interpreted as non-exclusive. Therefore, this specification and examples are intended to be considered as examples only, with the true scope and spirit being indicated by the following claims and the full scope of their equivalents.

[0124] The above description is intended to be illustrative rather than restrictive. For example, the above examples (or one or more of their solutions) can be used in combination with each other. For example, a person of ordinary skill in the art may use other embodiments when reading the above description. In addition, in the above-mentioned specific embodiments, various features can be grouped together to simplify the application. This should not be interpreted as an intention that a disclosed feature that is not required to be protected is necessary for any claim. On the contrary, the subject matter of the present application may be less than all the features of a specific disclosed embodiment. Thus, the following claims are incorporated into the specific embodiments as examples or embodiments, wherein each claim is independently a separate embodiment, and it is considered that these embodiments can be combined with each other in various combinations or arrangements. The scope of this application should be determined with reference to the appended claims and the full scope of equivalents to which these claims are entitled.

[0125] The technical features of the above-mentioned embodiments can be combined arbitrarily. In order to make the description concise, not all possible combinations of the technical features in the above-mentioned embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0126] The above-described embodiments merely represent several implementation methods of the present application. While the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the patent application. It should be noted that a person of ordinary skill in the art may make various modifications and improvements without departing from the spirit of the present application, and these modifications and improvements fall within the scope of protection of the present application. Therefore, the scope of protection of the present patent application shall be determined by the appended claims.

Claims

1. A multi-level resource scheduling method in an edge scenario, characterized in that: The following steps are involved: The CDN edge server is connected to the k8s cluster through the cloud-edge channel kubeedge and is exposed to business use through the virtual cluster virtual cluster; Apply for resources from the CDN resource scheduling platform based on the resource requirements of the business, and return a list of qualified resources on the premise of giving priority to CDN business; The business side maps the resource list to the virtual cluster according to the response, and evenly distributes it to all clusters. It generates CRD deployment orchestration and deploys it to the CDN edge cluster through the ESK platform. The CRD controller listens to the EdgeInfo resource and creates a pod based on the resource content; The business container reports the actual resource usage to the monitoring platform through indicators, and the resource scheduling platform performs global coordination based on the monitoring data and resource thresholds.

2. According to claim 1, a multi-level resource scheduling method in an edge scenario is characterized in that: The pod is synchronized to the CDN edge working cluster through syncer.

3. According to claim 2, a multi-level resource scheduling method in an edge scenario is characterized in that: To protect the controllable network resources of edge worker nodes, enable CNI and use the meta plug-in under the containernetworking project to limit the flow.

4. The multi-level resource scheduling method in an edge scenario according to claim 3, characterized in that: When applying for resources, specify one or more of the following dimensions: Dimension 1: total upstream and downstream bandwidth, in Mbps; Dimension 2: Application health value requirements, expulsion health value requirements; Dimension 3: single-machine CPU core number requirement, load application requirement, load eviction requirement, and load as a percentage; Dimension 4: memory size requirement, in M, memory load application requirement, memory load eviction requirement, load as percentage; Dimension 5: disk size requirement, in M, disk load application requirement, disk load eviction requirement, load as a percentage; Dimension 6: Link number load application requirement, link number load eviction requirement, load is the number.

5. The multi-level resource scheduling method in an edge scenario according to claim 4, characterized in that: The node dimension defines a resource usage template. After the service is applied to the resource pool, the node dimension resource usage is generated within the resource pool. The resource usage of each node is stated in EdgeInfo. The EdgeTask is used to define label selection, image version, number of replicas, and EdgeInfo instance information.

6. The multi-level resource scheduling method in an edge scenario according to claim 5, characterized in that: The controller schedules the pod to the idle Node node through node affinity, monitors the status of the node in the cluster, and deletes the corresponding Pod when the node status is abnormal and the idle Pod is deployed.

7. The multi-level resource scheduling method in an edge scenario according to claim 6, characterized in that: The K8s operator cooperates with the cloud-edge channel to implement edge node load scheduling, including the following steps: Operator registers custom resources of type EdgeTask and EdgeInfo with the api-server of k8s master to specify node usage rules and workloads. Operator starts a custom controller and maintains the number of EdgeTask replicas through recocile; The K8s node is registered with the k8s master through the cloud-edge channel composed of cloud-manager and edge-agent; When users operate businesses through the platform, the system will write information such as the specified machine range into the custom resource service orchestration of the EdgeInfo type, and write the image version, number of replicas, etc. into the custom resource service orchestration of the EdgeTask type; When the deployment is orchestrated to the virtual cluster, the custom controller reconcile will create a pod for each node in the EdgeInfo list. The custom controller reconciles according to the information in the EdgeInfo orchestration through reconcile, creates a task service pod, and submits it to the api-server; Use syncer to synchronize the pod to the CDN edge worker cluster.

8. The multi-level resource scheduling method in an edge scenario according to claim 7, characterized in that: The K8s operator cooperates with the cloud-edge channel to implement edge node load scheduling, which also includes the following steps: The cloud-manager in the cloud will watch the resource changes of the api-server in real time and identify the creation / update of the Pod; Cloud-manger distributes the pod orchestration modification message to the edge-agent at the edge through the cloud-edge channel; The edge-agent at the edge receives the orchestration modification message and calls CRI to manage the pod.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the multi-level resource scheduling method in the edge scenario as described in any one of claims 1-8 is implemented.

10. A device, characterized in that: include: A memory for storing instructions; A processor is used to execute the instruction so that the device performs operations to implement the multi-level resource scheduling method in the edge scenario as described in any one of claims 1-8.

Images