Distributed cloud resource scheduling method and system

Abstract

The invention relates to the technical field of communication, and discloses a distributed cloud resource scheduling method, which comprises the following steps of: 1, adopting an architecture in which a resource management layer is separated from a task scheduling layer; 2, cross-cloud center cluster scheduling is supported, and state information of a CPU, a memory, a network and the like of each cluster is obtained; 3, node grouping and tagging management are carried out, and affinity scheduling and taint node tolerance strategies are supported; and 4, integrating constraints such as task priority, resource requirements and SLA, and dynamically adjusting resource allocation by adopting particle swarm optimization and the like. According to the invention, a structure that a resource management layer and a task scheduling layer are separated is adopted, cross-cloud center cluster scheduling is supported, and the cluster scheduling is realized by obtaining state information of a CPU, a memory, a network and the like of each cluster, performing node grouping and labeling management, predicting resource requirements based on historical data, pre-allocating resources, dynamically adjusting and monitoring node states in real time. Therefore, the method has the advantages of low data transmission delay, high reliability and the like.

Description

Technical Field

[0001] The present invention relates to the field of communication technology, and in particular to a distributed cloud resource scheduling method and system. Background Art

[0002] In recent years, with the increasing number of network users and the widespread adoption of mobile devices, large-scale concurrent user access requests and multi-terminal access have become mainstream in everyday life and production scenarios. To meet the resource requirements and low latency constraints of services in highly reliable, low-latency communication scenarios, multiple computing platforms need to be deployed close to users to offload and process data locally, reducing the resource burden on services and application processing latency to meet the needs of everyday life and production. When business volume surges, existing computing resources need to be properly scheduled.

[0003] Most of the existing cloud resource scheduling methods currently adopt a single scheduling architecture, in which a central scheduler uniformly manages global resources and tasks. This can easily lead to transmission delays and poor reliability when processing large amounts of data. Summary of the Invention

[0004] (1) Technical problems solved

[0005] In response to the deficiencies of the existing technology, the present invention provides a distributed cloud resource scheduling method and system, which has the advantages of low latency and high reliability, and solves the above-mentioned problems.

[0006] (2) Technical solution

[0007] To achieve the above objectives, the present invention provides the following technical solution: a distributed cloud resource scheduling method, comprising the following steps:

[0008] Step 1: Adopt an architecture that separates the resource management layer from the task scheduling layer;

[0009] Step 2: Support cross-cloud center cluster scheduling by obtaining CPU, memory, network and other status information of each cluster;

[0010] Step 3: Node grouping and label management, supporting affinity scheduling (such as data locality) and tainted node tolerance strategy;

[0011] Step 4: Considering task priorities, resource requirements, SLAs (such as latency and throughput), and other constraints, dynamically adjust resource allocation using particle swarm optimization or other methods.

[0012] Step 5: Predict resource needs based on historical data, pre-allocate resources, and dynamically adjust them to reduce latency.

[0013] Step 6: Support fine-grained scheduling of multiple resources, including CPU, memory, storage, and network bandwidth, to address the efficiency bottleneck of traditional single-dimensional scheduling.

[0014] Step 7: Monitor node status in real time and automatically migrate tasks to other clusters when abnormalities occur.

[0015] Preferably, in step 1, the resource management layer is responsible for global resource monitoring and allocation, and the task scheduling layer handles the execution and fault tolerance of specific tasks. The task allocation constraint involves the formula: Meaning: Each task j can only be assigned to one physical machine or virtual machine i for execution, where xij is a binary variable (0 or 1) that represents the mapping relationship between tasks and resources.

[0016] Preferably, the step 2 includes selecting a target cluster in combination with a preset strategy (such as resource utilization, load balancing).

[0017] Preferably, the multi-dimensional resource scheduling in step six also includes introducing a knapsack problem optimization model to balance resource utilization and task completion time. Cloud resource scheduling must simultaneously meet multi-dimensional constraints such as computing, storage, and network, which is highly consistent with the multi-dimensional knapsack problem. The model regards virtual machines / containers as "items", and their CPU, memory, I / O and other requirements correspond to the "weight" of the items, while the service level (SLA) or user value corresponds to the "value".

[0018] Preferably, the step seven also includes supporting on-demand expansion / contraction of resources to cope with sudden traffic or business downturns.

[0019] A distributed cloud resource scheduling system includes a layered architecture model and core components.

[0020] Preferably, the layered architecture model includes a resource management layer, a task scheduling layer and a node execution layer, wherein the resource management layer is responsible for global resource monitoring and allocation; the task scheduling layer dynamically allocates task instances by the computing framework, and reduces network overhead in combination with data locality optimization; the node execution layer manages local resources through agents, and supports the rapid startup and recovery of containerized tasks. The core components include a resource manager, a scheduler and a service registration center, wherein the resource manager dynamically tracks node resources (CPU, memory, storage) and supports multi-dimensional resource description; the scheduler adopts a hybrid scheduling strategy (such as priority + fair sharing) and strengthens learning to optimize task allocation; the service registration center realizes service discovery through consistent hashing or DNS to ensure cross-cloud platform communication.

[0021] Compared with the existing technology, the present invention provides a distributed cloud resource scheduling method and system, which has the following beneficial effects:

[0022] 1. The present invention adopts an architecture that separates the resource management layer from the task scheduling layer, and supports cross-cloud center cluster scheduling. By obtaining the CPU, memory, network and other status information of each cluster, node grouping and labeling management, it supports affinity scheduling and tainted node tolerance strategies, as well as comprehensive task priority, resource requirements, SLA and other constraints. It adopts particle swarm optimization and other methods to dynamically adjust resource allocation, predict resource requirements based on historical data, pre-allocate resources and dynamically adjust them to reduce latency, support fine-grained scheduling of multi-dimensional resources such as CPU, memory, storage, and network bandwidth, solve the efficiency bottleneck of traditional single-dimensional scheduling, and monitor node status in real time. Therefore, this method has the advantages of low latency and high reliability in data transmission.

[0023] 2. In the present invention, the resource management layer is responsible for global resource monitoring and allocation, the task scheduling layer handles the execution and fault tolerance of specific tasks, and multi-dimensional resource scheduling also includes the introduction of the knapsack problem optimization model to balance resource utilization and task completion time. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] Figure 1 This is a block diagram of the distributed cloud resource scheduling system of the present invention. DETAILED DESCRIPTION

[0025] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0026] A distributed cloud resource scheduling method includes the following steps:

[0027] Step 1: Adopt an architecture that separates the resource management layer from the task scheduling layer. The resource management layer is responsible for global resource monitoring and allocation, while the task scheduling layer handles the execution and fault tolerance of specific tasks. The task allocation constraints involve the following formula: Meaning: Each task j can only be assigned to one physical machine or virtual machine i for execution, where xij is a binary variable (0 or 1) that represents the mapping relationship between tasks and resources;

[0028] Step 2: Support cross-cloud center cluster scheduling by obtaining CPU, memory, network and other status information of each cluster, including selecting the target cluster based on preset strategies (such as resource utilization and load balancing);

[0029] Step 3: Node grouping and label management, supporting affinity scheduling (such as data locality) and tainted node tolerance strategy;

[0030] Step 4: Considering task priorities, resource requirements, SLAs (such as latency and throughput), and other constraints, dynamically adjust resource allocation using particle swarm optimization or other methods.

[0031] Step 5: Predict resource needs based on historical data, pre-allocate resources, and dynamically adjust them to reduce latency.

[0032] Step 6: Support fine-grained scheduling of multi-dimensional resources such as CPU, memory, storage, and network bandwidth to address the efficiency bottleneck of traditional single-dimensional scheduling. Multi-dimensional resource scheduling also includes the introduction of the knapsack problem optimization model to balance resource utilization and task completion time. Cloud resource scheduling must simultaneously meet multi-dimensional constraints such as computing, storage, and network, which is highly consistent with the multi-dimensional knapsack problem. The model treats virtual machines / containers as "items," with their CPU, memory, I / O requirements corresponding to the item's "weight," and the service level (SLA) or user value corresponding to the "value."

[0033] Step 7: Monitor node status in real time and automatically migrate tasks to other clusters in case of anomalies. It also supports on-demand expansion / contraction of resources to cope with sudden traffic or business downturns.

[0034] See also Figure 1 ,A distributed cloud resource scheduling system includes a layered architecture model and core components.

[0035] The layered architecture model includes a resource management layer, a task scheduling layer, and a node execution layer. The resource management layer is responsible for global resource monitoring and allocation. The task scheduling layer dynamically allocates task instances through the computing framework and reduces network overhead by combining data locality optimization. The node execution layer manages local resources through agents and supports the rapid startup and recovery of containerized tasks. The core components include a resource manager, a scheduler, and a service registry. The resource manager dynamically tracks node resources (CPU, memory, storage) and supports multi-dimensional resource description. The scheduler adopts a hybrid scheduling strategy (such as priority + fair sharing) and optimizes task allocation through reinforcement learning. The service registry implements service discovery through consistent hashing or DNS to ensure cross-cloud platform communication.

[0036] The present invention adopts an architecture that separates the resource management layer from the task scheduling layer, and supports cross-cloud center cluster scheduling. By obtaining the CPU, memory, network and other status information of each cluster, node grouping and labeling management, it supports affinity scheduling and tainted node tolerance strategy, as well as comprehensive task priority, resource requirements, SLA and other constraints. It adopts particle swarm optimization and other methods to dynamically adjust resource allocation, predict resource requirements based on historical data, pre-allocate resources and dynamically adjust them to reduce latency, support fine-grained scheduling of multi-dimensional resources such as CPU, memory, storage, and network bandwidth, solve the efficiency bottleneck of traditional single-dimensional scheduling, monitor node status in real time, and automatically migrate tasks to other clusters in case of abnormalities.

[0037] While embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions, and variations may be made to these embodiments without departing from the principles and spirit of the invention, and that the scope of the invention is defined by the appended claims and their equivalents.

Claims

1. A distributed cloud resource scheduling method, comprising the following steps: Step 1: Adopt an architecture that separates the resource management layer from the task scheduling layer; Step 2: Support cross-cloud center cluster scheduling by obtaining CPU, memory, network and other status information of each cluster; Step 3: Node grouping and label management, supporting affinity scheduling and tainted node tolerance strategies; Step 4: Considering task priorities, resource requirements, SLAs and other constraints, dynamically adjust resource allocation using particle swarm optimization or other methods. Step 5: Predict resource needs based on historical data, pre-allocate resources, and dynamically adjust them to reduce latency. Step 6: Support fine-grained scheduling of multiple resources, including CPU, memory, storage, and network bandwidth, to address the efficiency bottleneck of traditional single-dimensional scheduling. Step 7: Monitor node status in real time and automatically migrate tasks to other clusters when abnormalities occur.

2. A distributed cloud resource scheduling method according to claim 1, characterized in that: In step 1, the resource management layer is responsible for global resource monitoring and allocation, and the task scheduling layer handles the execution and fault tolerance of specific tasks. The task allocation constraints involve the formula: Meaning: Each task j can only be assigned to one physical machine or virtual machine i for execution, where xij is a binary variable (0 or 1) that represents the mapping relationship between tasks and resources.

3. A distributed cloud resource scheduling method according to claim 1, characterized in that: The second step includes selecting a target cluster in combination with preset strategies (such as resource utilization and load balancing).

4. A distributed cloud resource scheduling method according to claim 1, characterized in that: Step 6, multi-dimensional resource scheduling, also includes the introduction of a knapsack problem optimization model to balance resource utilization and task completion time. Cloud resource scheduling must simultaneously meet multi-dimensional constraints such as computing, storage, and network, which is highly consistent with the multi-dimensional knapsack problem. The model regards virtual machines / containers as "items", and their CPU, memory, I / O and other requirements correspond to the "weight" of the item, while the service level (SLA) or user value corresponds to the "value".

5. The distributed cloud resource scheduling method according to claim 1, wherein: The step seven also includes supporting on-demand expansion / contraction of resources to cope with sudden traffic or business downturns.

6. A distributed cloud resource scheduling system, based on the distributed cloud resource scheduling method according to any one of claims 1 to 5, characterized in that: Includes layered architecture model and core components.

7. A distributed cloud resource scheduling system according to claim 6, characterized in that: The layered architecture model includes a resource management layer, a task scheduling layer, and a node execution layer. The resource management layer is responsible for global resource monitoring and allocation. The task scheduling layer dynamically allocates task instances through the computing framework, combining data locality optimization to reduce network overhead. The node execution layer manages local resources through agents and supports the rapid startup and recovery of containerized tasks. The core components include a resource manager, a scheduler, and a service registration center. The resource manager dynamically tracks node resources (CPU, memory, storage) and supports multi-dimensional resource description. Scheduler: adopts hybrid scheduling strategies (such as priority + fair sharing) and reinforcement learning to optimize task allocation; Service Registry: implements service discovery through consistent hashing or DNS to ensure cross-cloud platform communication.

Images