A distributed off-site multi-available area computing power scheduling method

Abstract

This invention relates to the field of distributed management technology, and in particular to a distributed multi-availability zone computing power scheduling method. By constructing a global resource view that includes network topology constraints and data gravity fields, and combining a three-layer topology consistency-first hierarchical scheduling strategy, the method improves the efficiency and reliability of computing power scheduling in distributed multi-availability zone scenarios, and avoids cross-availability zone data traffic overhead and network latency fluctuations. By writing scheduling decisions into a blockchain to form an immutable evidence chain, and combining an association rule mining algorithm to dynamically evaluate and self-optimize historical scheduling strategies, this method overcomes the shortcomings of existing technologies that rely on manual experience or offline analysis and lack a closed-loop iteration mechanism. It realizes the automatic evolution of scheduling strategies on multi-objective optimization weight parameters, enabling the system to adapt to changes in load characteristics and resource environment in real time, and continuously improve overall resource utilization and service quality.

Description

Technical Field

[0001] This invention relates to the field of distributed management technology, and in particular to a distributed multi-availability zone computing power scheduling method. Background Technology

[0002] With the full arrival of the digital economy era, the total amount of data has experienced explosive growth, and new application scenarios such as artificial intelligence, big data analytics, and the industrial internet are constantly emerging. Computing power has become a core productive force driving economic and social development. However, the current distribution of computing resources is extremely uneven across regions, levels, and entities: the eastern region has abundant data and high real-time requirements, but land, energy, and computing power costs remain high; the western region has abundant clean energy and huge potential computing resources, but local demand is relatively insufficient, forming a structural contradiction of resource and demand inversion. Therefore, distributed computing power scheduling is no longer limited to a single data center or a data center within the same city, but needs to support unified management and task distribution across regions, multiple levels, and heterogeneous resource pools, becoming a key infrastructure for ensuring large-scale business continuity and improving resource utilization. To further improve the high availability and disaster recovery capabilities of services, more and more computing resources are being deployed across multiple geographically dispersed availability zones, using a multi-availability zone architecture to avoid the risk of single points of failure and ensure that critical services continue to operate even under partial failures or disaster events.

[0003] However, existing scheduling methods often rely on manual experience or offline analysis for parameter adjustment, lacking a closed-loop mechanism for self-optimization and strategy iteration, making it difficult to adapt to dynamically changing load characteristics and resource environments. Summary of the Invention

[0004] The purpose of this invention is to provide a distributed multi-availability zone computing power scheduling method, which solves the technical problem that the parameter adjustment of the scheduling method in the prior art often relies on manual experience or offline analysis, lacks a closed-loop mechanism for self-optimization and strategy iteration, and thus is difficult to adapt to dynamically changing load characteristics and resource environment.

[0005] To achieve the above objectives, the present invention provides a distributed multi-availability zone computing power scheduling method, comprising: By deploying monitoring agents in each availability zone, the system collects and reports the status of compute nodes, network link quality, and the physical location and access frequency of data storage volumes in real time, thus constructing a global resource view that includes network topology constraints and data attraction fields. The centralized scheduler receives computing power requests, parses task types, data affinity requirements, and service level agreement parameters, and executes a three-layer topology consistency-first hierarchical scheduling strategy based on the global resource view, including the local scheduling layer, availability zone scheduling layer, and cross-domain scheduling layer. For stateful tasks, it forces the scheduling of computing resources to computing nodes in the same availability zone as the primary or hot replica of the data they depend on, forming a strong locality binding between computing and data. The key information of each scheduling decision is encapsulated into a scheduling record, a unique identifier is generated using a hash algorithm, and written into the blockchain network to form an immutable scheduling evidence storage chain. Historical scheduling records and corresponding task execution performance data are periodically extracted from the blockchain network. FP-Growth or Apriori association rule mining algorithms are applied to perform association rule analysis and pattern evaluation on historical scheduling strategies, and the multi-objective optimization weight parameters in the hierarchical scheduling decision-making steps are dynamically optimized.

[0006] In the encryption, evidence storage, and traceability steps of the scheduling process, the blockchain network adopts a consortium blockchain architecture. Each availability zone management node participating in the computing power scheduling acts as a consensus node. It uses a practical Byzantine fault tolerance or Raft consensus mechanism to jointly maintain the ledger of scheduling records, providing a reliable and auditable third-party evidence storage for scheduling decisions.

[0007] In the data mining-based scheduling algorithm optimization step, the data mining algorithm includes association rule mining, which is used to identify strong association rules between "specific task type - specific data location - specific availability zone resource status" and "optimal scheduling result". The newly mined rules are used as prior knowledge and injected into the initial decision model of the centralized scheduler to realize the adaptive evolution of the scheduling strategy.

[0008] In executing hierarchical scheduling decisions, the scheduler uses historical scheduling records and corresponding node power consumption data as training samples, and uses long short-term memory networks or temporal convolutional networks to train energy consumption prediction models to predict the expected energy consumption increment and carbon emission factor of each candidate computing node after undertaking the scheduling task in real time. The predicted unit computing power energy consumption, carbon emission intensity, and service level agreement constraints of the task are used together as dynamic weight factors in the multi-objective optimization function. The Particle Swarm Optimization (PSO) algorithm or a non-dominated sorting genetic algorithm with an elitist strategy is used to solve the scheduling solution set that satisfies the performance constraints and optimizes the global energy consumption. The task is scheduled to the availability zone with the lowest unit computing power energy consumption or the highest proportion of clean energy in the current time period.

[0009] When performing computing power scheduling across availability zones, scheduling instructions and transmitted task data are dynamically encrypted based on identity and context. The scheduler and computing nodes perform two-way authentication through an attribute-based access control model to ensure that a secure transmission channel is established only under the premise of satisfying the principle of least privilege, preventing malicious nodes from impersonating or data from being leaked.

[0010] The global topology and data gravity sensing steps include: Construct a network latency awareness matrix to calculate the end-to-end latency from the request entry point, computing node to data storage location in real time; A data gravity field model is established. Based on the size of the data volume, access frequency, and number of cross-region accesses, the "gravity coefficient" of the data in different availability zones is dynamically calculated. When the data gravity coefficient exceeds the preset threshold, the hierarchical scheduling decision step will automatically trigger the cache preheating or asynchronous data replication mechanism to push the data to the scheduling target availability zone in advance.

[0011] The hierarchical scheduling decision-making steps include: Allocate independent network links or priorities for the scheduler's own metadata communication and business data communication to ensure that the stability of control plane communication is not affected by data plane traffic during network jitter, and avoid incorrect node expulsion or split-brain due to misjudgment of the control plane. For core services that include multiple replicas, the scheduler enforces a strict anti-affinity policy based on availability zones and physical racks to ensure that multiple replicas of the same service are deployed in different fault domains. When a failure is detected in an availability zone, the scheduler assesses the total remaining capacity of the remaining availability zones. If the remaining capacity is insufficient to handle the entire load of the failed zone, a circuit breaker mechanism is activated, allowing only core services with a priority higher than a preset threshold to be scheduled across zones. Non-core services are placed in a waiting queue to avoid an avalanche effect caused by full scheduling.

[0012] The hierarchical scheduling decision-making steps also include: The centralized scheduler periodically analyzes the resource fragmentation rate and load distribution of each availability zone, identifies and filters low-priority or stateless tasks that can be migrated, generates rescheduling plans, and executes them during off-peak periods to dynamically distribute unevenly distributed loads and achieve global load balancing across availability zones.

[0013] This invention presents a distributed, multi-availability-zone (MAZ) computing power scheduling method. By constructing a global resource view that includes network topology constraints and a data gravity field, and combining it with a three-layer topology consistency-first hierarchical scheduling strategy, it significantly improves the efficiency and reliability of computing power scheduling in distributed, multi-availability-zone scenarios. For stateful tasks, it enforces strong locality binding between computation and data, effectively avoiding cross-availability-zone data traffic overhead and network latency fluctuations, ensuring deterministic performance of task execution. Furthermore, by writing scheduling decisions into a blockchain to form an immutable evidence chain, and combining it with an association rule mining algorithm to dynamically evaluate and self-optimize historical scheduling strategies, it overcomes the shortcomings of existing technologies that rely on manual experience or offline analysis and lack a closed-loop iteration mechanism. This achieves automatic evolution of scheduling strategies in multi-objective optimization weight parameters, enabling the system to adapt to changes in load characteristics and resource environment in real time, continuously improving overall resource utilization and service quality. Attached Figure Description

[0014] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the accompanying drawings used in the description of the embodiments or the prior art will be briefly introduced below.

[0015] Figure 1 This is a flowchart of the distributed multi-availability zone computing power scheduling method of the present invention. Detailed Implementation

[0016] The embodiments of the present invention are described in detail below. Examples of the embodiments are shown in the accompanying drawings. The embodiments described below with reference to the accompanying drawings are exemplary and intended to explain the present invention, but should not be construed as limiting the present invention.

[0017] Please refer to Figure 1 , Figure 1 This is a flowchart of the distributed multi-availability zone computing power scheduling method of the present invention. An embodiment of the present invention provides a distributed multi-availability zone computing power scheduling method, including: S1. By deploying monitoring agents in each availability zone, collect and report the status of compute nodes, network link quality, and the physical location and access popularity of data storage volumes in real time, and construct a global resource view that includes network topology constraints and data gravity field. In this specific implementation, the global topology and data gravity sensing steps include: Construct a network latency awareness matrix to calculate the end-to-end latency from the request entry point, computing node to data storage location in real time; A data gravity field model is established. Based on the size of the data volume, access frequency, and number of cross-region accesses, the "gravity coefficient" of the data in different availability zones is dynamically calculated. When the data gravity coefficient exceeds the preset threshold, the hierarchical scheduling decision step will automatically trigger the cache preheating or asynchronous data replication mechanism to push the data to the scheduling target availability zone in advance.

[0018] S2. The centralized scheduler receives computing power requests, parses task types, data affinity requirements and service level agreement parameters, and executes a three-layer topology consistency-first hierarchical scheduling strategy based on the global resource view, including the local scheduling layer, availability zone scheduling layer and cross-domain scheduling layer. For stateful tasks, it forces the scheduling of computing resources to computing nodes in the same availability zone as the primary or hot replica of the data they depend on, forming a strong locality binding between computing and data. In this specific implementation, the hierarchical scheduling decision-making steps include: Allocate independent network links or priorities for the scheduler's own metadata communication and business data communication to ensure that the stability of control plane communication is not affected by data plane traffic during network jitter, and avoid incorrect node expulsion or split-brain due to misjudgment of the control plane. For core services that include multiple replicas, the scheduler enforces a strict anti-affinity policy based on availability zones and physical racks to ensure that multiple replicas of the same service are deployed in different fault domains. When a failure is detected in an availability zone, the scheduler assesses the total remaining capacity of the remaining availability zones. If the remaining capacity is insufficient to handle the entire load of the failed zone, a circuit breaker mechanism is activated. Only core services with a priority higher than a preset threshold are allowed to be scheduled across zones, while non-core services enter a waiting queue to prevent a cascading failure caused by full scheduling. The hierarchical scheduling decision-making process also includes: The centralized scheduler periodically analyzes the resource fragmentation rate and load distribution of each availability zone, identifies and filters low-priority or stateless tasks that can be migrated, generates rescheduling plans, and executes them during off-peak periods to dynamically distribute unevenly distributed loads and achieve global load balancing across availability zones.

[0019] S3. Encapsulate the key information of each scheduling decision into a scheduling record, generate a unique identifier using a hash algorithm, and write it into the blockchain network to form an immutable scheduling evidence storage chain. In this specific implementation, during the encryption and traceability steps of the scheduling process, the blockchain network adopts a consortium blockchain architecture. Each availability zone management node participating in the computing power scheduling acts as a consensus node, and uses a practical Byzantine fault tolerance or Raft consensus mechanism to jointly maintain the ledger of scheduling records, providing a reliable and auditable third-party evidence for scheduling decisions.

[0020] S4. Periodically extract historical scheduling records and corresponding task execution performance data from the blockchain network, apply FP-Growth or Apriori association rule mining algorithms to perform association rule analysis and pattern evaluation on historical scheduling strategies, and dynamically optimize the multi-objective optimization weight parameters in the hierarchical scheduling decision-making steps.

[0021] In this specific implementation, the data mining algorithm optimization step includes association rule mining, which is used to identify strong association rules between "specific task type - specific data location - specific availability zone resource status" and "optimal scheduling result". The newly mined rules are used as prior knowledge and injected into the initial decision model of the centralized scheduler to achieve adaptive evolution of the scheduling strategy.

[0022] When performing computing power scheduling across availability zones, scheduling instructions and transmitted task data are dynamically encrypted based on identity and context. The scheduler and computing nodes perform two-way authentication through an attribute-based access control model to ensure that a secure transmission channel is established only under the premise of satisfying the principle of least privilege, preventing malicious nodes from impersonating or data from being leaked.

[0023] Secondly, when executing hierarchical scheduling decisions, the scheduler uses historical scheduling records and corresponding node power consumption data as training samples, and uses long short-term memory networks or temporal convolutional networks to train energy consumption prediction models, and predicts in real time the expected energy consumption increment and carbon emission factor of each candidate computing node after undertaking the scheduled task. The predicted unit computing power energy consumption, carbon emission intensity, and service level agreement constraints of the task are used together as dynamic weight factors in the multi-objective optimization function. The Particle Swarm Optimization (PSO) algorithm or a non-dominated sorting genetic algorithm with an elitist strategy is used to solve the scheduling solution set that satisfies the performance constraints and optimizes the global energy consumption. The task is scheduled to the availability zone with the lowest unit computing power energy consumption or the highest proportion of clean energy in the current time period.

[0024] Meanwhile, the scheduler periodically analyzes the historical resource usage curves of each availability zone, and combined with the task queuing length, uses a time-series prediction model to predict the peak resource demand within the future time window. When the prediction result exceeds the preset expansion threshold, it activates backup computing nodes or reserves resource containers in advance in the target availability zone to smoothly cope with sudden computing power requests.

[0025] When executing stateful tasks, the scheduler periodically writes snapshots of the task's intermediate computation results and execution context to the shared storage system across availability zones. When a failure is detected on the compute node where the task resides or a rescheduling is triggered, the task execution is resumed from the most recent snapshot point without restarting the entire task process.

[0026] Furthermore, dedicated computing resource quotas are reserved in the target availability zone for high-priority or critical tasks to ensure resource availability during scheduling. When resources are scarce and high-priority tasks arrive, the scheduler triggers a resource preemption mechanism to prioritize the removal of low-priority tasks or migrate them to other availability zones, releasing resources to meet the execution needs of high-priority tasks.

[0027] Secondly, the monitoring agent collects real-time power usage efficiency (PUE) data for each availability zone and real-time power consumption data for computing nodes. When the scheduler executes the hierarchical scheduling strategy, it uses unit computing power energy consumption as one of the multi-objective optimization weight parameters. Under the premise of meeting the service level agreement, it prioritizes scheduling tasks to availability zones with lower PUE values ​​or computing nodes with higher current power efficiency.

[0028] This invention provides a distributed, multi-availability-zone (MAZ) computing power scheduling method. By constructing a global resource view that includes network topology constraints and a data gravity field, and combining it with a three-layer topology consistency-first hierarchical scheduling strategy, the efficiency and reliability of computing power scheduling in distributed, multi-availability-zone scenarios are significantly improved. For stateful tasks, a strong locality binding between computation and data is enforced, effectively avoiding cross-availability-zone data traffic overhead and network latency fluctuations, ensuring deterministic performance of task execution. Furthermore, by writing scheduling decisions into a blockchain to form an immutable evidence chain, and combining it with an association rule mining algorithm to dynamically evaluate and self-optimize historical scheduling strategies, the invention overcomes the shortcomings of existing technologies that rely on manual experience or offline analysis and lack a closed-loop iteration mechanism. This enables the scheduling strategy to automatically evolve in terms of multi-objective optimization weight parameters, allowing the system to adapt to changes in load characteristics and resource environment in real time, continuously improving overall resource utilization and service quality.

[0029] The above-disclosed embodiments are merely one or more preferred embodiments of this application and should not be construed as limiting the scope of this application. Those skilled in the art can understand that all or part of the processes for implementing the above embodiments and equivalent changes made in accordance with the claims of this application still fall within the scope of this application.

Claims

1. A distributed multi-availability zone computing power scheduling method, characterized in that, include; By deploying monitoring agents in each availability zone, the system collects and reports the status of compute nodes, network link quality, and the physical location and access frequency of data storage volumes in real time, thus constructing a global resource view that includes network topology constraints and data attraction fields. The centralized scheduler receives computing power requests, parses task types, data affinity requirements, and service level agreement parameters, and executes a three-layer topology consistency-first hierarchical scheduling strategy based on the global resource view, including the local scheduling layer, availability zone scheduling layer, and cross-domain scheduling layer. For stateful tasks, it forces the scheduling of computing resources to computing nodes in the same availability zone as the primary or hot replica of the data they depend on, forming a strong locality binding between computing and data. The key information of each scheduling decision is encapsulated into a scheduling record, a unique identifier is generated using a hash algorithm, and written into the blockchain network to form an immutable scheduling evidence storage chain. Historical scheduling records and corresponding task execution performance data are periodically extracted from the blockchain network. FP-Growth or Apriori association rule mining algorithms are applied to perform association rule analysis and pattern evaluation on historical scheduling strategies, and the multi-objective optimization weight parameters in the hierarchical scheduling decision-making steps are dynamically optimized.

2. The distributed multi-availability zone computing power scheduling method as described in claim 1, characterized in that, In the encryption, evidence storage, and traceability steps of the scheduling process, the blockchain network adopts a consortium blockchain architecture. Each availability zone management node participating in the computing power scheduling acts as a consensus node. It uses a practical Byzantine fault tolerance or Raft consensus mechanism to jointly maintain the ledger of scheduling records, providing a credible and auditable third-party evidence storage for scheduling decisions.

3. The distributed multi-availability zone computing power scheduling method as described in claim 2, characterized in that, In the data mining-based scheduling algorithm optimization step, the data mining algorithm includes association rule mining, which is used to identify strong association rules between "specific task type - specific data location - specific availability zone resource status" and "optimal scheduling result". The newly mined rules are used as prior knowledge and injected into the initial decision model of the centralized scheduler to realize the adaptive evolution of the scheduling strategy.

4. The distributed multi-availability zone computing power scheduling method as described in claim 3, characterized in that, When executing hierarchical scheduling decisions, the scheduler uses historical scheduling records and corresponding node power consumption data as training samples, and uses long short-term memory networks or temporal convolutional networks to train energy consumption prediction models to predict the expected energy consumption increment and carbon emission factor of each candidate computing node after undertaking the scheduled task in real time. The predicted unit computing power energy consumption, carbon emission intensity, and service level agreement constraints of the task are used together as dynamic weight factors in the multi-objective optimization function. The Particle Swarm Optimization (PSO) algorithm or a non-dominated sorting genetic algorithm with an elitist strategy is used to solve the scheduling solution set that satisfies the performance constraints and optimizes the global energy consumption. The task is scheduled to the availability zone with the lowest unit computing power energy consumption or the highest proportion of clean energy in the current time period.

5. The distributed multi-availability zone computing power scheduling method as described in claim 4, characterized in that, When performing computing power scheduling across availability zones, scheduling instructions and transmitted task data are dynamically encrypted based on identity and context. The scheduler and computing nodes perform two-way authentication through an attribute-based access control model to ensure that a secure transmission channel is established only under the premise of satisfying the principle of least privilege, preventing malicious nodes from impersonating or data from being leaked.

6. The distributed multi-availability zone computing power scheduling method as described in claim 5, characterized in that, Global topology and data gravity sensing steps include: Construct a network latency awareness matrix to calculate the end-to-end latency from the request entry point, computing node to data storage location in real time; A data gravity field model is established. Based on the size of the data volume, access frequency, and number of cross-region accesses, the "gravity coefficient" of the data in different availability zones is dynamically calculated. When the data gravity coefficient exceeds the preset threshold, the hierarchical scheduling decision step will automatically trigger the cache preheating or asynchronous data replication mechanism to push the data to the scheduling target availability zone in advance.

7. The distributed multi-availability zone computing power scheduling method as described in claim 6, characterized in that, The hierarchical scheduling decision-making steps include: Allocate independent network links or priorities for the scheduler's own metadata communication and business data communication to ensure that the stability of control plane communication is not affected by data plane traffic during network jitter, and avoid incorrect node expulsion or split-brain due to misjudgment of the control plane. For core services that include multiple replicas, the scheduler enforces a strict anti-affinity policy based on availability zones and physical racks to ensure that multiple replicas of the same service are deployed in different fault domains. When a failure is detected in an availability zone, the scheduler assesses the total remaining capacity of the remaining availability zones. If the remaining capacity is insufficient to handle the entire load of the failed zone, a circuit breaker mechanism is activated, allowing only core services with a priority higher than a preset threshold to be scheduled across zones. Non-core services are placed in a waiting queue to avoid an avalanche effect caused by full scheduling.

8. The distributed multi-availability zone computing power scheduling method as described in claim 7, characterized in that, The hierarchical scheduling decision-making process also includes: The centralized scheduler periodically analyzes the resource fragmentation rate and load distribution of each availability zone, identifies and filters low-priority or stateless tasks that can be migrated, generates rescheduling plans, and executes them during off-peak periods to dynamically distribute unevenly distributed loads and achieve global load balancing across availability zones.

Images